Activity: Stealing horse for Woyzeck

So, one of the millers asked me to steal a horse from Merhojed for him. I did that. He was satisfied with the horse and I thought I’m done with taht particular task. I get on my horse(which I bought from the same stable and suddenly it tells me to bring the horse to the miller to fulfill the quest. This got annoying real fast since every time I get on my horse it says “Quest started”. Is there a way to cancel activities ?

Something similar happened to me - when I brought him a horse and get on my own (Pebbles) and later get off, the quest restarts. But I can continue with other Vojtěch’s quests. So I ignore it.