Brawls isn't working

So when I went to do the brawls in Rattay, it brought me to a little fenced area outside the castle. However once I left that fenced area I lost. While my opponent is outside it. How do I beat them?