Well stealth, poison, archery, and hand to hand skills do help a lot
There are several approaches to the town. The easiest to breach is the southeastern path over the stream. I typically sneak across the bridge, take a shot with a poisoned arrow at one of the guards posted there, and run back across the bridge if things don’t work out. It’s quite possible to mismantle most of the Cuman camp like this with the help of a few bane potions in the food pots.
The bandit camp in the upper portion of the town is a tougher nut to crack but it can be taken out similarly.