There is actually lot of good starting gear in that tower, unguarded.
You can have a nearly a full set of brigandine, and a few plate pieces in very easy chests.
Several medium tier long/short swords are in the area and lots of bows.
Because so much good loot is available early, if you know where to look – on higher quality armors there needs to be an agility and strength requirement so Henry can’s steam roll everything so early in game.
I think it’s perfectly reasonable to say that if you’re not fit enough to wear armor, then your effectiveness should be decreased.