If you want to see it in this way, then yes. Story is really the main thing here. But the game will give you rising numbers, skills and perks. Maybe you will like combat more, where you really can “get better” - in terms of numbers as well as in terms of your skill as a player. Or archery. Or even alchemy, if you started with that. For a long time you can just forget about whole story, roam the land and kill bandits.
And thieving! If you like growing numbers, try to break into chest of every merchant in Rattay. You will get better in thieving and get more and more money.