I might have just found something here, I run a Ryzen 3800X with SMT off, so it’s an 8 core CPU, and it’s quite good !
When I load a save at Rattay, the performance goes up from 20, to 45, to 55 FPS, and that’s great especially considering I push some settings quite high.
But after lets say 15-20 seconds, suddenly, I drop back to 40 FPS, while standing still and nothing new is happening on screen.
For a long time I thought the game was just very heavy in resources, but then I took a closer look to my CPU utilization and noticed something really weird, at the exact same time as my frramerate when immediately from 55-60 to 35-40, while CPU 5 6 7 8 (or 4 5 6 7 in real affinity numbers) still stay heavily solicited at ± 80%, CPU 1 2 3 4 (or 0 1 2 3 in real affinity numbers) drops from ± 80% to only 30% !! And the drop in CPU utilization is brutal, like a switch being toggled on somewhere !
So yes, there is something weird going on, and the performance problems in this game could be explained by something nasty hidden deeply into the game code, or the engine itself.
I can do a video recording of what I explained if that is required.
I know this game has been released for quite some time now… But I’d be extremely happy if someone from the team would like to give it a look 
I commented everything in my user.cfg just to be sure it wasn’t some nasty config modification, but it does the exact same behavior with a pure and clean one.
No mods installed except for bush collision remover and translucent compass, hardly think they have anything to do.
Of course game is legit on steam and latest version.