Everybody is suspicious all of the sudden

For some reason the last few days every regular NPC is suspicious when I am nearby.

For example:
I am entering a larger area, like a farm. Even from far away I can already hear them go “Is somebody there?” “Hey, who goes there?” “Didn’t I hear something?” But they still go on with their work and don’t go looking for me.

But, I am not in a restricted area. There is no blue symbol telling me that I am in a private area.

Even worse: When I approach them and stand directly in front of them they still go “Huh? Who is there?”

This finally breaks immerson for me completely. I can’t wander through streets or rural areas anymore because every regular NPC is thinking I am intruding somewhere, while I’m still outside of private areas. Can’t go on playing like this…

Platform: Xbox One