Really … don’t. Well, quest items anyway. Even regular items can have some impact. Take only what you need if an existing quest demands it. Taking quest items could make it difficult to impossible to complete quests later in the game.
Also, get stuff off of your horse before entering the monastery. Your horse is not accessible. With some difficulty your other possessions can be accessed. Having said that, if you don’t steal from the monastery then you really don’t need to leave it either.
Why even bother entering the monastery? You can just jump into the monastery via roof access, stab Pious to death, go to jail, and get out and complete the quest anyway. Or just snipe him if he’s in the open and you’ve got the archery skills to nail him on the first shot.
I stole that place blind!
Not a chalice, cross, food or bauble left behind.
Clothes, books I read already and other low value stuff I left alone ( except for the circators clothing.)
Everything of value!
Even the monks stash. I did give pious his dagger back( seemed to mess up things in order after I took it back later.)
I was tempted to have all the monks in their skivvies but let that pass.
The 2 circators I took a dislike to so I caught them alone in the hallway and choked them out and stole their clothes. John I repeatedly knock out ( poor guy has fainting spells apparently 
) .
I had previously opened some doors to the outside before entering and used them to sneak out to get my stuff from the chest you are to use.
Ok, granted I used mods for carry weight and lock picking and zero conspicuness. But I had fun!
I did the whole monk thing, the pages quest, transcription job( save , try again), alchemy was a snap ( stole there too!), got the circators off a novices back, and completed the quest and slammed pious into the dungeon .
I received a complete.
The only bug I have is a side quest to go eat with the other monks. BUT no constant reminder that I am neglecting my daily chores.
I didn’t steal the wine tho and didn’t get the nasty award.
I like to play through the quests in character. I find that i really enjoy the quest dialogue and interactions. Often times the non violent quests are the most fun.
SO … I did sort of the same thing LongShot did. I grabbed every bit of food that I could find and bought a lock pick. Once I had that I had access to the outside world and my stuff. I scooped up the quest items that I had stolen earlier and all was well from there.
Thanks for this awesome advice. I will stick to only stealing the souls of everyone in the monastery.
They can keep their trinkets. In fact I’m such a good guy I’ll bury them with their objects.
That’s a Spoiler! Thumbs down.
