Charisma is affected predominantly by several things:
Alcohol (Which can provide short-term buffs whilst drunk so long as you don’t get carried away)
Clothing makes the man, finer attire will generally net a charisma boost, bear in mind charisma boosts only come from VIEWABLE clothing, if you have say a jacket on over something that would provide a charisma boost, that can negate it.
Cleanliness, next to godliness, Bloody and dirty attire can also negatively impact your appearance. Noone wants to talk with someone covered in blood and mud.
Charisma/speech perks. There ARE ways to give yourself a perk-boost to Charisma, the chief one is ‘fragrance’ (under Maintanence) which, when carrying a few fragrant herbs will give you a +2 Charisma bonus, there is also “Ken”, which will give a +2 to charisma at a -1 to Strength. Fragrance is an easy one to net as it requires little more than hitting the grind-stones a few times to keep your weapons in shape (and using grind-stones to maintain blades is free.)
All told, it takes a substantial amount of coin, but you can pull together clothing to get you to the 14-16 charisma range, then use those perks to boost you to 20.
Edit log(s): Changed wording for opening, perks
PS: Ken (Which is under main levels) will in fact give a -2, not -1, apologies.