Complaining about so-called miscasting, for some, IS about not liking attempts to be more diverse with main characters not just being white. They might be fine with a black actor playing some other role, but they’d damn well better not make a normally white character black!
Some people flipped their shit when a black actress was cast as Hermione Granger in Harry Potter and the Cursed Child on stage, and they flipped their shit even more when J.K. Rowling defended it. People also flipped their shit when she said Dumbledore was gay.
Don’t bother trying to talk in absolutes to me. I’m not playing that game with you.