Racism is prejudice plus power. Always. So is sexism, and all of the other *isms that don't make it really clear that the people they're used against are always minorities (see: transphobia and homophobia). Minorities don't have the power--institutional or personal--to subject a majority to anything other than simple prejudice.

That also applies to the hateful grandfathers who rarely leave their houses and don't vote. Because racism/sexism/etc is so vast and pervasive that even people who rarely exercise their power still have it in spades.

And if I ever have to hear "But I was in a mostly [insert minority group here] place, and they were meeeeaaaan" again, I'm going to stab someone, or scream, or lose all respect for the person who said it.