It bothers me when it's used to refer to an actual mental illness. In other words, "That's a crazy idea," doesn't bother me, but "My brother went crazy and they put him in the crazy house," would bother me. (Speaking as a person who is medicated and has been in mental hospitals blah blah blah my mental illness cred let me show it to you.)

But I agree with you that my person opinion about when it bothers me really doesn't trump anyone else's opinion.

(I do reserve the right to use the term to refer to myself, but I've learned the hard way not to talk much about having a mental illness.)