See, the reverse bothers me.

When you're talking about actual mental illness, at least you're maintaining some consistency. It's offensive and a slur, but you're still talking about the same thing. When crazy is taken to mean other things, like ridiculous, unwise, excessive, or illogical (which are the kinder side of the ill use), you're associating mental illness with something mental illness is not. It's kind of like the difference, I think, between obvious and veiled prejudice. When people are open about their prejudice, it's easier to recognize what they're doing and the harm it can cause. When they are more subtle about it, the harm gets ignored, even though it still exists.

But it also depends on how much you associate "Crazy" with "Mental illness". Obviously a big part of my argument is that there's a strong association with the two, so associating anything else with crazy also associates it with mental illness. That's not the way it is for everyone.