More to the point, it doesn't really matter if he was a bully when he was a student. He was a bully as an adult. To students who weren't able to fight back. Not only because he was a position of power but because he as a teacher he knew a lot more then the students - particularly the first years. That sort of thing is much worse then most of the stuff that James pulled during his time.
Being the victim of bullies himself (and not just the Murderers) may explain his bitterness as an adult, but it does NOT excuse his actions.