Well to be fair, by the time of the books I don't think he was. He was many things, but racist was not one of them. As a child then teenager (before Lily's death and therefore when she knew him) however....