Yeah, I took it as not white because of how different Katniss and Prue appeared in the book. I always figured Collins meant that Katniss' family was interracial (I hate this term but I can't think of another), plus evoked historical recollections of colonialism etc. to highlight just how surprising it was that the white woman left to marry the not-white miner. Power of love and all that.

But, yeah, 'olive skin' wasn't the greatest word choice.