The other problem I have with that scenario is the idea that the locals would automatically fit the newcomers into their social paradigm. I think it's far more likely that they'd see them as foreigners and realize they have different beliefs, even if the locals don't understand or respect them. That could create some interesting conflicts and situations. You wouldn't need to gender-swap, however.