The Impact of Scatter in the Galaxy UV Luminosity to Halo Mass Relation on Lyα Visibility During the Epoch of Reionization