It's not driving away his target market, it's fully changed it. His target market is now MAGA brain-rot Republican weirdos, who also want an EV. It's not a big market.
People with concern for and a desire to do something about Climate Change are the largest demographic buying EVs, and they are a very left-leaning bunch. To say Elon's rhetoric has pushed that market section away is a dramatic understatement.