A demonstration against MP Yvette Cooper was fuelled by "troublemakers" from outside the constituency, a local councillor said.
Joshua Spencer, co-founder of Brexiteers of Normanton, Pontefract and Castleford, said he had organised Sunday's march to demonstrate local dissatisfaction with Ms Cooper.
He said: “It was good. We had people asking us what it was about and saying they were on our side even though they couldn't join in."
In a tweet, Ms Cooper said: "Some far right “Tommy Robinson” supporters from out of town came into Castleford today, inc one apparently calling for me to be burnt - just as we were marking @great_together for Jo Cox across West Yorks.
"Shows Jo’s #moreincommon legacy is more important than ever #hopenothate".
Speaking before the march, Mr Spencer said: "We're part of a cross-party campaign to ensure all parts of the United Kingdom leave.
"What we've noticed after weeks and weeks of campaigning in the streets in Castleford and Pontefract is that the consensus is that people, remain or leave, are sick and tired of the backstabbing and betrayal of their MP.
"We're asking for Yvette Cooper to pipe down on the backbenches and ensure a no deal Brexit.
"She can either carry on with the rhetoric that she has, or she can call a by-election and let the people decide.
"It's a peaceful demonstration. We reject the politics of hate.
"People are quite clear that they’re not scared to leave without a deal. People don’t believe, don’t feel that they will lose their jobs."
Mr Spencer said that around 250 people had taken part in the march, though local councillor Tony Wallis said he had only seen a crowd of about 50 people.
Coun Wallis said he believed that “troublemakers” from outside the constituency were interfering in local politics and “demonising” Ms Cooper.
He said: “I think most of the shoppers just said the demonstration was happening, but they weren’t actually talking about the issues or the subjects. I’ve come across it when I talk to people, mainly to do with Brexit
"But I think there’s a lot of troublemakers that have come along.
“I think what we’ve got is frustration about Brexit. I do understand it. Round here a lot of people voted to come out of Europe and though I keep explaining that Yvette’s not trying to stop it, she’s just trying to get a deal, I think the view for people out there is she is (trying to stop Brexit).
“I’m not saying there aren’t concerns, there are people in Castleford who, rightly or wrongly, are concerned how she’s doing things about Brexit.
“But I think there’s a lot of outsiders exploiting that and making it worse than it is.
“I think there’s a lot of anger and frustration in the town, I don’t doubt that. I think unfortunately it has been turned into this nonsense when really what we should be doing is having a healthy discussion, respecting people with different ideas and not demonising them.”