Wakefield Council has launched a review of its six markets, including South Elmsall, which is open five days a week from 8.30am to 4pm on Barnsley Road.
On Monday, traders were urged not to pack up and leave before closing time to avoid creating a "negative perception" of the market for shoppers by the council's markets manager Ben Cook.
But while stall holders acknowledge early leavers are a problem, they say the market's issues run deeper.
Many of the traders, who've been regulars at the market since the early 1980s, said their rapport with "loyal" customers was one of the best things about the job, but said the venue needed to be reinvented for it to thrive again.
Trading rep Susan Fretwell said: "I've been here 35 years. I've seen how the market's changed over the years.
"I used to trade in Wakefield and you weren't allowed to leave early. You couldn't pack up before 3.30pm. But at the end of the day if you've got people coming round, you'll stay."
Mr Cook told a scrutiny committee meeting that South Elmsall was a "low rent market" compared to neighbouring ones, and suggested rates were unlikely to be lowered.
But some traders claimed they could get a shop unit in the town for a better price.
Shoe and clothes seller Bill Evans, who's been using the market for 12 weeks, said: "The rents are just way too high.
"Because I've not been here long, I've had discounts so I know I've got a good turnover at the moment.
"But once it goes up to full price I don't know what I'm going to do.
"I think the market needs better advertising. It's no good putting stuff about it on social media when it's the older generation who use it. They're not on social media.
"I'd also like to see it condensed too. It's better looking at one busy aisle than three aisles where there's not a lot going on.
"It's a shame because it's a great market, the customers are really loyal and you can have a laugh with them."
Sweets seller Trish Wilson said: "It's not a problem when people pack up and leave at 3pm - it's more when they go at 12 that's an issue.
"But it's always been a morning market. It does get very quiet in the afternoon.
"Thiry years ago we didn't have much in the way of supermarkets around us, but now we've got Asda, Tesco and now Aldi coming too.
"I pay £24 a month to park my car here, which is cheap but I still think they could do to lower it. If parking for us was maybe free after 1pm that would help bring more custom in for the afternoon.
"Not everything's negative, I absolutely my love my job and hopefully I'll be here for a few years yet. But we need more help and we need the market to be busier."
Michelle Clark, a clothes trader at the market since 1985, said: "The problem is that a lot of traders are retiring and there aren't the younger ones coming through to replace them because there's no real incentive to do it.
"If they could look at reducing the rents it would really help. A lot of the other markets like Pontefract and Castleford are on a precinct that's busy anyway, whereas here it isn't."
Another stall holder, who didn't want to be named, suggested the market should be open for longer in the summer, and shut earlier in the winter.
"On a day like today, you can't blame people for leaving early," he said. "We're not robots, we're going to feel cold."
The council's review will look at support available for the markets it runs across the district, including South Elmsall, Pontefract, Castleford, Normanton, Ossett and Wakefield.