The new two-million square foot facility is being built off Newmarket Lane in Stanley, but those living nearby say the area is permanently covered in a film of dust.
After promising discussions with the contractors, residents say they their concerns are now being ignored.
One resident, who did not want to be named, said: "It's constant, we've had it for too long, and everything is covered in dust.
"There's been lot of promises but they haven't delivered anything.
"It's horrendous. People are getting rich from this at our expense.
"We came here to be surrounded by greenery, but we can't even sit in our garden. Everything is just filthy."
Planning permission was given for the warehouse in April of last year and will be occupied by a logistics firm.
It is expected to be around half the size of the NewCold warehouse, which sits on the south side of Newmarket Lane.
The new building is on the north side of the lane, close to the M62, but is closer to the row of terraced homes.
As a result, they say that hundreds of HGVs are arriving and leaving daily and they are unable to leave doors and windows open.
Cars parked outside their homes are covered in dust daily, with ISG offering to reimburse the owners for car cleaning once a fortnight, if they provide a receipt.
They also fear the value of their homes has plummeted.
The resident said: "Nobody would buy our houses now.
"I think we've lost about £100,000 off our homes and we're at the end of our tether."
But contractors ISG say their engagement with the residents has been "called out by residents themselves as extremely good".
A spokesman for the company said: "Neighbours have been informed at every stage of our project through frequent resident newsletters.
"Over the period of the project, which commenced mid-April 2021, we have had extensive dust mitigation measures in place, including dust monitoring equipment, misters and wheel wash facilities in operation and have always worked in partnership with our neighbours to respond quickly to any issues raised."
ISG say they are a founder member of the Considerate Constructors Scheme - an organisation dedicated to minimising the impact of construction to neighbours.