Almost one in four (23%) did not take a break in the past 12 months, compared to just 10% in 2011, the poll of 2,003 adults showed.
Pressure on household finances appears to be the key factor in the decline, with the biggest reduction in holidays coming from those with the tightest budgets.
In 2011 some 81% of those classified in the lowest social grade - such as low earners, the unemployed and people relying on the state pension - took a holiday, but fewer than half (49%) will take a break this year.
Despite this, the average number of trips taken by Britons in the past year was 3.2, up from 3.0 in 2014.
This rise is being fuelled by affluent people going on more holidays.
Those in the highest social grade took an average of eight holidays in the past year, compared to 7.4 four years ago.
Londoners are the most prolific holidaymakers in the UK, averaging 4.4 a year, followed by people in the North West (3.5), Yorkshire (3.1) and the West Midlands (3.1).
Northern Irish residents take the fewest holidays at 1.8 per year, behind the South West (2.5) and East Anglia (2.5).
The survey results, forming Abta’s consumer holiday trends report 2015, were announced in the Peloponnese, southern Greece, where the travel organisation is holding its annual convention.
Abta chief executive Mark Tanzer commented: “The travel industry has seen a steady return to growth in recent years following the credit crunch and worldwide economic crisis.
“It is very encouraging to see optimism and growth with many people choosing to take more holidays, particularly longer overseas breaks.
“However, some people are clearly still feeling the pinch and are either cutting back on the number of holidays, or not taking a holiday at all.
“This suggests the long tail of the recession is still having an impact on people’s ability to afford at least one holiday a year.”
The overseas market appears to have benefited most from the increase in holidays per person.
Britons took an average of 1.5 foreign holidays in the 12 months to August, compared to 1.2 in the previous year.
This could be partly due to the strength of the pound against a range of foreign currencies, in particular the euro.
Meanwhile domestic breaks fell from 1.8 to 1.7 over the same period.
Abta said this latest figure represented a market correction and was still higher than before the financial crisis hit in 2007/8.
The poll also showed that people were more optimistic about their future holiday prospects, with almost a quarter (23%) saying they will spend more money on their holidays next year, compared to just 15% intending to spend less.
A city break remains the UK’s favourite type of holiday and its popularity is increasing.
Some 54% of people took one in the 12 months to August, up from 42% last year.
Beach holidays were a close second, rising from 38% to 50%.