Living in the city is also pricey, which makes it even harder for the 20% of residents who live below the poverty line to get by.

4. Riverside/San Bernardino

Stress factor: Unemployment, long commutes

Zen factor: Shorter work hours

Once a booming area for the construction industry, the recession hit the cities of Riverside and San Bernardino especially hard.

"It took us longer to bounce out of that recession than others around the nation," said Larry Vaupel, economic development manager for the city of Riverside. "We had to retool."

In fact, the city of San Bernardino is currently working its way through federal bankruptcy protection, which it filed for in 2012.

And many of the area's residents are still struggling to recover. Nearly 20% of the metro area's residents live in poverty and the local unemployment rate remains several percentage points above the national average at more than 8%.

Those who have jobs face long commutes, with roughly 20% of residents traveling to Los Angeles or Orange County for work. Commuters spend just over an hour on the road each day, on average.

Things are starting to look up: employment now exceeds the pre-recession years and the construction industry is starting to make a comeback, said Vaupel. And he noted that houses in the area are more affordable than many other Southern California cities.

5. Houston

Stress factor: Longer work hours and commutes

Zen factor: Plenty of jobs, low cost of living

Workers in Houston give even New Yorkers a run for their money when it comes to working long hours.

Known for its many oil jobs -- which can require 80-plus hour weeks in the oilfields -- the metro area has the longest average workweek of all 55 places CNNMoney analyzed.

Getting to work can also be a major frustration due to traffic congestion. Houston's massive sprawl means that residents are often forced to brave the roads.

Blame it on the booming economy. Close to 400,000 jobs have been added to the area in the past four years, said Patrick Jankowski, vice president of research at the Greater Houston Partnership.

"Yes there is traffic," said Jankowski, whose organization is looking into ways to improve traffic conditions. "But on the other hand the economy is doing really well here so no wonder there is the traffic and the stress."

So what keeps Houstonians going? Jobs and money.

Low unemployment and a moderate cost-of-living means most people can easily afford to live here.

Plus, all those work hours mean many workers are earning overtime or higher salaries, Jankowski said. "Some people want the opportunity to earn the extra money," he said. "Sometimes the stress is by choice."

6. Chicago

Stress factor: Bad weather, heavy traffic

Zen factor: Low property crime rate

Brutal winters aren't the only thing raising Chicago residents' stress levels.