As a result, luxury brands like Tiffany & Co. have been struggling.
"We've seen such a slow down in the high end," Brian Yarbrough, analyst at Edward Jones told Business Insider.
While lower-end stores, like Ross and TJ Maxx are performing well, Tiffany's comparable sales declined 9% in the most recent quarter. Luxury department store Nordstrom has also been struggling.
At first glance, the weakness in luxury is puzzling because consumer confidence is high and Tiffany's customers still have discretionary income to spend.
But Yarbrough points out that people are increasingly spending on categories other than luxury
"People are still willing to spend on auto, and they're still willing to spend on their homes which are bigger-ticket items," he said.
"They're more interested in cars or homes or experiences," he said, pointing to movies and traveling as examples - not apparel.
An analysis by Jharonne Martis, director of consumer research for Thomson Reuters, confirmed that people were spending their money on hotels and casinos, too, Business Insider's Hayley Peterson reported earlier this month.
Thomson Reuters
But there's hope for apparel retailers: this trend may not last forever.
"Typically, you're seeing a lot bigger growth in durables or no durables," Yarbrough said, but noting that "at some point that will stop," once consumers take on the debt from these larger-ticket items.
Until then, the apparel market may be dismal.
Tiffany & Co. has tried to rectify this problem with newer, more fashion-driven collections, as Neil Saunders, CEO of the consulting firm Conlumino, pointed out in a note to clients.
The company is blaming a variety of factors for its downturn, and it says it is working to fix its problems.
"As expected, this was a difficult quarter in terms of both sales and earnings growth," CEO Frederic Cumena said in a release. "We faced numerous challenges, including continued pressure from foreign tourist spending in Europe, the US, and Asia, particularly in Hong Kong.
Yarbrough believes that these issues won't be abated this year, though he believes the company will eventually bring people back into its stores. In the meantime, however, things look grim.
"Their numbers have gotten worse and worse .... when I went back and looked, I don't think we've seen numbers this bad since the economic downturn," he said.
For now, Tiffany & Co. might just need to figure out how to appeal to value-conscious customers.