AP Images
The teen retailer announced Tuesday that Arthur Martinez, the former head of Sears, will be named non-executive chairman.
Jeffries has served as chairman since 1996. He will remain a director and the CEO of the company.
Abercrombie's stock rose 2% on the news in premarket trading.
The company also announced Tuesday that it has added Terry Burman and Charles Perrin to its board of directors.
Jeffries was criticized last year for saying he only wanted "cool kids" to wear the brand's clothes. The company has also drawn fire for failing to offer larger sizes for women. As a result of the backlash, Abercrombie announced it would start offering more sizes.
Abercrombie's U.S. sales fell 18% in the third quarter.