Reuters/Shannon Stapleton
Ullman, who was also Johnson's predecessor, was just asked to return to the job for the first time over the weekend, according to Cotten Timberlake and Matt Townsend at Bloomberg News.
JCPenney announced Monday that Ullman would be taking over for Johnson, who had a disastrous 16-month reign.
The short notice suggests that Ullman won't stay at the post long, according to Bloomberg.
The company's SEC filing on the move revealed that he doesn't even have an employment contract.
But it also suggests that Ullman wasn't JCPenney's first choice for the job, and that the decision to hire him was hasty.
JCPenney's desire to take the company in a new direction was made clear last Friday, when Bill Ackman, the hedge fund investor who recruited Johnson, called his tenure a "disaster."
The comment cast even more doubt on his ability to execute a turnaround.
With JCPenney's future more uncertain than ever, speculation is swirling that the company could go private soon.
"You're already worried about what's going on there, you have new strategies that are in motion, you're still remodeling stores, and employee morale is even worse than ever," Brian Sozzi, CEO and chief equities strategist at Belus Capital, told us. "If they intended to stay public, it would make more sense to at least wait for Johnson to finish out parts of his strategy."