Don Duet, who ran the division alongside Walker, will become its sole head, according to an internal memo seen by Business Insider.
Walker, 45, will stay on as an advisory director to the bank.
People who know Walker say that he's admired for the way he engages with young employees of the firm, and for his smarts. He is known as a mentor to young engineers and holds office hours for them.
"During his tenure at Goldman Sachs, Paul has played a critical role across a number of our businesses, leading many of the Technology Division's initiatives and most recently driving our engineering strategy from concept to implementation," chief information officer Marty Chavez wrote in the memo.
At its core, Goldman Sachs is still a financial-services company. For the bank, investing in technology that adds services or makes its employees more efficient, has become an important business opportunity.
Goldman Sachs leadership has begun to frequently refer to the firm as a "technology company."
Walker joined Goldman Sachs in 2001 as a vice president in FICC strategies. He was named managing director in 2004 and made partner in 2008. In 2013, Walker was named co-head of the technology division around the time that Marty Chavez was tapped to assume the role of chief information officer.
Walker is also a member of Goldman Sachs's Partnership Committee and the Firmwide Technology Risk Committee.