Banks are at their most pessimistic about the future since the end of the financial crisis
A survey of 103 financial services firms by business group CBI and accounting firm PwC found that nine out of 10 cited Brexit as their main challenge.
Overall, 10% of firms said they were more optimistic about the overall business situation compared with the previous quarter, while 45% were less optimistic, giving a balance of -35%, which is the lowest score in eight years.
Andrew Kail, head of financial services at PwC, said: "Financial services companies face many challenges to their business models from competition, regulation, technology and Brexit and, as a consequence, are having to take some big decisions about their future strategy.
"While companies are relatively positive about short term business volumes and profitability, they continue to need to make significant investments to protect their future," said Kail.
"The first quarter of 2017 and beyond will see many start to fine tune and activate their Brexit contingency plans as the reality of life outside the single market and the EU begins to dawn."
Last week Prime Minister Theresa May confirmed that Britain plans to leave the single market as part of its withdrawal from the European Union. The prime minister told an audience of foreign diplomats and ambassadors that she would terminate Britain's membership of the free-trade area to have full control over immigration from the European Union.
In the days after May's speech, HSBC, JPMorgan, and UBS all warned about job relocations, and there were reports that Goldman Sachs planned to shift hundreds of bankers out of London.
The Brexit effect hasn't hit quite yet, with May still yet to trigger Article 50 and start the formal talks to take Britain out of the European Union.
Around a fifth of firms said that business volumes were up, while 17% said they were down, giving a balance of +2%, which compares with +34% in September.