And now Barclays is adding its name to the list of doubters, calling the top in a note sent to clients this week.
The bank downgraded two challenger banks, OneSavings Bank and Paragon, both of which have big buy-to-let mortgage books. Barclays says it thinks new rules to reign in the sector could be a "game-changer" and bring and end to the good times for lenders in the market.
A bubble?
The number of mortgages issued to people buying a house to rent out has rocketed in recent years to pre-crisis levels. This is in part down to rocketing house prices and record low interest rates.
Bank of England
But Mark Carney, the governor of the Bank of England, signalled in a recent interview that he's keeping a close eye on the sector over concerns that investors could all seek to sell at the same time if there were a general decline in house prices. That could cause a stampede and send house prices diving.
The problem with BTL mortgages is that their interest rates move higher if the Bank of England's raises its rate - something that's looking increasingly likely.
Lending requirements for BTL mortgages also aren't as strict as regular mortgages, meaning if and when rates rise those paying BTL mortgages could be more likely to struggle. All those factors add repayment risks for the banks who hand them out.
Meanwhile, George Osborne earlier this year added pressure on homeowners in the sector, increasing taxes on buy-to-let properties and cutting tax incentives.
The government announced on Thursday that the Bank of England would be given powers to regulate the buy-to-let lending market.
'Wall of worry'
All this contributes to the "wall of worry" around the sector, Barclays analyst Daniel Garrod and his team say, but the "game-changer" is new capital requirement rules for the buy-to-let sector coming out of the Basel Committee, a group of top central bank regulators.
Here's Barclays:
On 10 Dec the Basel Committee surprised the market by recommending a strong hike in standardized risk weights for Buy-To-Let Mortgages. BTL risk weights are proposed to increase from their current ~35% to 70% for <60% LTV [loan to value], 90% for 60-80% LTV and > 120% for LTV over 80%.
Loan-to-value is the ratio of money the bank is lending someone to buy a house versus how much they're putting in personally.
If you borrow £80,000 to buy a £100,000 house, the LTV ratio is 80%. The higher the percentage, the higher the risk for the bank, as if the value of the house declines or the loan goes sour they're on the hook for more of the losses.
Banks are required to set aside cash to cover this risk for each loan it makes and the big jumps in the amount required proposed by Basel would mean they have less money to lend.
That will have a knock-on effect for earnings and could bring an end to the good run of form challenger bank stocks have enjoyed.
Barclays says:
Whilst it is likely that the Basel proposals may be watered down, we believe some incremental capital against BTL [buy-to-let] is increasingly probable, reducing future ROEs [return on equity].