Warren Buffett thinks this is Berkshire Hathaway's biggest problem

Berkshire Hathaway has been a successful company, to put it mildly.

Over the years Warren Buffett's and Charlie Munger's firm has had a bit of a golden touch, but that may also be its downfall according to its CEO.

"Size is the enemy of performance," said Buffett at the company's annual meting Saturday.

Said another way, the biggest danger for the company, according to Buffett, is too much success.

Earlier in the day, Buffett remarked on the company's recent propensity to acquire firms that require a lot of capital expenditures to run. Early in his career, Buffett was a staunch advocate of only investing in companies that didn't have a lot of fixed costs, but that has since changed.

"It's part of the problems of prosperity," said Buffett. "We'd love to buy a company for $10 or $20 or $30 billion that is not capital intensive. And we may. But that is harder."

Recently, Berkshire purchased capital intensive businesses such as Precision Castparts, which makes airplane parts, and BNSF, a railroad giant.

The upshot here is that as Berkshire has grown so too has its cash pile, eventually forcing the company to make bigger acquisitions and Buffett to stick to investments in ever-bigger companies.

Also on the topic of capital allocation, Buffett and Munger decried the growing number of share repurchases being executed across the corporate landscape.

And Buffett noted that he only likes to do purchases of Berkshire shares at 1.2x book value.

But Buffett did say this "trigger" could be moved up he found the company with so much cash and so few opportunities that repurchasing shares at a higher level was the best use of this cash.