Warren Buffett will decide when it's time for changes at American Express

American Express needs a change.

Shares of the credit card giant have lost about a third of their value over the last year as investors steadily lose faith in the company's direction and its long-tenured CEO Ken Chenault.

On Thursday afternoon American Express reported earnings that disappointed and on Friday shares of the company were down as much as 12%. Analyst commentary following the news was decidedly downbeat.

But perhaps the biggest development was news that activist hedge fund manager Jeff Ubben, who runs ValueAct Capital, has sold his entire position in American Express.

Ubben, who is known for taking big stakes and big companies and looking for management shakeups, took a $1 billion stake in American Express in August 2015. At the time, The Wall Street Journal reported that Ubben had spoken with Chenault but that no specific demands had been made.

And Given that Ubben is done with his investment, it's clear any discussions regarding specific proposals were going nowhere. The simple reason, of course, why this was always the most likely outcome is because changes at the very top of American Express' management team are going to come from one person and one person only: Warren Buffett.

Buffett, who is American Express' largest shareholder and owns more than 15% of the company, has long been clear and public about his philosophy behind investing, specifically as it relates to management. (Disclosure: I own a few Berkshire Hathaway shares so that I can go to the annual meeting. I'm not planning to buy more or sell any stock anytime soon.)

In short, Buffett chooses to let his portfolio companies run themselves autonomously, making decisions about capital allocation and what to do with the earnings these companies send back to the parent company Berkshire Hathaway. And CEO changes, when they do happen, will only be made by Buffett.

And while Berkshire "only" owns 15% of American Express, it is unlikely the company's board or a majority of shareholders would move to shake up the company without Buffett's endorsement or support.

Over at ValueWalk, author Activist Stocks wrote a great post about the challenges Ubben and ValueAct ultimately faced when dealing with what is essentially a Buffett-run outfit.

On one hand, you've got Buffett's long-held aversion to the style of activist investors, with Buffett once saying, among other things, that activists just want a "quick hit." Of course, this doesn't necessarily capture ValueAct's intentions with American Express nor does it accurately describe all activist investors.

But it does illustrate the fundamental split Buffett sees in the investing world and which side he stands on.

Buffett is a champion of entrenched management and operating models and is loathe to make or agitate for changes, particularly those that are perceived to be in response to a falling stock price. In fact, the Buffett philosophy says you should probably prefer the stock price of any company you own to come down so you can acquire more of that company's shares.

And so the other side of Ubben's challenge with getting through changes at American Express simply comes down to Buffett's long-state preference for not making changes.

Additionally, Activist Stocks made the astute point that in announcing $1 billion worth of restructuring and investment programs American Express basically gave ValueAct all it was ever going to get from management.

In a way, the activists won.

As Activist Stocks noted on Friday, Buffett's right-hand man Charlie Munger said at the 2015 Berkshire annual meeting that American Express' path to "prosperity" doesn't look as easy now as it once did. The basic problem is that American Express' "moat" - you can think of this as any company's "margin for error" - has been eroded over time.

But with a roughly $8.5 billion stake acquired for around $1.4 billion, Buffett still has a big winner on his American Express investment. Which, whether you think it's right or not, gives him the ability to wait - perhaps a lot longer - before urging change at the company.

In his most recent letter to shareholders, Buffett reiterated his view on American Express - as well as Wells Fargo, Coca-Cola, and IBM, which he calls his "Big Four" stock holdings - writing:

These four investees possess excellent businesses and are run by managers who are both talented and shareholder-oriented. At Berkshire, we much prefer owning a non-controlling but substantial portion of a wonderful company to owning 100% of a so-so business. It's better to have a partial interest in the Hope Diamond than to own all of a rhinestone.

There's an old market maxim that the market can remain irrational longer than you can remain solvent. And a similar idea seems to apply with American Express.

Something, then, is going to need to change at American Express. But only when Buffett decides it's time.