"That place is such a brain trust," he says. "It wasn't a workforce that I was normally subjected to - you don't work with a lot of Stanford graduates in the kitchen."
He said the hyper-intelligent employees seemed intimidating at first, but they were friendly and good listeners. The personal connections he made with many of them would be a catalyst for new opportunities throughout the rest of his career, he says.
For example, he developed a really great rapport and friendship with one of Google's earliest investors, Ram Shriram, who loved how Ayers whipped up atypical vegetarian meals. When Ayers eventually left Google in 2005, Shriram called him up and got him food consultant positions at a handful of tech companies that he worked with. Shriram kick-started Ayers' consultant career: He has since worked with companies like Dropbox, Zazzle, Facebook, Palantir, and Nest (which Google bought this year for $3.2 billion).
Ayers says that several former employees also invested in his Palo Alto restaurant, called Calafia Cafe & Market A-Go-Go, which he opened in 2009. Another paid for him to have a life coach.
Ayers says he was also impressed with how willing Google was to invest in the local culture and community, even in the early days. During the recession of the early 2000s, Google kept several local food businesses afloat by using them as vendors for its cafeterias, he says.
By the time Ayers left Google in 2005, he had built a team of five chefs and 150 other kitchen workers, and they were serving 4,000 lunches and dinners a day. Ayers wrote a cookbook with tips and recipes based on his experience and told Business Insider that he still occasionally works for Google in an advisory role. His legacy also lives on at the Googleplex: Charlie's Cafe dishes out a wide variety of healthy cuisines.