Poop from pets and infants created flaws in key science at uBiome, and now the buzzy startup is conducting an internal review of its foundational research
- The science behind uBiome's products is flawed, according to insiders and outside experts. They say that raises new questions about the company's future.
- Founded in 2012, buzzy startup uBiome raised $105 million from investors to explore the microbiome, a "forgotten organ."
- The FBI raided uBiome's headquarters in April, reportedly related to the company's billing practices. Then, the company's top executives departed.
- uBiome built a big set of data based on the human microbiome, but the data was flawed in ways that risk making uBiome's tests unreliable.
- After Business Insider contacted uBiome with these concerns, uBiome confirmed problems with the data and said it's conducting an internal review.
- The scientific journal that published the data is now investigating, too.
- Click here for more BI Prime stories.
When Elisabeth Bik, then the science editor at microbiome startup uBiome, told the company's CEOs that its primary dataset - an analysis of poop samples - was soiled with data that didn't belong, she was waved away, she said.
The dismissal irked Bik.
Leaving in the irrelevant data could undermine work that was then underway to create uBiome's first clinical test, Bik said. That test, called SmartGut, would become uBiome's biggest money-maker, according to 11 former employees. It was designed to tell people about the health of their guts and their risk of diseases like irritable bowel syndrome.
The value of uBiome, a buzzy Silicon Valley startup that raised $105 million from high-profile investors including OS Fund and 8VC, lies in its accumulated knowledge of the microbiome, the trillions of bacteria living in and on our bodies and believed to powerfully influence our health.
If that knowledge is misinformed, as Bik and six others told Business Insider, it raises another question about the company's prospects, after an FBI raid and the departures of top executives.
After Business Insider contacted uBiome with these concerns, uBiome confirmed on Tuesday that the dataset Bik described included data from 45 minors and at least one non-human. The company said it's continuing to review the data.
"Our goal going forward is full disclosure and transparency," Karthik Bhavaraju, uBiome's acting chief operating officer and a senior director at consulting firm Goldin Associates, said.
Mapping the human microbiome
Scientists have yet to paint a definitive picture of a healthy human gut microbiome. Doing so would be lucrative: research suggests these vast stores of internal bacteria can influence everything from our weight to our mental health.
At the start, uBiome's goal was to create such a picture - to illustrate what a healthy or normal gut microbiome might look like. So to do it, the company started an online fundraiser that enabled it to collect poop samples from thousands of volunteers.
At one point, people were allowed to send in samples of poop that belonged to pets and children as well as adults, but three ex-employees said that uBiome decided ultimately against using that data. Then, uBiome analyzed the samples in a lab, and used the analysis to pinpoint which bugs (and how many of each) could be reliably found in the human gut.
Initially, uBiome used this analysis - which it published later in a prominent science journal - to sell a casual, $89 test called the Explorer, according to 21 former uBiome employees.
Someone who took uBiome's Explorer test sampled their poop at home, sent it to uBiome, and got back an online report comparing the bacteria in their guts to that of the other uBiome volunteers in the company's database.
If a customer's levels were on par with the results in the database, the report told them so, and gave them a "wellness score" out of 100. If their levels diverged from the database, the report noted it, and suggested a possible link to illness.
In fine print at the bottom of each report, it read, "These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure, or prevent any disease."
Scientists and clinicians have questioned the utility of such a test.
Rusha Modi, a gastroenterologist at the University of Southern California, previously told Business Insider that uBiome's approach has some important limitations. For example, he said it simply compares one person's results against those of other uBiome volunteers, not against the wider population.
Also, the results only provide a one-time glimpse at the microbiome. The bacteria in our guts can change dramatically, sometimes in as few as 24 hours - meaning a problem that shows up in one test might disappear on another test taken days later.
"Given the rapid variability in the microbiome, single samples are simply not that informative," Kevin Honaker, the cofounder and CEO of a microbiome research startup called BiomeSense, told Business Insider.
'It's like going to a really expensive homeopathic doctor'
Around 2016, uBiome started making plans to sell a clinical test called SmartGut. The new test was designed for adults, according to uBiome's website, but relied on the same volunteer database that powered the Explorer test, according to 21 former employees.
Only now, if the data in the analysis was off, it could have clinical repercussions: healthy people might be told they were sick, while sick people might be told they were healthy. Adult patients who took the SmartGut test on a doctor's recommendation might now use it to make medical decisions.
"With SmartGut, you and your healthcare provider can gain valuable insights to better understand what's going on inside your gut, then take steps for you to feel better," uBiome's website read.
The site continued, "SmartGut is the world's first sequencing-based clinical microbiome screening test based on our patented technology and extensive peer-reviewed research."
"The test detects beneficial and pathogenic microorganisms associated with gut conditions like irritable bowel syndrome (IBS), and inflammatory bowel disease (IBD), including ulcerative colitis and Crohn's Disease," the website said.
Seven people involved in the research, including Bik, told Business Insider that there were several problems with the research uBiome used to build its microbiome database.
Consider one issue, they said: poop samples from infants and pets had likely been included in the data, which was then used to create the adults-only test. That likely occurred because people sent in specimens from their kids and their pets, rather than their own stool. uBiome could have removed the misfitting data, according to three ex-employees, but leadership at the company chose not to do so, they said.
Bik added that while the inclusion of babies and pets shouldn't have a huge impact on what the reports tell people about their disease risk, they would nonetheless skew someone's results. Three former employees said that in January 2018, the company made an attempt to clean up the data by removing the misfitting data.
The data was vulnerable to other human errors too, such as customers being sick or having taken drugs that would skew the makeup of their microbiome.
Two former uBiome employees who handled the data said that while removing the infant data would have been easy, they had a hard time filtering out data from people who'd taken antibiotics, for example. People who'd taken the drugs would often fail to disclose they'd done so, they said, because they believed enough time had passed for their effects to disappear. But antibiotics are thought to impact the microbiome for up to a year.
One ex-employee who worked on the science likened taking a uBiome test to pseudoscience.
"It's like going to a really expensive homeopathic doctor," the person said.
The questions that insiders are now raising about uBiome's scientific work add a new problem to the list of obstacles confronting the company as it works to rebound from a troubled stretch.
An FBI raid, executive departures, and layoffs
In April 2019, the FBI raided uBiome's San Francisco headquarters as part of an investigation into its billing practices, according to the Wall Street Journal.
On the heels of the raid, uBiome stopped selling its two clinical products, leaving only one test - the Explorer - for sale. Then the company's cofounders and co-CEOs resigned, along with John Rakow, the company's chief legal counsel and interim leader.
In June, consulting firm Goldin Associates, brought on by uBiome's board to steer the ship in their stead, put three new leaders in charge. The following month, uBiome laid off half its staff.
"Though the developments of the past few months have been unexpected and disappointing, we continue to believe in the underlying value of uBiome's technology at a time of growing demand in the market," uBiome's board said in a statement when the Goldin executives took over.
But the new revelations raise questions about the value of uBiome's technology and the Explorer test.
Jack Gilbert, a professor of pediatrics at the University of California, San Diego and a group leader in microbiology at Argonne National Laboratory, told Business Insider that he believes uBiome's approach is fundamentally flawed.
The data uBiome compiled is highly subject to human error because it was gathered from people's homes, as opposed to as part of a clinical study, he said. Also, the test simply tells people their comparative bacteria levels, he added, and as such should not be used to influence medical decisions.
"It's complete crap," Gilbert said. "It's ridiculous."
Goldin's Karthik Bhavaraju and Eduardo Morales, uBiome's vice president of research and development, told Business Insider that they stand behind the Explorer product. They added that Explorer would be the company's focus going forward.
Business Insider provided a summary of the reporting in this article to a spokesman for uBiome cofounders Jessica Richman and Zachary Apte.
The spokesman said in part: "Business Insider's unsubstantiated allegations appear to come from disgruntled former employees."
The spokesman added: "Because Business Insider has not provided Dr. Apte or Dr. Richman with any data analysis, statements by other authors on the peer-reviewed scientific paper, or specific concerns from other experts in the field there is no way to know whether their assertions are based on real data or simply idle gossip and guesswork."
After Business Insider provided the statements from Bhavaraju and Morales, the company's head of research and development, the spokesman for Apte and Richman declined to comment further.
uBiome's 'gold standard' includes data that belongs to babies and pets
Bik's complaints came too late, she was told. By the time she was brought onto the company full-time in October of 2016 after freelancing for the company, uBiome had already submitted the paper to the science journal PLOS One.
In May 2017, the journal published the paper. It was authored by nine uBiome employees including Bik and co-CEOs Richman and Apte, and identified 28 common types of gut bacteria, along with their relative amounts. The analysis, based on samples from 897 people, supported a medical test that would help "assist in the clinical diagnosis of certain health conditions," the authors wrote in the paper.
The test was SmartGut. Shortly after the paper was published, Richman began talking about it at conferences and in interviews.
In a 2017 keynote at a Harvard conference, Richman said uBiome's scientific approach to creating SmartGut was ground-breaking and relied on top-notch research methods.
"There just aren't gold standards, so we had to set them," she said, adding, "we pulled out people that were healthy by a set of very rigorous criteria."
uBiome's data collection relied on the honor system, however:
"Fecal samples were self-collected by participants at home using commercially available uBiome microbiome sampling kits," the researchers wrote.
In other words, people sampled their poop at home, in private, and then answered a series of key questions that included asking whether they were healthy or sick.
If the answers were wrong or incomplete, it could provide a distorted sketch of the microbiome, insiders and outside experts have said. Volunteers could have been sick at the time they submitted their samples. They could also have taken antibiotics, which can distort the microbiome. And they could have sent in poop samples that didn't belong to them.
To be sure, plenty of research relies on self-reported data, and scientists acknowledge its potential limitations.
In this case, as many as 20 of the samples in the dataset likely belonged to infants, Bik and two other former employees who spoke to Business Insider anonymously out of fear of retribution said. The three ex-employees said the babies were easily identifiable based on the high presence in their samples of Bifidobacteria, a strain of microbe that's specially suited to breaking down the ingredients in breast milk.
Another handful of the samples likely belong to people's pets, said Bik and two of the other ex-employees, pointing to data in the paper which shows strains of bacteria at levels not found in humans.
The insiders were hesitant to speak publicly, they said, because of the ongoing investigation and out of fear that they might be reprimanded by uBiome and not receive their severance pay.
On Tuesday, Morales and Bhavaraju told Business Insider that they'd re-run the analysis and found data belonging to 45 minors and at least one non-human.
In an emailed response to Business Insider, a spokesperson for the journal PLOS One said that it was not aware of the concerns with the paper and would look into them.
"We are deeply committed to the integrity of the published record, and follow up on all concerns raised to us per the guidelines set forth by the Committee on Publication Ethics, of which PLOS One is a member," they said. The journal later said it couldn't comment further until its investigation is complete.
Efforts to clean up the data
Bik resigned from uBiome in December 2018 when she felt like its values no longer squared with hers. A molecular biologist and former Stanford researcher, Bik also grew tired of having her questions dismissed by uBiome's leadership, she said.
Two former uBiome employees told Business Insider that in January 2018, there were efforts to fix some of the problems in the data, including removing the babies and pets. Currently, uBiome's website says SmartGut is based on 865 samples, rather than the 897 described in the published paper. But the new data hasn't been published.
On Tuesday, Morales said that efforts to clean up the data did take place in January 2018, but he said the number of samples had been changed to 867, not 865. He declined to explain the discrepancy.
Despite the objections from Bik and at least two others, uBiome leadership held up the published 2017 paper to tout the company's SmartGut product. In talks, conference appearances, and on social media, Richman and Apte frequently referred to the science behind the kit.
"We now have this clinical test, which is truly a medical product," Richman added. "Without the data, this test just doesn't exist."