[03:11] Neil’s Background
[05:32] How Labdoor Does Its Work
[07:29] The Kind of Things Labdoor Finds In Supplements
[08:48] The Labdoor Operation
[11:15] How the Information Goes To the Website
[13:07] How the Supplements Are Ranked
[17:04] What Labdoor Takes Into Consideration When Ranking
[18:33] Fish Oil
[21:00] Energy Drinks and Gels
[25:52] The Supplement Industry’s Good Side
[29:35] End of Podcast
Ben: Hey, everybody. It’s Ben Greenfield. And I was just actually, before we even began recording, having a fascinating discussion with today’s guest about how your protein powder may not actually have in it what you think it has in it. And while it may not be crushed up beetles and bugs, unless you’re using that brand new-fangled cricket protein stuff, there may be some issues with your protein. So before we even jump into Neil Thanedar, my guest’s story and what it is that he does, Neil what’s going on with protein powder, man?
Neil: Alright, so there’s something called amino acids spiking. And so what you would expect when you see one of those fancy products on the shelves at your local supplement store is 100% whey protein isolate. And what we’re seeing is that a lot of products don’t actually live up to those claims. Many of them have maybe 30, 40, 50% protein, and the rest of it are amino acids. And so if you had a free amino acid like taurine, or glycine, or a glutamine added to your product, you’re actually only getting, in terms of a full spectrum of amino acids that you really need, only 30 to 40% of what you think you’re getting.
Ben: Is that ‘cause that’s just like cheaper to make that way or is that just poor quality control?
Neil: It’s many times cheaper, and it’s actually due to a very interesting FDA loophole. So the FDA actually only requires that manufacturers test the nitrogen content of their products. And so to a nitrogen test, amino acids or 100% pure whey protein isolate look the exact same. But to a more complex test, which is the kind of testing that Labdoor runs, we can actually see the difference between the two.
Ben: Interesting. Okay, cool. And before you guys run for the hills thinking that Neil is a protein powder salesman and he’s about to tell you, “But wait, there must be a better way,” he’s not a protein powder salesman. He actually runs a fascinating website that tests supplements for their quality. It’s basically a totally unbiased website that’s doing some really cool thing. So it’s called Labdoor and it’s over at labdoor.com, and Neil’s going to fill us in about what that is and what they do because it’s actually a pretty cool resource for you. But before we jump into that Neil, what’s your story, man? What’s your background? Were you a biology major back in college or something like that?
Neil: Yep. So it’s actually chemistry and molecular biology. So I actually grew up, and my dad ran testing labs my whole life. And so growing up, that’s what I thought I wanted to do and I just loved the science behind these kinds of analytical laboratories. So you basically could take any product, put it in the lab, and break it down to all of its components. From the best active ingredients, all the way down to the smallest parts per million, parts per billion contaminant. And so when I got out of school, that’s what I knew how to do. So I started a laboratory and we did basically a lot of work for pharmaceutical companies. We became an FDA registered lab, and that meant we were able to do the method development, stability testing for these companies. And so we would be the ones who actually put that expiration date on your product. We knew whether product needed to be pulled off the shelves and retested. So we were actually qualified to do all of that testing.
And what we actually found out while running that lab is that supplement companies aren’t put up to the same standards. But what’s happening is the manufacturers themselves, they wanted either themselves, or much more likely they wanted their competitors to be set up to these standards. And so they would buy all of their competitors’ products, ship them to our lab and say, “Tell us what’s wrong with all of these products.” And so we would and we’d find all these problems. We would find energy drinks with caffeine levels well over 200 milligrams in a two ounce shot. We found these protein powders that had high amounts of lead in them. We found fish oil products that had less than half as much omega-3’s as they claimed. And we just had to ship that information out to the manufacturer that paid for it. And so what I really wanted to do is say, “Let’s turn this business model on its head and say, ‘Let’s have consumers pay for it. Let’s have the consumers drive what’s tested and let’s have them get the information delivered directly to them.'” And that’s what Labdoor is.
Ben: Gotcha. Okay, cool. So at Labdoor, exactly what happens? Are you guys going out and walking into, whatever, Walmart and Costco and just grabbing this stuff off the shelf, taking it back, and testing it? What kind of your protocol or the wringer that you would put a supplement through when you get your hands on it?
Neil: Yup. So the first thing we would do is actually buy a product directly from a retail site. And that’s really important to us. I think a lot of places, manufacturers want us to kind of handpick a sample and send it to a lab, but we want to get exactly the same way that…
Ben: Wait. A manufacturer of a supplement, if they want to have their supplement checked out by a lab, it’s not the lab that gets to pick it? A manufacturer can just pick some off the, whatever they want and send it into the lab to test?
Neil: Exactly. And in some cases, they test it in very special ways. So for fish oil, they’ll do a vacuum sealed chilled package directly to the lab in just perfect conditions. And we don’t want to do that. We know that these products are…
Ben: Versus sending it to somebody’s house via UPS taking two weeks and having it sit in a hot box for 14 days.
Neil: Of course. And so that’s really what people are taking. And so we want to make sure that the stability lasts that long. And so that’s the first thing we do. And so we buy all these products, we bring it in, and we actually do all of the work here. We’ll actually take pictures of all of the products. So everything on the site, we actually do that ourselves. And then we get it into an FDA registered lab, and that’s where kind of, really, fun begins where we’re starting to break down these products. We look at every active ingredient and then we look for really key purity concerns. So if it’s a fish oil, mercury and PCB are big issues. If you’re looking at a creatine product, we can look at creatinine and another kind of breakdown products of creatine. And so finding all of the stuff that’s supposed to be on that label, make sure that that’s accurate and then making sure that the product is as pure as it claims.
Ben: Okay. So tell me about what kind of stuff you find in supplements. We’re talking about contaminants and things like that just because I think a lot of people maybe, it’s kind of woo-woo when you say maybe something is contaminated or whatever, but what exactly are we talking about in terms of chemicals you see that you might not want to see.
Neil: So it’s something as simple as a protein powder that has too much sodium, two or three times what it says on the label. Or it could be something really as complex as lead or mercury…
Ben: Why would that be an issue? Why would two to three times as much sodium be an issue?
Neil: So one of the things that we’re finding the protein is that if you’re starting to look for gains, if you’re trying to gain muscle, you’re trying to put on a little bit of good mass, you may think that you’re actually making a gain when you actually could just be getting bloated by a lot of excess sodium. And so that’s something that we’ve seen in muscle [0:08:04] ______ products, for example.
Ben: So it may not be, like if you have a bad reaction, like a whey protein, it might not be the whey protein, it might be that it just has crap loads of extra sodium in it?
Neil: You may actually think that you’re gaining weight. You may actually feel like you’ve actually put on a little bit of mass, but it could not actually be that case.
Ben: Was that doorbell somebody delivering supplements to your door?
Neil: It probably was. We’re trying to get up to basically 50 to 100 products per month tested in our facility.
Ben: Wow. So how many people are there? I mean is this just like this huge lab where you guys are constantly testing supplements? What kind of an operation is this?
Neil: So we actually have seven of us in San Francisco that are doing all of the technology work. So the web and mobile applications. And then we basically got the ability to use the lab that I founded in Michigan that continues to do a lot of the work for us. And so that’s about a 25% facility, but not all of them obviously are doing all of our testing all the time.
Ben: Okay. I want to hear what you’re doing with all of this information that you’re getting. But first before that, is this just a huge labor of love? How can you make money doing this, just testing supplements?
Neil: Absolutely. So that the two things that we’ve been doing, kind of the key thing is for consumers. We really followed the Open Table-style Model. And so if you look at Open Table, they’re just the best destination for all of this information about these restaurants. And then the way they make money is basically they have the ability to make a reservation on their site, and basically they make a percentage of that reservation. And we’ve basically got the exact same model. So you do all your research, we’ve got our scientific reviews, we’ve also got user reviews. And when you’re ready to make a purchase, we basically just have a “Buy It Now” button that right now just connects you to Amazon. It’ll eventually connect you to any retailer that you’re interested in, and then those retailers basically give us a percentage of those sales.
Ben: Okay. I see that now. Like you have a bunch of rankings here for energy, and fish oil, and protein. So if I went to research what was the best, I could get the product report on your site. And then if I wanted to buy it from there, it’s just kind of like an affiliate-based system basically.
Neil: And we really like that because it basically connects our business model directly to helping consumers. We’re the best, most trusted resource. You’re actually going to buy through us and we’re able to get a percentage on that.
Ben: Yeah. I like it. It’s kind of like, I don’t know if you’ve seen examine.com, but they’re a really good source for kind of finding out the actual research, the peer-reviewed research behind certain supplements and supplement stacks. And that’s another website, it’s kind of similar. You can go there and do all your research, and it’s like you have an opportunity to get what it is that you’re looking for at this site as well. It’s very similar, but your website is way different in that you’re actually testing these products for contaminants. So once you test and you measure the active ingredient content and you identify these potential contaminants, then what happens? Like how does it go from that to your website?
Neil: Sure. So we actually built really early on in the process basically calculations and algorithms to take that information and turn it into a single grade in ranking. I think what we’ve always found out is that sometimes there’s so much information out in the supplement space and we didn’t want to be just one more place that shows up on your nightly news talking about contaminated supplements, but actually said something where we actually need to translate this into something that consumers can understand. And so basically we do all these calculations. We have an [0:11:45] ______ and then we rank each product in a category by both quality and value. And we see that actually consumers are pretty evenly mixed between the two. It’s basically almost 50-50 buy based on quality or value.
Ben: So people will buy something cheap even if they know that’s low quality and that they’re going to be eating it?
Neil: Well the thing that we kind of find out is something that, let’s say it’s the number four product on the quality side but it’s…
Ben: Okay. So they’re trying to get that optimum combination of quality and affordability?
Neil: Yup. And so that’s what we consider our value calculation. So if it’s the 4th best product, but it’s the 20th most expensive product, you’re actually getting really great value there.
Ben: Okay. So now I want to ask you some scary questions ’cause I’m on your website and see you’ve got things ranked from A all the way down to D. I’ve got your multi-vitamins page up right now. By the way, it doesn’t look like you’ve analyzed the multi-vitamin I take yet, the Thorne one. So you guys need to get some of those in stock. I’d send some to you, but it doesn’t sound like that satisfies your criteria for receiving it. Okay. But you’ve got some on here that are rated D. What would make something like a multivitamin rank as a D with your protocol? What kind of stuff is in there?
Neil: So we would look at, in some of these cases we’re looking at preservatives, artificial colors. Those are definitely things that could be of an issue to us. We’re also looking at kind of a really…
Ben: I see that Centrum gets a D.
Neil: So some of it is preservatives, some of it is artificial colors. Some of it could just be inaccurate labels. So we’ve had products that were actually 90% below or 90% above their label claims for some of these ingredients. There’s a lot of reasons that could happen. One of kind of the simplest reasons it could happen is that the manufacturer just didn’t put enough of an ingredient in. But the other parts of it could be like a product, like a vitamin C for example. Vitamin C actually degrades over time. And so if you bought a product a year after it hit the lab, even if a manufacturer meant to put the product in there and they didn’t really kind of pay attention to stability, that vitamin C could be mostly or all gone by the time you actually take the product.
Ben: Wow. Okay. So I’m looking at proteins. You’ve got Muscle Milk as a D+. And I’ve always told people that Muscle Milk is not all that great for them, but I haven’t seen this before. You say on here it’s got 358% higher levels of sodium than what it says on their label?
Ben: Holy cow!
Neil: And amazing part about that, we’ve seen that the ready to drink products, those shakes that you get, those usually are worse offenders than even the powders. But in this case, the powder was bad as well.
Ben: I’ve been wary of those ever since I used to be sponsored by the bodybuilder, it was similar to Muscle Milk. I’m blanking on the name of it now. I was one of those Man-In-A-Can pre-mixed protein shakes. They would send me like cans and cans of it to my doorstep when I was bodybuilding and I would be crapping out a straw all week long and just be like red in the face, and have acne, and all this stuff. I got pretty swole, I put on a lot of muscle. But I’ve suspected since back in that day that that stuff is not all that great for you, the pre-mixed protein powders.
Neil: And for us, I think it’s something where you just don’t even notice. You may have a negative reaction to it. You can maybe guess, maybe you don’t ever take it again. But there’s always going to be that new generation of people who are going to take that product, and look at it on the shelf, and say, “Oh, that looks easy.” And that’s part of Labdoor’s job is say, for the first time, we’re going to publish all of this information and then you can do with it what you want.
Ben: Yeah. Okay. Vitamin D. You’ve vitamin D on here ranked from A down to D. What would make something like a vitamin D rank poorly?
Neil: So the vitamin D products were actually the best on average out of all of the categories. So the vitamin D is actually, I think 17 out of 20 of them actually were in the A range. And so many of the products were actually very good. I know that that category, I think what’s very interesting to us is we find that the simpler that the formulation is, the better the products do.
Ben: Yeah. Actually I see that now. Like the worst ranked vitamin D in your website only has a C+.
Neil: Yup. And then the second from worst is actually an A-. Basically just the simpler the product formulation is, there’s a lot less that can go wrong in manufacturing.
Ben: But when I look at your top-rated vitamin, like you have Carlson Labs here as your top-rated vitamin D. Do you ever take into consideration, for example, in this case vitamin D toxicity or an imbalance of vitamin D in relation to vitamin A or vitamin K. Do you pretty much primarily look at the supplement and then not really look too much into whether or not it’s balanced well with the things that would naturally need to be accompanied by that in the human body?
Neil: So right now we are looking at the supplements independently the same way an analytical lab would. The thing that we’ve really focused on is right now, everything on the site would be what I would consider a global ranking, a global grade. We’re basically taking studies that have the average of all humans and saying, “Based on that and our testing, these are the best products.” But our goal has always been to move from a global number one to a personal number one. So we want to actually understand what is your diet. Maybe you can connect in your food tracking. Tell us what other supplements, tell us what medications you take, and let’s start trying to get a better, kind of fuller understanding of who you are to the point where we know there’s a difference between a triathlete and someone who’s much more sedentary. Some of that’s vitamin D, you’re outside more. We can start figuring those things out. But some of them are significantly more complex. How does your diet plus your exercise, plus your supplements all fit together? But to me it feels like scientifically and medically, it seems like a solvable problem. It’s just something that we’re going to have to work on very hard.
Ben: Okay. So a ton of people take fish oil, and that’s actually one of the things that I recommend quite a bit to people. Especially athletes. Now when you’re looking at fish oils, what exactly do you look at? Is it just how much EPA is in it? Do you look at oxidation? What are the things that people should be taking into account on something like a fish oil? ‘Cause so many people use fish oil.
Neil: So the same way as all the other products, we’re looking at purity and potency. So we’re looking at not just the total omega-3 content, but yes we’re also looking at EPA and DHA specifically and seeing how those levels match up. We also look at Mercury and PCB levels…
Ben: Is there an optimum range for EPA and DHA that you look at or do you just see if it matches what they say it’s on the label?
Neil: So we’ve looked at a lot of different optimum ranges, and the problem is there’s basically no one single medical report that says this is exactly how much EPA and DHA you can take. And so largely we’re comparing them to the labels, and we’re generally comparing them to the recommendation that you should get at least one to two grams of total omega-3’s. So those are the recommendations that we’re following right now. But beyond that, we’re also looking at oxidation. So we actually can run tests on primary and secondary oxidation to see if a product has (a) either started to go bad or (b) has actually already started the oxidation process.
Ben: Interesting. Have you found that the way – so you’re ordering these or getting these products the same way that a consumer might get them off the shelf. Have you found there’s a difference between a fish oil that comes in a bottle versus in a packet versus in a canister or anything like that?
Neil: So we are trying to be very careful with light-based oxidation and oxygen-based oxidation. And so both of those are going to be significant issues. The liquid products even though that’s something that wasn’t as big of an issue in our testing, you would expect the products that are in one big liquid container to potentially have more risk of oxidation because you’re now, instead of with the capsule, you’re just exposing that one capsule to the air as soon as you take it. With a liquid product, you’re kind of opening and closing that bottle on a regular basis. And so definitely air is a big issue that people should consider. And lights and other storage conditions are very important. So we try to store these products really carefully. But it should make sense that basically the capsulated products would actually score better on these types of oxidation and freshness tests than the liquid-based products.
Ben: Gotcha. Okay. And then you’ve got energy and energy drinks. Now this is kind of interesting ’cause obviously you’ve got everything from Rockstar to… interesting, how come you mixed energy gels in with energy drinks? What’s the rationale there?
Neil: So right now we’ve got kind of a limited set of categories. And so sometimes, we’ve got the Recovery Shot, will eventually go into a different category.
Ben: Yeah. Okay. Gotcha.
Neil: But for now, we’ve kind of put them together. So we’ve put in the strips, and the gels, and the drinks, and the shots all in one category. Eventually we would expect, the same way that protein will eventually split up into mass gainers and shakes versus powders. But for now, basically because we have so many limited categories, we want to kind of bring all those products together.
Ben: Wow. GU Roctane got a D. I know a ton of triathletes, and marathoners, and stuff that use GU Roctane. Why would that have a D?
Neil: So one of the things that we actually found on that, and that may actually be due to the category, that it basically, relative to the energy category, it didn’t pull in a lot of basically a lot of high quality energy. So that’s actually something that could be, it may actually be something that could be better fit in a recovery category.
Ben: I’m looking here at breakdown though. It also exceeded 20% variance. Does that mean that it didn’t have in it what it said, what the label claimed?
Neil: Yup. So basically what we’re saying is it’s basically had lower, the label actually was pretty low. And we also were a little bit worried about its preservatives there. So that was kind of a… they had one of the watchlist ingredients. But I think this product could do better in its own category. That’s kind of one of those products where you’re taking Roctane for an entirely different reason than you’re taking Monster.
Neil: So sometimes its efficacy score isn’t as great because it’s competing against products with four or five times as much caffeine as it has. But it’s worth understanding why is that label accuracy so bad, how could things get better with that. And I think one of the things we’re trying to figure out is how to get products like this and how to get this information out to consumers, but we’re increasingly getting, every time we release these stories, we get calls immediately from the manufacturers. And so we’re, in some cases, the first thing we do is just get them as much information. We give them the lot and batch numbers, the expiration dates, and then they can go in and do some quality control. But really over the long term, we want to be, being that resource, that place where, the same way that the FDA sets standards for a lot of these other industries, that we can go in and say this is the standard of quality and this is the standard of quality that every consumer is now going to see.
Ben: Yeah. You’ll probably get a call from GU after this. There was one day where I had 46 GU Roctanes in a single day. It was during an Ironman triathlon. Actually I added it up. That’s a lot of Roctane. What else would cause an energy an energy drink or an energy compound, like I see MiO Energy here, the M-I-O Energy stuff, that also got a D. Would that be the same reason or are other things like contaminants and stuff?
Neil: So that one was a lot based on, there was an artificial color issue. There was a lot of watchlist ingredients in that product. So what we actually do, this is kind of an interesting little insight into what Labdoor does, we actually buy actual versions of all these products, and in some cases we actually test one or multiple batches of product in-house. Just like literally us individually doing user reviews. Something that you can actually tell just from tasting a product like MiO, you can taste this absolute chemically taste product.
Ben: Oh. You mean you test them by, like you’re not doing independent testing with your lab equipment. You’re actually using yourselves as guinea pigs?
Neil: Every once in a while we basically just try these things. Sometimes we’ll take one bottle as if we were professional taste testers. Like try all these different things. I remember no one got through the MiO taste test. We gave everyone a couple ounces of a drink and it was, everyone was just, you can kind of have this immediate sense of artificial flavors that just kind of hit you. And so some of those products, I think in general, just how we talk about it, different categories have different highs and lows, and the energy drink was kind of skewed lower than the rest. And I think a lot of it was due to all of these chemicals that are put into those products.
Ben: Yeah. You guys need to hire that out. You need to get an intern, or a college student, or something in there to just feed stuff to all day long. Just keep him in a cage and kind of toss a little supplements in and see what colors they change and what happens to their energy levels.
Neil: We try not to do too much because the problem is we know exactly what these products. So we try to limit our own exposure to it as well.
Ben: It’s crazy. So what do you take? Like if you wake up in the morning, are you using supplements? Or has this just totally scared you away from using any supplements at all?
Neil: Yeah. So that’s actually something that’s really important to talk about is that, we’ve talked a lot about kind of the negative issues about the industry. But there are some definite positives. So once you find a fish oil product that’s actually a high quality, A-grade product, that is, most doctors, I personally tell people to take the fish oil. You personally tell people to take fish oil. It’s a great product to take as long as you can find the right one. And so I actually do take fish oil right now and I pulled it right off the top of Labdoor’s list. Same thing with the multivitamin. So those are the two that I take every single day. And I’m actually about to start taking a probiotic. And as a quick tease, we’re actually releasing our probiotics within the next two weeks. And so we’ll actually have data on that as well.
Ben: Okay. Give us some kind of a spoiler, dude. Throw us a bone. Tell me what you’re taking or what you guys have found to be just freaking awesome for a probiotic.
Neil: So I haven’t actually started taking the products yet, but we have seen a few products, Renew Life Products scored very well. But we’ve found a number of products that were just excellent, hit all their label claims. And so we were…
Ben: That’s got to be tough to do with the billions of probiotic bacteria that you have in a single capsule.
Neil: Yeah. The probiotics was the highest variation in highs and lows that I had ever seen. You’re talking about places that claimed 1 billion and had 50 billion. You we’re saying the opposite. You’re saying things that claimed 10 billion and were under a billion. It was all over the place. And so it’s absolutely the category that I think you need Labdoor the most. Just because there was just these incredible variations.
Ben: Yeah. I need to have my wife send you some of her kimchi. We can add that into the probiotic category. That’s my primary source.
Neil: So many requests.
Ben: Yeah. You got to start doing foods now like all these functional foods, and nut butters. Man, all sorts of potential here. This is pretty cool. And you guys have a magazine on the website, you have all your rankings, all your products. I like it. If folks are listening in, labdoor.com is the place to go to check this out. Pretty cool resource. Neil, anything else you want to share with folks who are listening in?
Neil: The last thing is actually that we are putting in big updates into our iPhone and Android apps actually today. And so it’d be a great place to basically take these kinds of reports into the store with you so you know exactly what you’re buying even when you’re in retail.
Ben: Okay. Cool. Nice. I’ll link to all this over in the show notes over at bengreenfieldfitness.com. If you want to check it out, just go to bengreenfieldfitness.com/labdoor and you can check it all out. So Neil, thanks for coming on the call, man.
Neil: Thanks for having me.
In every drugstore aisle in America, shoppers cast the classic pose: two products in hand, confusion on their faces as they attempt to decipher the advertising claims and safety labels. It’s an important decision, but it is so hard to be certain you’re making the right call.
A new company called LabDoor buys dietary supplements and energy drinks off retail shelves and sites. Then, they send each product to an FDA-registered laboratory for a detailed chemical analysis. Their technical team then collects the laboratory results and builds algorithms to translate this data into simple grades and rankings.
The result is that you get to find out what’s good, what’s bad, and whether your protein powder, fish oil, vitamin D, energy drink or multivitamin actually contains what it says it contains, and whether it has any nasty contaminants.
In this episode with Neil Thanedar from Labdoor, you’ll find out:
How supplement companies “cheat” when it comes to getting their products analyzed…
Why your protein powder may not actually have in it what it says it has in it…
The shocking truth about fish oil quality…
The one energy drink that is the biggest chemical cocktail in the supplement industry…
And much more!