WELLNESS RESEARCH – The “Last Man Standing” Fallacy or Why It’s Not Nice to Play with Denominators

Dr. Jon Robison with mathematical validation and editorial assistance from Al Lewis.

Workplace wellness research is notorious for what we refer to as The “Last Man Standing” fallacy. This approach typically only includes measurements on people who complete consecutive years of an assessment, screening, or other intervention, which is often a minority of the originally participating population (the bulk of the population being lost to dropout). By definition, of course, a self-selected highly persistent population is not reflective of the whole.

Published April 2015 by Taylor & Francis Group

Published April 2015 by Taylor & Francis Group

 

Nothing in the health fields is more consistently evidence-based than the abject failure of weight loss interventions to achieve long-term success for any but a tiny minority of those who participate. This does not stop proponents authoring weight loss studies from claiming just the opposite. To make it look like long-term weight loss is actually happening, these researchers often base their claims on the relatively few “survivors” who stay to the end of the intervention after everyone else has dropped out. To demonstrate how they perpetrate the “last man standing” fallacy, we examine a weight loss drug study from the literature recently suggested to us as proof that weight loss drugs can result in successful long-term weight loss.

The Study (“Effect of sibutramine on weight maintenance after weight loss: a randomised trial” – Lancet 2000; 356: 2119–25)

In this study, 605 “obese” (BMI 30 or greater) individuals were put on a diet (600 calorie deficit per day), exercise regimen (30 minutes a day), and 10 mg of the weight loss drug Sibutramine for six months. Those who lost more than 5% of their bodyweight during that period were then blindly randomized to another 18 months of either continued drug treatment or placebo with the primary outcome being weight loss maintained at two years.

The Rhetoric:

The authors claim: “This individualised management programme achieved weight loss in 77% of obese patients and sustained weight loss in most patients continuing therapy for 2 years.” Out of context, this sounds like a pretty successful treatment and outcome… but:

The Reality:

By playing with the numbers —particularly the denominator — the authors make it seem as if the study is much more successful than it really is. This happens quite a bit in workplace wellness research and is almost ubiquitous in the area of weight loss. Using this study as an example, here is how they do it:

  • Of 605 people who started the study, 138 (23%) lost less than 5 pounds in the first six months (even though they were taking the drug, restricting caloric intake, and exercising). They were eliminated from the study.
  • This leaves 467 (77%) of the original 605 participants. Of those 467, 352 were assigned to continue the drug for another 18 months and 115 were assigned to the placebo.
  • Of the 352 continuing on the drug after six months, 148 (43%) dropped out. So, now we are left with 204 folks of the original 605 or about 34% still taking the medication — meaning that 66% of the original sample has now been eliminated from drug treatment.
  • Of those 204 who actually completed the two years, 89 (43% – of the 204, which was 34% of the original 605) – or actually 89/605 (14.7%) “succeeded” – meaning they kept off 80% or more of the original losses or anything over 5% of their original weight).
  • A 5% weight loss for a 227-pound person (average initial weight of participants) is about 11 pounds; 80% of that 5% loss is about 9 pounds or a bit more than 1 BMI unit.
  • A 10% weight loss for a 227-pound person (achieved by only 20% (18) of the 89 participants left standing or 3% of the original sample) would be about 23 pounds — 80% of that would be about 18 pounds or about 2.5 BMI units.
  • Anyone (14.7% of the original group) who was “left standing” after two years who lost 9 or more pounds was considered to be a success. Because the average BMI of the original participants was about 37, almost all those who “succeeded” would still be “obese” after the two-year treatment. And it is not known what percentage of the original large group would have lost 9 or more pounds anyway.
  • Additionally, according to the article (Figure 2 on p. 2121 – Mean Bodyweight changes during weight loss and weight maintenance phases), the weight trajectory for the 14.7% who “succeeded” with weight loss maintenance was clearly increasing even though they were still participating. (And even though many were put on increased doses of the drug to counteract the weight regain.)

The Bottom Line

Let’s remember that the authors concluded that “this individualised management programme achieved weight loss in 77% of obese patients and sustained weight loss in most patients continuing therapy for 2 years.”

It is true that the program achieved weight loss in 77% of the people who started the program. But almost every weight loss program does that — and many result in much greater weight losses in the first six months. And it is true that most patients (although less than 15% of the original sample) who lasted the full two years sustained some weight loss. However, it is clear that even while still in the program and on the drug these individuals were gaining their weight back.

And that does not even consider the iatrogenic consequences of being on these medications long term, which is likely why most of them (including this one) are now illegal. The persistence with therapy rates for this drug before it was removed from the market was less than 10% for one year and 2% for two years (Int.J.Obesity 2007;31:1567-1570.) — pretty dismal outcomes for a “successful” weight loss treatment. The real question, as expressed in the immortal words of the American philosopher Jack Nicholson is, “What if this is as good as it gets?” The conclusive answer, if we examine the weight loss literature over the past three decades, is that it is indeed!

Although claiming the 77% plus “success” for the people who managed to stick with the program for two years may be numerically correct, it is clinically all but irrelevant because the appropriate denominator — the number of original participants — was not 204, but 605. Using 204 as the denominator is plain and simple data manipulation. For all intents and purposes, this intervention failed completely for more than 85% of the folks who started out, and even for those who were “successful,” their weight was returning toward baseline anyway, just as has been the case with every other weight loss treatment ever studied.

Unfortunately, advocates of weight loss drugs persist in betting that people will continue to believe them, even when it’s perfectly obvious to a lay observer that these drugs are simply a bad idea from the perspective of both health and cost. And this same “creative math and outcomes reporting” is found in numerous studies claiming efficacy of traditional 4P (pry, poke, prod and punish) “Wellness or Else” programs.

More information on this and other data manipulation failures can be found at http://theysaidwhat.net/2014/07/23/aetna001/.

Jon Robison, PhD, MS, MAJon is an accomplished speaker, teacher, writer and consultant. He has spent his career advocating that health promotion shift away from its traditional, biomedical, control-oriented focus, with a particular interest in why people do what they do and don’t do what they don’t do. Jon has authored numerous articles and book chapters and is a frequent presenter at national and international conferences. He is also co-author of the book, “The Spirit & Science of Holistic Health — More than Broccoli, Jogging and Bottled Water, More than Yoga, Herbs and Meditation.” This work formed the foundation for one of the first truly holistic employee wellness programs — Kailo. Kailo won awards in both Canada and The United States, and the creators lovingly claim Jon as its father. Contact Jon at: jon@salveopartners.com or jonrobison.net.

View all articles by Jon Robison »