6 Reasons Not To Trust (most) Workplace Wellness Research

No one can accurately accuse workplace wellness research of lack of consistency. Indeed, study after study done by vendors and related organizations and journals with a vested interest in the answers come up with the same conclusions. As one proponent commented to me recently on a LinkedIn Discussion,

“I don’t think I have ever read a research article on a workplace
wellness program that did not show savings.”

 The question of course is: Are the conclusions valid? Even a relatively quick look at the research gives us a pretty decent answer to that question and sheds some light on why these studies always come up with the same positive conclusions.

By way of introduction to this discussion, it might be instructive to consider a variant of a quotation often attributed to Albert Einstein, but more likely originating from the group Narcotics Anonymous: “Insanity is doing the same thing over and over again but expecting different results.” As we turn to a brief overview of the research on workplace wellness, the variant might look something like:

“Insanity is doing the same things over and over again and hoping no one will notice.”

Let’s see what the research is doing that might give us reasons not to trust the conclusions: *

1) Comparing motivated participants with unmotivated non-participants — With few exceptions, studies claiming positive ROIs for workplace programs do so by comparing participants to non-participants. As one vendor honestly put it:

“This approach is used because there is a need to compare the program participants to
something in order to judge whether there have been improvements.”

 One way of thinking about why this is problematic is to consider the case in which you have 50 smokers who want to quit and 50 who don’t — all with the same demographic profiles. What do you think might happen if you follow both groups for a year with no intervention? If you said the group that claimed it wanted to quit would do better — you would undoubtedly be correct. Bottom line: Motivated participants will always outperform non-participants in the short term. So yes, you are comparing them to something, but the ROI you come up with using this type of comparison means nothing.

2) Ignoring “regression to the mean” when it comes to changes in risk factors — When calculating ROIs, the research typically counts people who go from high risk to low risk while ignoring people who go from low risk to high risk. If we use the 100 smokers example again — this time assuming that they comprise the total employee population and that half of them quit and restart in alternate years — by only counting those who quit (high risk to low risk), we would come up with 100% reduction in smoking each and every year even though the smoking rate would remain at 50% forever.

3) Claiming to reduce a number by more than 100% — The vendors’ literature abounds with claims that their programs resulted in employees who:

  • Were 300% less likely to be absent due to illness
  • Had a 350% decrease in appointment waiting time
  • Had a 240% decline in the number of people on disability

Unfortunately, one of the fundamental rules of mathematics is that you cannot reduce a number by more than 100%. Think about it: If employees were 300% less likely to be absent, does that mean they came in on the weekends and did not report the hours they worked?

4)  Applying the “last man standing fallacy” — This is a relatively common technique by which researchers only measure changes in people who have completed a number of consecutive years of a particular program. Of course this usually ends up being a relatively small minority of the population. So the 25% of participants who the research claims lost a certain amount of weight ends up being 25% of the 25% (6.25%) who remained in the program after two years.

5)  “Spinning” study results — The “spins” routinely put on the research would make any politician proud.

a. Take for example the recent meta-analysis of workplace wellness ROIs in the “American Journal of Health Promotion.”The results of randomized controlled trials (RCTs) — the Gold Standard for research — showed the ROIs had an overall mean value of -0.22, which means for every dollar invested in wellness programs, only 78 cents was returned. When less valid means (incorporating some of the techniques cited above) of evaluation were used, some programs did show a positive ROI. The authors’ conclusion was that “the mean weighted ROI of workplace health promotion demonstrated a positive ROI.” As one critic pointed out, however, averaging research methods with different validities is like averaging Ptolemy and Copernicus and concluding that the earth revolves only halfway around the sun.

b. And who can forget the recent claim by another industry leader that a 1-pound decrease in weight in a workplace program constituted “meaningful” weight loss.

6) Ignoring iatrogenesis — If all of this chicanery was not enough to invalidate any claims of cost savings, rarely if ever does the research consider the possible unintended consequences of its interventions. From overdiagnosis and overtreatment due to the “medicalization” of the workplace, to the host of negative consequences associated with applying extrinsic motivators, to the dangers of repeated weight cycling that follow all weight loss interventions, the likelihood of saving money (or even breaking even) from traditional workplace wellness programs is miniscule.

We end this blog with a variant of another well-known quotation originally attributed to the 14th-century English monk and poet John Lydgate who said:

“You can please some of the people all of the time; you can please all of the
people some of the time; but you can’t please all of the people all of the time.”

When it comes to workplace wellness research, an adaptation of this quotation usually attributed to President Abraham Lincoln seems particularly apropos:

“You can fool some of the people all of the time, and all of the people
some of the time, but you cannot fool all of the people all of the time.”

If we take this quotation together with the definition of insanity we introduced earlier: “Doing the same things over and over again and hoping no one will notice,” we believe a wake-up call for the industry is imminent because:

1)      The percentage of people who are being fooled is rapidly diminishing,

AND

2)      More and more people ARE noticing.

 

* For more in-depth information on and explanation of many of these issues, see www.theysaidwhat.net

 

Jon Robison, PhD, MS, MAJon is an accomplished speaker, teacher, writer and consultant. He has spent his career advocating that health promotion shift away from its traditional, biomedical, control-oriented focus, with a particular interest in why people do what they do and don’t do what they don’t do. Jon has authored numerous articles and book chapters and is a frequent presenter at national and international conferences. He is also co-author of the book, “The Spirit & Science of Holistic Health — More than Broccoli, Jogging and Bottled Water, More than Yoga, Herbs and Meditation.” This work formed the foundation for one of the first truly holistic employee wellness programs — Kailo. Kailo won awards in both Canada and The United States, and the creators lovingly claim Jon as its father. Contact Jon at: jon@salveopartners.com or jonrobison.net.

View all articles by Jon Robison »