How statistics can protect your willpower

Today is Valentine’s Day (Happy Valentine’s/Bitter Single’s Day!). So I thought I’d give my post a brain-washing theme (cynical much? 🙂 )

With the advent of Instagram, Facebook, Twitter and now, Vine, the power of the testimonial has never been greater. It’s easy to be bombarded by images of amazing before and after, and even “now” photos of clients of trainers and nutritionists, all trying to cajole their way into your mind and ultimately into your wallet. Googling the terms “fitness”, “marketing” and “testimonial” returns over 100,000 hits, the first few pages of which are testimonials on how testimonials can help grow a fitness business. How’s that for meta?

And while my friend Brad Pilon had some interesting thoughts on transformation contest winners, which I completely agree with, I can’t help but put my two cents’ worth (which you cannot get in Canada anymore since we have now abolished the penny) on the whole deal.

I think that it goes without saying that if someone presents only their best work, that everyone would agree that that would be a biased view of their work overall. This is an easy-to-understand concept. I don’t need to write a whole blog post on the weakness of testimonials as evidence (I hope.)

But clearly, the testimonial (or, case report, in medicine) has powerful influence as a tangible presentation of a result; otherwise they wouldn’t be so popular!

The illusion, however, lies in the fact that you’re clearly not seeing the whole picture, especially when someone can post results every week or 2–it seems like overwhelming proof that they have one of the magic bullets to make you look like you’ve always wanted to look, and beyond.

So how is it that in the face of what we already know about the inherent bias in case reports that testimonials still thrive as one of the most successful ways to sell product? And furthermore, knowing that testimonials influence our impressions, is there a way to counter that effect to avoid being scammed into spending time/money/energy/willpower in something that might not have any substantial effect?

Krupat et al. Generalizing from atypical cases: How general a tendency? Basic and Applied Social Psychology, 19(3):345-361, 1997.

In an interesting experiment, researchers gave 45 subjects either an anecdote or statistical information on a the performance of a fictional brand of car (called the Clipper) which were both negative. The anecdote was a review of a single person’s experience with the car, while the statistical review was a factual summary of the car’s performance statistics with respect to how many complaints it had, and what percent of owners had problems with what car systems (fuel system, engine, etc.) The subjects were then asked to rate their impressions of the car. The purpose of this initial phase was to test whether both essays were approximately equivalent in their negativity, which was achieved, despite their faulty use of a statistical test to back it up.

In the second phase, 291 subjects were given one of the essays (anecdote or statistical), but then given an “Editor’s note” which was also either oppositely statistical or anecdotal in nature that was a positive review of the car (e.g. if you read the anecdote first, you got the statistical Editor’s note.) The results of the second phase showed that ratings of the fictional car were lower if the subjects read the anecdote first than if they read the statistical review first (i.e. If both groups started at a neutral opinion of a car they had never heard of before, the anecdote provoked a stronger negative rating than the “factual” review). However, after if subjects started with the statistical information first, that they tended to not be swayed by the Editor’s anecdote (i.e. they rated the car negatively initially and continued to rate the car negatively.) What’s interesting is that the subjects who started with the anecdote first _were_ swayed by the Editor’s statistical-styled note, and rated the new car more positively than after reading the negative anecdote.

In a second experiment, the researchers wanted to know whether prior impressions would have any bearing on how conflicting anecdotal vs statistically-based information might change subject impressions. So this time, they gave 569 subjects the same experiment, but this time, each subject given information on either a Honda Accord (which at the time was generally considered a good car), a Yugo (at the time a well-known bad car) or the fictional Clipper.

Initial ratings prior to any essay favoured the Honda, then the Yugo and in last place the Clipper. No matter which initial essay the subjects got first, their ratings of the cars went down more if they got the statistical review first than if they goto the anecdote, suggesting that a single anecdote won’t displace a pre-formed opinion much, but that statistical information will. Interestingly, in the larger sample, the anecdote did not result in a lower rating than the statistical review as in the first experiment.

When confronted with either a positive anecdote or positive statistical information, the same trend happened. Statistical information tended to increase the car’s rating (regardless of the car) more than anecdotes, though both did increase ratings, regardless of their initial pre-essay ratings.

I don’t think there’s anything wrong with using a testimonial to get business. A positive review from a trusted source, or even word-of-mouth by friends are both testimonial-type evidence that we use daily to decide things from where to go for dinner to buying a car to deciding what toilet paper to buy. Stories and narrative are inherent parts of our social culture and communication. However, when it comes to making important decisions about your body, and your health, where you’re going to spend a significant amount of money, time, physical effort and mental energy, it’s important to keep our inherent in-born bias towards anecdotes in mind. The reality is that you don’t know if the testimonial results are typical, or atypical. Did these people succeed BECAUSE of the program/supplement/diet they were on, or IN SPITE of it? How many testimonials aren’t being shown? How many clients have been unhappy with their results? What’s the number of clients that any given trainer/training method/diet/supplement has to have to produce one amazing transformation? What is that elusive denominator? There’s a Numbers Needed to Treat in medicine that is the number of patients who would have to be treated to give one patient benefit. Maybe there should be a NNTT (Number Needed to Transformationally Train) for trainers and nutritionists?

As Brad puts, in a much shorter read, what’s in common with all honest testimonials is their willingness to commit to the goal and to do what’s required to get there. The rest is just marketing for the purposes of selling. However, it turns out that appropriate statistics and well-performed scientific evidence might have some protective effect against the deluge of inherently biased testimonial presentations!

The Bottom Line: Information presented in a statistical manner seems to have a higher impact on impression ratings than information presented as anecdote or testimonial. However, according to this study, this effect may hold regardless of whether that statistical information is actually correct, since none of the essays presented to the subjects was factual. So even though statistics could protect you from the influencing effects of testimonials, the problem is being able to figure out if the evidence is of value or not; which is not always so easy. So in a totally self-serving manner, my advice is to keep reading this blog ;). ‘Til next time, guard your willpower.