Does Your Personal Experience Prove Anything? (Anecdotes are Sexy but Not Conclusive)

ben-white-178537-unsplash.jpg

Many of the book manuscripts I’ve seen over the years present a system for behavior change that’s based solely on a sample of one — the author’s personal experience.

The premise of these books is: I did X, and here’s how you can do it, too. (For X, substitute any desirable outcome: lose 25 pounds, make a million dollars, attract a loving partner, find parking spots on a regular basis, etc.)

What’s easy to miss is the underlying assumption: If it worked for me, it will work for you.

This is exactly where many would-be authors build a towering edifice of content on a crumbling foundation.

Why N = 1 Matters

Scientists who design experiments use the capital letter N to denote the number of subjects involved. If a psychologist includes N = 2000 in a paper about her study, it means she observed 2000 people.

Why pay attention to this number? Because the more subjects, the better. A study based on observations of 2000 people (or 20,000 or 200,000) inspires more confidence than a study based on 20 people. With more subjects, we have more confidence that the conclusions will generalize to the population at large.

If N = 1 (meaning that the author is the only subject), then the work inspires little — if any — confidence that the observed results will apply for the rest of us.

Why N = 1 Fails

There are many reasons why what works for one person might fail for the rest of us:

It’s possible for people to draw dumb lessons from their personal experiences. Case in point: Years ago I read a post by a technology journalist who concluded that Apple was in big trouble. Why? Because he went to a single convention and saw more people with Android phones than iPhones.

This journalist didn’t offer any survey results or other data that could be independently verified. Rather, it was his general impression that more people were using Android phones. And from there he leapt to a far-reaching prediction about the fate of a large company.

I know that this example seems obvious — perhaps even absurd. But any of us can fall into this type of lazy reasoning. We can even construct entire beliefs systems based on it.

Correlation does not equal causation. The outcomes that an person experiences might be due to coincidence and have nothing to do with the program he recommends.

For example, medical researchers have a term to describe symptoms that resolve themselves simply with the passage of time: These symptoms are described as self-limiting. In other words, they simply disappear over time for reasons that we may not understand.

If a problem you’re dealing with is self-limiting, then simply waiting it out might be as effective as any new behaviors that you try to implement. Those behaviors changes might correlatewith a solution but not cause it.

For more examples of the correlation equals causation fallacy, see these wonderful posts by April Hamilton:

Life is complex. Human behavior is multi-faceted. Any event can have multiple causes. And each of our readers already has a personal ecosystem of interrelated habits that differs from our own.

When we recommend one strategy or technique as the way to produce a given result, we ignore the differences between all those ecosystems.

Limit your assertions to this is how it worked for me

I appreciate authors who tell their story in a moving way without the arrogant assumption that events will unfold in the same way for the rest of us.

This is one of the reasons I enjoyed Walter Isaacson’s biography of Steve Jobs. There are many lessons that we might draw from Jobs about how to live well (and how to not live well). But Walter makes no attempt in his book to offer a program for “succeeding the Steve Jobs way.”

Perhaps this seems obvious. Yet many of the proposed self-help books I see are biographies or memoirs in disguise. In brief, they’re ultimately about the author — not me or you.

Test, test, test as best you can

If you can conduct a formal scientific study to test your ideas — great.

If you can do survey research the way that BJ Fogg does for his Tiny Habits program — great.

Many of us don’t have the resources or skills to do either of the above. So recommend your system to as many people as possible. Then track their results over time, getting feedback in any way that you possibly can.

This is one of the reasons that I appreciate David Allen’s Getting Things Done system. David personally coached hundreds of people. And, he did this for decades beforepublishing a book based on what his clients experienced.

This is much different than writing a book and collecting a bunch of testimonials. David watched people change their behavior as they implemented specific new behaviors. And then he personally observed the results. A testimonial is a value generalization (often ghostwritten) that doesn’t reference any specific behavior change.

Present a process that readers can test for themselves

Powerful how-to books help readers to discover what works for them.

I enjoy James Altucher’s Choose Yourself for this reason. The book is loaded with personal anecdotes (including times when James failed miserably). Yet his core recommendation is a set of daily practices. You can test them for yourself and see what sticks.

The work of Ramit Sethi has the same spirit. He recommends specific behaviors for us to adopt and encourages us to systematically track results.

In summary

Any of us who seek to influence the way that people think and behave can benefit by remembering to:

  • Offer possibilities, not prescriptions.
  • Recommend specific behaviors to implement rather than vague generalities.
  • Get feedback on our content — and be willing to revise our content based on feedback.
  • Be humble. As James Altucher once put it: “Note, this is only what works for me. For you, I have no idea.”

And … a story

When I try to explain all this to prospective authors of self-help books, I get a variety of reactions. I’ll illustrate with a composite and fictional example:

Me: Your book lists 123 techniques that readers can use to change their self-defeating habits. Where did those techniques come from?

Author: Well, I read a lot of self-help and spirituality books.

Me: Great. Me, too. Readers will want to know what’s new or different about your book.

Author: Well, I put my own spin on everything. Plus, the techniques worked for me.

Me: Really? You personally tested all 123 techniques?

Author: Well, not all of them. Most of them.

Me: What about the techniques you didn’t test?

Author: Well, my friends like them.

Me: Your friends? How many?

Author: Well … actually, my wife read the first draft of the manuscript. She really likes everything.

Me: Did anyone else read it? Maybe someone with a more objective point of view?

Author: Ahhh … ummm … well, not yet.

Me: OK. No problem. We can work on that. Now, did you do a literature review?

Author: Literature review? What’s that?

Me: It means you look through any research that is relevant to the subject of your book. There are hundreds of published studies about habit change.

Author: Really? Where do you find that stuff?

Me: In professional journals that publish peer-reviewed papers.

Author: I don’t bother with that stuff. It’s way too dry.

Me: So as of right now, we don’t have solid evidence that your techniques work.

Author: Of course they work. They worked for me, and my friends, and my clients. I can give you lots of good anecdotes.

Me: Well, anecdotes aren’t quite enough. They are useful as stories that you can use to lend a human touch to your writing. But a few isolated anecdotes don’t count as evidence for your ideas.

Author: No? Why not?

Me: For lots of reasons. Because anecdotes are only based on the experiences of a few people. Because they’re often embellished when retold over time. And because anecdotes don’t involve careful observation or isolate any variables. When you produce a new result in your life, how do you know that one of your 123 techniques created it?

Author: Hmmm… . (Thinks to himself: This guy is going to be a pain in the ass. I need another editor.) Well, this is all real interesting. I’ll get back to you.

Photo by Ben White on Unsplash