How Reporters Get Psychology Research Wrong

Back in 1998, the Internet was still new to most of us. Newspapers could get mileage from headlines such as Isolation Increases With Internet Use.

Yet the story behind such headlines is a reminder to keep your crap-detector up and running when reading about research on human behavior.

What made those early headlines about Internet use so misleading was a patina of credibility. After all, a published study was often cited — Internet paradox. A social technology that reduces social involvement and psychological well-being?

The abstract for this article even noted that:

… greater use of the Internet was associated with declines in participants’ communication with family members in the household, declines in the size of their social circle, and increases in their depression and loneliness.

What the reporters forgot is that all studies are not created equal. Reading beyond the abstract for this one, you discover that:

  • The study involved only 256 people from a single city in the United States — Pittsburgh.
  • Before the study was completed, 87 people dropped out.
  • Only people new to the Internet were included. Anyone with an existing online connection was excluded.
  • Depression and loneliness were measured subjectively. Rather than being assessed by an expert, people simply reported how they felt.
  • Even the heaviest Internet users reported only slight increases in depression and loneliness.
  • The above limitations were openly acknowledged by the study’s authors.  (Notice the question mark at the end of the article’s title.)

In fact, later research indicates that greater Internet use links to having more offline relationships.

The lesson here is to think critically about any research report. Ask questions such as:

  • How many people were included in the study? (In research parlance, what was the sample size?)
  • Can we generalize findings about those people to the population at large?
  • How did the researchers measure the critical variables in the study (in this case, Internet use, loneliness, and depression)?
  • Does the study actually support a cause-and-effect relationship between variables (such as Internet use causes social isolation)?
  • Was any cause-effect relationship strong enough to matter?
  • Could something else explain the relationship between variables (perhaps Internet users were depressed and lonely before the study began)?
  • Do other relevant studies point to a similar — or different —conclusion?

For more suggestions, see the entertaining Joe Hanson on Crap-Detecting Science News.