The Failure of Self-Reporting

There are three recent articles that demonstrate something that’s hard to get across. It’s this: people are bad witnesses to their own behaviors. Anyone who has ever done an academic study with human subjects knows about this.

In response to questions, people say things that cannot be supported by evidence, things based on opinion, false memories, bad memories, wishful thinking, good intentions (trying to be helpful), bad intentions (wanting to mislead). Most of all, I think this happens because they don’t know their own minds or selves.

“Do you get enough sleep?” is an innocuous question, but because it depends upon self-reporting, it’s answer is not very valuable. Even a question like, “How many of hours of sleep do you need a night?” is bound to return largely useless results, even given a large sample size, because some people exaggerate or understate for effect (to seem different, to seem better, to support other behavioral claims—”I work a lot, I’m stressed, I’m sick.”).

It’s the same in the language business. When someone says, “I remember that word from 1955; you can trust me” but there’s no other record of the word existing at that time; or they say, “It was me: I coined that word” but they have no proof and there are records saying otherwise; or when someone says, “No, you’re wrong, I know that word doesn’t mean that” in contradiction to respected reference works; or when someone says, “I’ve never heard that word; it can’t be a real word”—those are all problems with self-reporting.

These are all reasons why opinion polls are mostly useless: any question beginning with “Do you think…?”—any question asking for a judgment based on little or no evidence—sets off a chain of ego, self-delusion, self-aggrandizement, rationalization, and calculation that puts the poll’s results into question. Think about the differences between pre-election surveys and election results. They’re often very different. The surveys are flawed because they rely on self-reporting, but the poll results are more accurate because they record behavior.

I’ve seen a good number of advertising focus groups. They are, to me, the most extreme form of bad self-reporting. A room full of paid nobodies, all asked to make snap decisions on carefully controlled information, to a video camera or people behind a two-way mirror. Every focus group I’ve ever seen has been so manipulated by the process that the only outcome has been to convince the client to support the agency’s creative campaign—naturally, since focus groups are usually run by advertising agencies. On top of that, the paid nobodies are the squirmiest, wishy-washiest, say-anything, predictable bunch you could ever want to meet. They’re on the spot. They feel a bit important. They feel it’s their duty to sound strongly opinionated, so they loudly and forcefully say whatever comes to mind and back it with rickety justifications. Bad self-reporting at its worst.

The following three stories are about that, about our inability to know or judge ourselves—and to know when to not have an opinion.

Perceptions: Everybody Except My Slender Friends and Me by Adam Nagourney reports, “Nine out of 10 Americans think most of their fellow citizens are overweight, a new survey shows. But only 70 percent believe that most of the people they know are.”

Here’s Why by Malcolm Gladwell explains that the types of reasons we give for our behavior are very much tied up in our relationship to the people who are affected by our actions.

I’m O.K., You’re Biased by Daniel Gilbert talks about how we perceive ourselves as mostly impervious to outside influences, such as advertising or marketing, but we perceive other people as being very susceptible to such messages. For example, “A 2001 study of medical residents found that 84 percent thought that their colleagues were influenced by gifts from pharmaceutical companies, but only 16 percent thought that they were similarly influenced.”

Posted April 18, 2006

Related Posts

Comments are closed.