Stat-Spotting A Field Guide to Identifying Dubious Data by Joel Best
California, 144 pp., $19.95
As the current economic apocalypse reminds us, the most valuable lifetime text on money--or almost anything else, particularly in Washington or Wall Street--was not authored by John Maynard Keynes or Friedrich von Hayek or Adam Smith or Niccolo Machiavelli. It's The Emperor's New Clothes by Hans Christian Andersen.
This merry fairy tale, in which two Bernie Madoff-type tailors convince a vain king that they've woven him a suit so elegant it's invisible to the stupid or incompetent, ends with the king parading, naked as Clintonian ambition, before a cheering populace unwilling to admit it isn't smart enough to recognize quality threads. Cheering, that is, until a small child points out that the emperor has no clothes.
That child obviously went on to become a properly skeptical reporter. He clearly escaped indoctrination in pre-K from the likes of Foucault, Derrida, and other current icons of academe who cross-stitch the notion of truth with the same needle and thread that clothed the emperor. Alas, today he's an endangered species.
Now comes one Joel Best, author of Damned Lies and Statistics, with another earnest little book to help us scissor through the lumpy statistical featherbed that pads every fabric of our public and private life. Want full employment? Want honest stockbrokers? Want health care? Who ya gonna call? Stat busters!
Best, a professor of sociology and criminal justice at the University of Delaware, is actually a bit of a political Pollyanna, useful as his slender volume is. He assures us that some statistical errors are accidental or inadvertent, which those who toil amid the spinning wheels of the nation's capital know to be largely hoo-ha.
Everybody's cooking the numbers. Causists do it to plead their passion. Candidates do it to justify their politics. Federal agencies do it to swell their budgets. Scientists do it to glean research money. Even journalists, who should know better but pretend they don't, do it to grab headlines and boost careers. Show me a Washington worker bee wedded to statistical integrity and I'll show you someone underappreciated, underpaid, and starving for truth. Not that there's anything wrong with that.
Nevertheless, Best's prescribed body armor for making one's way through these statistical shootouts is made of the very components Aesop or Socrates advised: skepticism, multiple perspectives, comparison, and common sense. For example, when confronted with fantasyland feminists shrieking that 4 million American women every year are battered to death by husbands or boyfriends, it helps to know that the number of deaths in 2004 of both sexes in the United States from all causes was only about 2.4 million. Just over half of those died from either cancer or heart disease. In comparison, such highly publicized causes of death as traffic accidents (43,000), suicide (32,000), homicide (17,000), and HIV/AIDS (16,000) each accounted for only about 1 or 2 percent of all deaths--a far smaller proportion than many headlines and fundraisers would have us believe.
Likewise, it helps to know that the U.S. population is something over 300 million, and that about 4 million babies are born in the nation each year, fairly evenly divided by sex. Thus, if 4 million women were being battered to death each year (never mind by whom), the nation's population would be undergoing a fairly precipitous decline.
Such benchmark statistics are available in the annual Statistical Abstract of the United States--one of the few government publications turned out by an agency (the Census Bureau) that has no political axe to grind. It's available online.
Best points out that the 4-million-battered-women figure is recirculated regularly on various websites, despite its obvious falsity: "We have no way of knowing what led the creator or the [first] website to make this error," he says charitably. But my own experience suggests we do.
In January 1993, while idly surfing through wire stories as a reporter for the Washington Post, I encountered a number of stories claiming that more men beat their wives and girlfriends on Super Bowl Sunday than on any other day of the year. The claim had a certain aura of plausibility. You know: beer, testosterone, gridiron violence. Among the many reporting this "fact" were the New York Times, the Associated Press, the Boston Globe, and NBC. I had no reason to doubt the reported claim that women's shelters reported a 40 percent increase in domestic violence each Super Bowl Sunday.
But then I stumbled on a secondary story that gave me pause. A women's group in California had held a news conference ramping up the Super Bowl claim. One of those women was Sheila Kuehl. The story identified her as managing lawyer of the California Women's Law Center, but she was better known nationally for having played Zelda, Dwayne Hickman's nagging wannabe-girlfriend, on the 1960s sitcom The Many Loves of Dobie Gillis. While this needn't necessarily void her mastery of jurisprudence, it did spur me to examine her claim. And that involved an Old Dominion University study purporting to say that women's shelters in Northern Virginia reported 40 percent more cases of domestic violence whenever the Washington Redskins played.
Now, that didn't compute. In its annual orgiastic coverage of the Redskins, the Post had published numerous stories over the years examining the effect of Redskins fever on the Washington metropolitan area. One enduring finding was that the city and its suburbs are never as safe, and never as quiet, as on days the Redskins play. Police calls, hospital admissions--all were down. Both the crooks and the cops were watching the games. Could the paper have missed something?
Intrigued, I called the authors of the Old Dominion study cited by Kuehl. They said she had totally misrepresented their findings: There was no meaningful statistical relationship between domestic violence and football. Certain there must be a grain of truth somewhere in the Super Bowl violence story, I phoned other authorities, surveyed hospitals and women's shelters, and was astounded to discover that the whole thing was myth. Apparently it had all been ginned up by a yappy lefty "media watchdog group" called--and wouldn't the tailors of the emperor's new clothes love this?--Fairness and Accuracy in Reporting (FAIR).
The Post's front-page story debunking the Super Bowl violence myth prompted embarassment and retractions in the journalistic ranks, and howls of outrage from Kuehl, FAIR, and others who predictably attacked me as some sort of front man for wife beaters. I always urged anyone doubting my story to find verifiable statistics showing a domestic violence surge on Super Bowl Sunday either before that 1993 story or in the 16 years since. No one ever has.
Yet the Super Bowl violence myth still circulates. It turns up every January, on the Web and elsewhere. I still get calls and emails about it. As Joel Best writes: "A bad statistic is harder to kill than a vampire."
One of his rules for spotting questionable statistics involves the general tendency of suspect statisticians to confuse frequency and severity. In reality, he points out, the worse things are, the far less common they are. For example, we hear all the time that so many millions in America "go to bed hungry at night." This summons up Depression-era images of skeletal children and wasted adults on the verge of starvation.
Yet that kind of hunger--the sort of epidemic found in Haiti or Zimbabwe or parts of India or Bangladesh--is virtually unknown in the United States. Granted, we have the occasional story of some penniless octogenarian subsisting on canned dog food; and granted, we have no shortage of poverty. But hunger is not the same as starvation, especially in a country where one of the most serious and widespread problems among the poor is obesity.
It may seem, Best declares, that we're bombarded by such statistics, "but the ones we encounter in news reports are only a fraction of all the numbers out there. They have been selected . . . tailored to shock and awe, to capture and hold our attention." Sometimes they're presented with hyperbole--"the worst disaster in U.S. history"--and sometimes with changing definitions--is everything that was a wetland still a wetland?--and sometimes with slippery measurements--like equating mean income with average income--and sometimes with peculiar percentages or comparisons.
One example of the latter which Best does not cite--and how could he have omitted it?--was the famous claim some years back that a single woman over 35 had more chance of being struck by lightning than ever getting married. Surely you remember the hoorah it caused; Newsweek even put it on a magazine cover. The statistic was hyped, and subsequently debunked; but it still haunts the psyches of the fearful because, like the Super Bowl violence myth, it appears to validate a certain kind of dread.
How else can you spot dubious data? Do they equate correlation with causality? (Masturbation causes acne!) Do they report emotion as fact? (Blacks believe whites are prejudiced!) Do they leave variables uncontrolled, take facts out of context, or grind an obvious axe? If they do, they may have been sown (as well as sewn) by the tailors of the emperor's new clothes. Or as Stephen Colbert assures us in the preface: "The statistics you don't compile never lie."
Ken Ringle, longtime reporter and cultural critic for the Washington Post , writes from retirement.