Let the Wild Rumpus Begin!

Over the weekend, bloggers and columnists have been trashing Steven Levitt and Stephen Dubner's new book, SuperFreakonomics. Brad deLong, Ezra Klein, and Paul Krugman trashed their findings that drunk driving is safer than drunk walking and that scientists use the wrong solutions to deal with global warming. The authors respond here and here. 

Andrew Gelman discusses some reasons why their book contains such shoddy statistical work. He wonders whether blogging could have interfered with their scholarship. They spent too much time online and believing their adoring commenters. 

I think the failure of this book has more to do with the publishing industry, which is only interesting in publishing "rogue" academics. Academics don't sell books, unless they can package themselves as controversial or paradigm smashers. Serious, traditional academic work is too nuanced and detail oriented to appeal to a mass market. And most academics aren't very charismatic. I'm sure there was a lot of pressure on the authors to be as controversial as possible, even if their conclusions were questionable.

Were these chapters actually peer reviewed or did they go straight to the book?

Comments

[Julie G.]

Dude. I don't know what you're talking about, that academics aren't charismatic.

[Wendy]

I've been thinking about this wrt Nurtureshock (and even Outliers). Do publishers value only the stuff that is "rogue" and "mavericky," or do these books hold up?

I liked Nurtureshock a lot, but I couldn't help being distrustful of it for exactly this reason.

[MH]

Were these chapters actually peer reviewed or did they go straight to the book?

How does peer review work for a book that is expected to generate income? For journals, I do it because it is sort of a 'citizenship' issue if you publish yourself. But, working anonymously for free on somebody else's best seller would be different. I suppose the publisher must pay, but even then I doubt you can send 'Revise and Re-submit' to a publisher and have them pick your view over the guy who made them money on the last book.

[Janice]

I'm pretty certain that these chapters weren't peer-reviewed in any way, shape or form. I expect that a lot of topics were brainstormed between authors & publishers and every one had to "up" the ante on shocking insights as they readied for press.

[MH]

I want to be un-peer-reviewed. Or at least to find a peer reviewer who isn't so worried about stepwise regression.

[Julie G.]

For academic books, the process involves paying several reviewers and a great deal of editor discretion. I imagine there is less scrutiny for income generating books, given the market concerns. Since most publishers of academic books are nonprofit, operate at a loss, or incorporate other profit generators to buffer their academic lists, there can be more attention to the quality of the substantive input.

[bj]

"I liked Nurtureshock a lot, but I couldn't help being distrustful of it for exactly this reason."

Yeah, me too. I've now read several of the chapter, some of which cite some dramatic results. But, they do so is a suspicion raising way -- for example, they have a chapter on sleep, where they blame lack of childhood sleep for obesity & poor academic performance. They then cite a statistic saying that a high school that delayed its start time by 1 hour saw a 20+ point rise in the SAT's scores (of the top 10% of the students). Suspicious, first, 'cause they're subdividing the data. Second, because they've added a bibliography to the book, but they're not cited in the text, so it's impossible to tell which particular manuscript contained that particular piece of data. To find it, you'd have to guess and look through all the cites at the end of the chapter, and some of them are to abstracts, newspaper articles, and casual reports.

I'm going to spend some quality time on 3 chapters in the Nurtureshock book tracking down legitimate cites (on sleep, intelligence testing in early years, and the praise chapter).

Bronson & Merryman (Nurture shock) are not academics; they're journalists. The format of their book probably doesn't violate journalist ethics. But, the Freakonomics dude is an economist. If he's been amping up the information for shock value, he is violating "academic ethics" (whatever they might be).

[MH]

"academic ethics" (whatever they might be).

In medical research, most of those concern parking and who gets listed as author (and in what order).

[Ragtime]

At some point, politics became all about sound bites. That was a bad thing, but even worse is when everyone agreed that politics is all about sound bites, which suddenly made it okay to attack or defend little sound bites.

It sounds to me that everyone is screaming about "carbon dioxide is not the enemy here," because it has "sound bite" potential.

Whether it's true, or out of context, or mis-representative is besides the point because everyone can instantly imagine the politician on the floor of the House saying, "Even the authors of Freakonomics, who believe in climate change, agree with me that carbon dioxide is not the enemy."

So, now, instead of debating the correct policy to deal with climate change (which the chapter was about), it is now a debate about the sound bite -- which is just stupid.

[laura]

In academia, articles get far more scrutiny than books. But in mainstream publishing, books are reviewed very, very loosely. My guess is that Levitt's book was reviewed by whoever has the blurbs on the back cover of the book. Those guys were more interested in getting their names on the back cover than giving an honest assessment of the book. My understanding of the first books is that each chapter had been through a rigorous vetting through journals and conferences. This one didn't go through that process.

[Marya]

I've been suspicious of them since I read the baby Name Wizard's take-apart of their chapter on names. They used urban myths as name sources, and "predicted" the popularity of names that were already super-popular.

[Doug]

"more attention to the quality of the substantive input."

For fairly specific definitions of the word quality.

Would it be fair to say that DeLong et alii are Superfreaking out?

Daniel Davies finally finished his five-part review< of Freakonomics. Just in time. (The rest of the blog is terrifically invigorating, too.)