Nicholas Kristoff starts off his column today with a question, "Ever wonder how financial experts could lead the world over the economic cliff?" His answer is that experts don't really know what they are talking about. He cites the research of Philip Tetlock. Tetlock finds that experts' forecasts on a variety of topics is only slightly better than random forecasts.
The expert on experts is Philip Tetlock, a professor at the University of California, Berkeley. His 2005 book, “Expert
Political Judgment,” is based on two decades of tracking some 82,000
predictions by 284 experts. The experts’ forecasts were tracked both on
the subjects of their specialties and on subjects that they knew little
about.
The result? The predictions of experts were, on average,
only a tiny bit better than random guesses — the equivalent of a
chimpanzee throwing darts at a board.
difference whether participants had doctorates, whether they were
economists, political scientists, journalists or historians, whether
they had policy experience or access to classified information, or
whether they had logged many or few years of experience,” Mr. Tetlock
wrote.
The only thing worse than an expert, he writes, is a famous expert. Famous experts, ie Jim Cramer, make worse predictions than the chimpanzee and the dart board.
Meanwhile, the political scientists at the Monkey Room have been loudly complaining about the lack of expertise in political journalism, in response to an article by Matt Bai. Bai wrote that political science experts know less than journalists about the world and political affairs. Maybe Kristoff saw that debate. Or maybe he's smarting over recent criticism of his columns.
Does my PhD and years of teaching/research give me any edge of understanding politics? How about all you economists and business leaders? Do education professionals know the best way to educate kids? When I taught at a premier teachers college, students were horrified that parents and average citizens could have a hand in crafting education policy. Do you need a PhD to know that your kid learns better one way and not the other?
Expertise does a much better job at explaining the past than it does in predicting the future. It provides a context for present predicaments, it provides various lens for understanding the present, it is able to provide options for future plans, and it can take certain options off the table based on past experiences. Experts shouldn't be in the business of crystal ball reading.
Any expert worth his salt has to admit his own ignorance. Like Socrates, we should know the limits of human wisdom.
Are we in the midst of a mini-populist revolution here? The masses are crashing the gates of academia and high wage financial CEOs. They are taking apart the media pundits. They taking back bonus money and e-mailing YouTube clips of Jon Stewart. Part of me is highly amused. Part of me is shifting in my seat.
Apt. 11d 
I hate it that Kristof doesn’t have cites in his articles (does he have them somewhere?).
Say, for example, the study that shows that rats are better than Yale undergraduates at finding food in mazes does not address the question of expertise/prediction. My guess is that rats are indeed experts at finding food in mazes, while Yale undergrads are not. I’m pretty firmly convinced that the Yale undergrads can write a better essay than the rats.
My guess is that experts are better at predicting outcomes when we actually understand the system, and not better when we don’t. It’s not the individual that matters, it’s the body of data they’re relying on. For example, I think a physics expert would do better than chance at predicting trajectories of a ball tossed in the air (other experts might do better, but I think that there are experts who will do better than random).
The cite to clinical psychologists and their secretaries would be useful, too. One possibility is that clinical psych is a meaningless body of knowledge (at least in its predictive power). Another possibility is that the secretaries are also experts, with their expertise gained through non-formal education. Do we think that a oncologists secretary would be just as good at diagnosis, too? (I don’t).
LikeLike
Well.
Analyzing expert predictions of election outcomes strikes me as somewhat like meteorological predictions, or stock market predictions. These are very, very complex systems. No one fully understands them.
Does that mean experts never know anything? Take a process that’s less complex than global weather but still pretty f’ing complex, such as the oncology that bj mentions, and I think we’ll all go with the expert thank you very much.
I think we are in the midst of a mini-populist uprising, but IMHO that’s more a reaction to *wanting* everything to be simpler and just as accessible to the ordinary guy, than to any sort of statement about life *actually* being simpler. There was also a populist uprising in the US as we transitioned to mechanized agriculture in the late 1800s.
LikeLike
I do think that experts have a role to play in almost any field, but it’s not in prediction. No one can predict much of anything, I’m convinced. But can I provide more information about how to use technology effectively in education than a random person off the street. Yes! Can you explain the 2008 election or the ins and outs of government better than me? Yes! But I think there’s a role for non-experts in almost everything too. I think it’s important for non-experts to share their thoughts about something, to express concern, to contribute what they can to the conversation. I often find that non-expert think outside the box more often than the experts.
LikeLike
I think the question is somewhat irrelevant to the current mess. I wouldn’t describe what the quants do as prediction, except in the very formal sense of statistical prediction, which is not at all what Kristof means. Prediction, in the soothsayer/Buffett sense, is more in the realm of analysts, who did not come up with convoluted financial products that caused so much trouble.
LikeLike
Prediction is mostly nonsense, whether it’s experts or psychics doing it.
On the other hand, experts who are actively wrong about what they’re doing or knowing right at this moment are a different issue, especially when they lack the humility to admit it or examine what was missing.
I did note that Kristof closed with a plea to hold experts (including himself) accountable without saying what that means, exactly. What is he prepared to do when he gets something demonstrably wrong?
LikeLike
Kristof’s invitation to follow him on Twitter makes me doubt HIS credibility.
LikeLike
Quit the Times and take up sackcloth and ashes? hahahaha
Quite clearly, “hold accountable” means “occasionally embarrass a little bit before business as usual continues.” Or is James Cramer no longer on CNBC?
LikeLike
“Prediction, in the soothsayer/Buffett sense, is more in the realm of analysts, who did not come up with convoluted financial products that caused so much trouble.”
But wasn’t there another scandal we’ve forgotten in which analysts were giving positive “buy” ratings to companies that their investment arms were investing in (at the big financial houses)?
LikeLike