The Wisdom of the Crowd: When is Collective Decision Making Better than Asking the Expert?
I just finished an intriguing book, Think Twice by Michael Mauboussin. The author is an investment strategist with a large firm in the financial services industry as well as Chairman of the Board of Trustees of the Santa Fe Institute. In the book he makes the case for better decision-making by “harnessing the power of counter intuition.” Sounds like a mouthful, but he raises several good points including one that deals with what is commonly referred to as “the wisdom of the crowd.” He points out that decisions made based on the collective wisdom of a group of individuals are frequently better than the judgment of an expert in the field of interest. When the decisions have probabilistic outcomes (as opposed to rule-based outcomes) and when there are a wide range of choices the wisdom of the crowd typically prevails.
Mauboussin cites the work of Scott Page (the professor, not the saxophonist) and replicates the following equation in his book related to the diversity prediction theorem:
Collective error = average individual error – prediction diversity
The average individual error is a reflection of ability, and the prediction diversity reflects how different the individual answers are. The equation basically states that if you have a group of incentivized, or otherwise motivated individuals, with diverse backgrounds, the crowd will always predict more accurately than the average person. Yes that is always, not sometimes. Of interest, the collective is often better than many of the experts in a particular field. In the words of the author, “This is not good news for experts and it is deeply humbling for all decision-makers.”
As I was mulling this over, a friend informed me he was participating in an office NCAA tournament pool. You are probably familiar with the idea, which is typically frowned upon by senior management, but goes something like this: Participants donate, say their lunch money, and then pick the teams they believe will win each of the 63 single elimination games played during three successive weekends in March and April. One of those teams will win six games in the tournament and be crowned the National Champion, which coincidentally will occur in Atlanta, Georgia, tonight.
My buddy said there were almost 30 people participating in the office where he works. Is there a place for the diversity prediction theorem here? We have an incentive (the lunch money), a diverse group of individuals and, likely, some ability in this group. I asked him to provide me with access to that sports’ authority, ESPN, where the groups’ picks were housed. Then, I walked through the following exercise:
- During the first two days of the tournament there were 32 games. Collectively ,who did this group of almost 30 souls think would win each of those games?
- Using the averages, 21 of 32 were called correctly and, not surprisingly, this 21 out of 32 was right in the middle of the pack for those who had donated their lunch money to the cause.
Over the next two days, another 16 games were played to determine the “Sweet Sixteen” and on the following weekend after 12 more games were played, the field was whittled down to the “Final Four.” Imagine my surprise when I walked through this exercise and found the collective wisdom of this group of participants scored better than 85% of the original participants through the end of the first two weekends. What is perhaps more impressive is how this isolated collective beat the pants off of several experts in the field. (Our good friends at ESPN post a celebrity and expert page for this tournament and, yes, even the President of the United States participates in this annual ritual.)
Granted, this is very small sample, and the results may be more related to chance than the wisdom of the crowd. But for a moment, think about this counter intuitive approach to decision-making.
Crowd sourcing has become prominent in several arenas. One example in the world of informatics is Kaggle. Toss your data analytics problem into the world of Kaggle, and if the incentives are appropriate, you could have several solutions to choose from.
What about closer to home? As nephrologists, we are often considered the experts and are frequently called upon to make a decision. Is there room for considering the wisdom of the crowd in our world?
In Mauboussin’s opinion, we would be wise to consider the collective when we face problems that have a wide range of outcomes and are probabilistic in nature. Clearly, this is not the case when we are considering how to deal with a potassium of 7, but what about the therapeutic regimen in an unusual case of glomerular disease? Or the best course to take in a complicated renal transplant patient? Perhaps, we would do well to “think twice” in circumstances such as these.
Can you think of circumstances in nephrology where the wisdom of the collective might be useful? If so, drop us a comment and join the conversation. Oh, and about that game tonight, the collective wisdom of my buddy’s office pool had Louisville facing Indiana in the Championship. The Hoosiers did not make it to Atlanta, but will the Cardinals prevail? Enjoy the game.