Stephan Shakespeare: Can a combination of experts and public opinion improve our decision-making processes?

Most of us share a fantasy: that really smart well-informed people meeting behind closed doors would make better decisions for the long-term national interest than a bunch of squabbling politicians grubbing for the votes of an unruly mass via our distorting media. How can childish parliamentarians playing to the gallery, booing and hissing in the Chamber, point-scoring on Newsnight then smooth-talking on the Today Programme possibly be an appropriate medium for serious debate and optimal resolution?

Michael Portillo once told me how in Cabinet he would look around the room and wonder if this group of people around the table were really running the country. He imagined there was a secret door leading to another room in Downing Street where "the real cabinet" was meeting. We probably all share a belief that somewhere there are Wise Ones who, if only we could let them get on with it, would sort things out properly. But where are they?

If we can’t have wisdom, at least we want "evidence-based policy". Well, I refer once again to the amazing evidence of Professor Philip Tetlock, an academic at the University of Pennsylvania who made a 20-year study of the forecasts of 284 experts from academia and practitioners, including viewpoints from Marxists to free-marketeers; 28,000 predictions were tested, and the analysis showed that they were only slightly more accurate than random, and worse than basic computer algorithms. It’s an eye-opening study and if you want to check it out for yourself, here’s the Amazon link.

If expert opinion isn’t reliable, what about public opinion? It may not be informed, deliberated or even truly measurable, and yet it dominates decision-making at every level of government: nobody wants to be seen going against what the voters appear to want. So is that a good alternative, can we trust in the "Wisdom of Crowds"? I suspect a scientific test of that proposition would look a lot like Tetlock’s study: slightly better than random.

"Slightly better than random" may not be too bad. Carry that on over very many different kinds of decisions by very many generations and you do get progress; you may agree with Professor Steven Pinker ("The Better Angels  of our Nature") that civilisation is steadily winning in spite of all the problems. In fact, progress is accelerating, because we do have much better tools by which to share information, experience and influence. The squabbles, swindles, pure foolery and mass manipulations balance each other out and act as a kind of channel for the accumulation of experience to take us forward.

I say this as we begin the second two-day YouGov-Cambridge Forum at Magdelene College this morning. Can we combine the Wise Ones with The Crowd to improve the process? That fantasy team of wise and experienced people in a secret room would probably look like our advisory board (including as it does Lord Griffiths, Jonathan Powell, Lord Wilson, Polly Toynbee and Jimmy Wales). And with our partners – Cambridge University, RUSI, the Guardian and the British Council – we are bringing smart, well-informed people together with in-depth opinion research towards our aim of creating an Institute of Public Opinion. If you want to follow the event, check out Twitter handle #YGC2012 or go to the live blog. Our participants include:

Conference Chair: Michael White

Speakers and panellists:

General The Lord Dannatt, Andrew Lansley MP, Diane Abbott MP, Lt Gen Sir Robert Fry, Lord Michael Hastings, Gary Hoffman, John Studzinski, Rt Hon Hugo Swire MP, Sajid Javid MP, Professor Michael Cox, George Galloway MP, Professor Andrew Gamble, Wael Ghonim, Moya Greene, Matthew Hancock MP, Fraser Nelson, Lord Dick Newby, Martin Gilbert, Professor Colleen P. Graffy, Rosemary Banyard, Nicholas Beadle, Julian Borger, Ian Cheshire, Martin Davidson, Sir Vernon J. Ellis, Jane Kinninmont, Chandu Krishnan, Zahra Langhi, Terry Macalister, Mohsen Marzouk, Dan Milmo, Richard Norton-Taylor, Irfan Siddiq, Martin Kettle, Jill Treanor, Michael Willis, Matthew Jamison, David Hearst, Polly Toynbee