Can you trust ConservativeHome's grassroots surveys?

There has been some questioning of ConservativeHome's polls of the 'Tory grassroots' recently and it is right that we respond to legitimate questions that have been raised, most notably by PoliticalBetting's Mike Smithson.

Stephan Shakespeare - co-founder of YouGov and owner of ConservativeHome - has written some extended thoughts below but let me briefly summarise some reassurances:

• The ConservativeHome survey began in December 2005 with a very accurate prediction of the outcome of the leadership contest between David Cameron and David Davis.  69% said that they would like David Cameron to be the next Tory leader and 31% said that they hoped David Davis would become Tory leader.  The actual result was 68% to 32%.  The survey was accurate because we recruited from organisations known to have Tory Party members - including the TRG, the Conservative Christian Fellowship and the email lists of the Davis and Cameron campaigns.
• The panel that made the accurate prediction has remained the reference point for all subsequent surveys. We look at how that initial panel voted and compare it with the answers of new respondents.  The differences have been incredibly small over the last four years but if a large number of UKIP or Labour supporters attempted to infiltrate the poll we would notice because the answers of people voting via publicly-advertised hyperlinks would be different from those who were members of the initial and accurate panel.
• Over the last year approval of David Cameron has been consistently over 90% (it has dipped lower in earlier stages of his leadership). This is not a panel full of UKIP supporters as is sometimes suggested.
• I have never made any claims about the representativeness of people who leave comments on this site. At various times there have been pushes by opponents of the Conservative Party to flood ConHome threads.  They can do that to threads but for the reasons I explained above they cannot corrupt the monthly surveys. 

We recently approached CCHQ in the hope that we could agree a mechanism for verifying whether people who say they are members are in fact members. I'\d like to agree this mechanism but it would only gold plate what we already have in the first bullet point above. Mike Smithson's suggestion of joining the British Polling Council had not occured to me until he raised it on Saturday. It is right that we make ConHome polling more transparent and BPC membership might aid that. Concluding our talks with CCHQ is my next priority however.

Another question is why poll at all? Does it matter what grassroots Tories think?  Is it helpful to know what they think?  Here is the answer I gave to that question in May 2006:

"The reason why ConservativeHome will continue to survey the Tory grassroots/ webroots, however, is our fundamental belief that the party is not owned by the current leadership or any leadership. Gone are the days when CCO can purport to speak on behalf of members to the press without fear of contradiction.  ConservativeHome believes that the leadership deserves loyalty and respect from the grassroots but loyalty and respect is a two-way thing.  The party must listen to the grassroots.  We are the people who raise the party’s funds and deliver the leaflets.  ConservativeHome’s first campaign was the successful championing of the membership's right to a vote in the leadership election and, much more recently, the membership’s right to be able to inspect the candidates’ A-list from which the next generation of Tory MPs will be drawn. ConservativeHome’s mission involves helping to build a more open, democratic and participative Conservative Party.  The voice that the Members’ Panel gives to the grassroots is a fundamental part of that mission."

Tim Montgomerie

Stephan Shakespeare: Can you trust ConservativeHome polls?

When pollsters of the general public try to demonstrate the validity of their results, they do so by comparing their sample to the characteristics of the population. This is easy to do in theory, but not in practise, because only some characteristics are known and measurable, and others are unknowable. We know a lot about the demographic profile of the public, and we know about the demographic profile of our sample, and using weighting we can make the two equivalent. But we cannot, by definition, know the difference between the attitudinal and behavioural characteristics of the two groups: we don’t have any way of knowing the difference between the people who take our surveys and those who, for whatever reason, choose not to. No methodology can randomly sample, pollsters at best can only randomly invite, and the majority of those invited - by any method – decline to participate. We can never know whether those who agree to participate have the same attitudes as those who don’t. It’s for this reason that new methodologies emerged which attained greater accuracy by better modelling.

Generally, we judge the validity of pollsters’ results by a combination of the rationale of their methodology and the accuracy of their results in those few instances where polls seek to describe behaviours that are objectively measurable (ie election results). Now apply that to ConHome surveys. It’s much harder, because there are no objective measures of the membership of the Conservative Party. Very little is publicly available about membership. So to what profile would one weight? And anyway, what does ‘the views of the membership’ really mean? Large sections of the membership play no normal ‘member’ role in the party. When we hear of the views of Conservative Party members, what are we really interested in? All those who for whatever reason continue to pay subscriptions even though they take no part in party activity? Or only those who have some effect the thinking and decisions of the party? Who matters? Who actually owns brand? Maybe one should only seek the views of the constituency chairmen? Or their committees? Or the national committee? Or the shadow cabinet? Or the inner sanctum? It depends on what we’re trying to understand. The purpose of ConHome surveys is to represent the views of people who see themselves as actively committed to the Conservative cause, but who have no other organised collective voice. This has a reasonably clear meaning, though admittedly it remains very hard to define the edges. It is difficult precisely because it is difficult these days to define the meaning of purpose of a political party. Can anyone explain exactly who had ownership of what? I intend to fall back on the simplest criterion, and the only one I can justify: a poll of the Conservative Party ought to be able to predict accurately the outcome of the one party activity which is objectively measurable: the vote for the leader.

I oversaw the publication of ConHome’s first significant published poll, on the eve of the election of David Cameron as party leader. I used exactly the same method as I did some years earlier when I conducted the poll for YouGov which predicted, with zero error, the result of the previous leadership election (then for the Daily Telegraph). I asked our sample about demographics and levels of participation in party activity (leafleting, attending meetings, etc) and matched them to the little information we had, an academic study of the party membership conducted much earlier (WHITELEY, P., SYED, P. AND RICHARDSON, J. 1994. True blues: The politics of British conservative party members), which I very slightly adjusted according to my own subjective notion of how it might have changed over the intervening period.

The result of ConHome’s leadership survey came within 1% of the actual result.

Since that time, ConHome has used an absurdly simple, but I suggest effective, method for ensuring the validity of its survey results. I have the original data from that first, accurate survey. Some of that sample has disappeared over the years, but most has remained stable. Periodically, I have compared the results of the current total ConHome sample with the subgroup of the original sample: it still has the right ‘Voted-David-Cameron’/’Voted-David-Davies’ split, and the difference between the sub-sample and the total sample is negligible. Therefore we have concluded that the results from the unweighted ConHome sample is a valid representation of the views of Conservative Party members, as near as can reasonably be done.

It should be noted that the original core sample for the survey was recruited via invitations from various organisations within the Conservative Party, notably the email lists of the David Davis and David Cameron campaigns. The argument that the survey respondents consist entirely of the audience of ConHome is therefore quite wrong. It is drawn from a variety of constituent parts of the grassroots organisation. It is also wrong to assume that the respondents 'represent' only the audience of the ConservativeHome site, or indeed that the ConservativeHome site is 'represented' by the people who leave comments or by the editors. The site has become something of an institution within the party and is used a a message-board for all sections of the party, as anyone who has been a regular visitor will know very well.

The methodology is imperfect, as it necessarily must be given the undefinable nature of "Tory grassroots" or even "Conservative Party". However, things change, and the status of ConHome has considerably changed, and the role of its survey has developed. It is therefore right that Mike Smithson raises the questions he has, and ConHome will now formalise its methodology. This will take us a little time, and we welcome input from any interested parties."