Opinion polling ahead of the 2017 Turkish referendum

A few words on Turkish polling as this website resurrects the opinion poll tracker for the April referendum

Bookended with unexpected election victories by David Cameron and Donald Trump, the past year-and-a-half have been tough for pollsters. But observers of politics in Turkey have contended with troublesome polls for years.

This website has a long and generally frustrated relationship with Turkish opinion polls. Remember when SONAR put CHP ahead of the AK Party in a 2010 poll, or how Konsensus grimly predicted a parliament without the MHP in 2011, or the first Gezici poll after the 2013 Gezi protests?

These were polls that charted a shift in the public mood and made for good stories.

The trouble was that there were limits to how much these polls could be trusted. Granted, all three of the companies named above are owned by reputable pollsters who have been running surveys in Turkey for many years.

But Turkish pollsters are not bound by the ethics of the trade that help us understand if western polls are reliable.

UK example

Take Britain, for example. The main pollsters there that put out political surveys today – Comres, ICM, Ipsos Mori, YouGov, and the like – keep to what they call “standards of disclosure”. For each major public poll, you can look up the specific questions that were asked and break down the responses by certain broad categories: age, gender, location, and so on.

The pollsters do this as members of the British Polling Council (BPC), whose primary objective is to ensure consumers “have an adequate basis for judging the reliability and validity of the results”.

Turkey’s BPC equivalent – a body called TÜAD – plainly does not have the same hold over the polling industry. Its vision and mission page makes only a skirting reference to “increasing the quality of surveys”; in a classical bit of Turkish prose, its primary objective is to raise its members’ reputation. But worse, the polling companies that most casual observers of Turkish politics would recognise – ORC, Metropoll, AKAM – are not even TÜAD members.

The point is this: Turkish public polling operates with fewer quality standards. This makes it harder to assess an individual survey’s methodology or compare it when a competitor produces a radically different result. (I say public polling, because there are Turkish pollsters that put out surveys exclusively to their paying subscribers. The quality of these surveys is dictated by those subscribers – if they weren’t happy, they wouldn’t pay.)

Partisan polling

The dire quality of Turkish polling is not a new problem. Only a handful of pollsters publish the precise questions they ask. None release data tables of their results, like they do in Britain.

There is a lot of polling data out there and much of it is heavily partisan. The Objective Research Center (ORC) brings out the irony in its name by historically churning out results that are very friendly to the government. MAK is in the same bandwagon.

Gezici and Sonar, meanwhile, tend to occupy the opposite end of the spectrum.

How to compare them?

Two years ago, this website came up with a way of trying to squeeze something useful out of the publicly-available data: the rolling average of opinion polls.

The drill was incredibly basic: we took the results of the five most recent voting intention surveys, weighed it down with the previous five results to help dampen any extremities, and churned out the result on a regular basis. That was it.

For this year’s constitutional referendum, the rolling tracker is back – but not without controversy. Here’s how it was announced on Twitter (but please note a correction below):

The No campaign has a narrow lead in the Turkish executive presidency referendum, our opinion poll tracker suggests https://t.co/S47sy8lMXb pic.twitter.com/LAvEjTEvyJ

— JamesInTurkey.com (@jamesinturkey) February 27, 2017

Predictably, it was hailed by the No camp and dismissed as the act of a foreign agent by Yes supporters. It also triggered some constructive criticism on the reliability of the underlying data – that is, the poll results.

Given the lengthy preamble above, it’s worth making a few quick disclaimers about our own poll tracker:

• The polls that feed this tracker are not uniform. Each pollster has different methodologies and biases. These are not immediately balanced out: we don’t have an anti-government poll out the same day as a pro-government one.
• The tracker is based on headline results only. Pro-government pollsters in particular avoid publishing the number of undecided voters. The tracker sadly cannot compare figures that don’t exist.
• Mathematical minds urge caution. People who can crunch numbers far better than me warn that there’s limited value in comparing such disparate data. See the warning from Erik Meyersson of the Stockholm Institute of Transition Economics on this point.

The show goes on

Despite this, this website will publish and update the tracker in the weeks between now and 16 April. Here are a few points I would make in its defence:

1) The tracker is curated by me, a journalist who has followed Turkey for many years

I include pollsters that I know have been in the field for at least as long as I have. I approach any new kid on the block with extreme care.

Take THEMIS, whose YES (42.4%) NO (57.6%) poll last week was widely covered in the anti-government press. This is a company that I don’t think existed before 2017 and, until I learn a little more about who they are, they won’t be in the tracker.

2) The tracker formula, although basic, dampens extremities

This scatter chart shows the basic poll results we’ve had since the turn of the year.

You’ll see January and February have produced polls that alternately put Yes or No in the upper 50% range – or even the low 60s. Our tracker attempts to dampen these by weighing down recent extremities with earlier results.

3) It’s worked before

The tracker was fairly effective at identifying the changing trends in Turkish voting ahead of the June 2015 general election, when support plunged for the governing AK Party and correspondingly rose for the far-right MHP and pro-Kurdish HDP.

Ahead of the November 2015 election, it showed a rise in AK Party fortunes and a drop for the MHP and HDP – although, like just about everyone else, the tracker underestimated the magnitude of these.

Essentially, the tracker is as good as the polls fed into it. Only time will tell whether it will be a useful measure of the changes in public opinion during the referendum campaign. Just don’t use this to bet your house on the referendum outcome.