The business of polling

DHANANJAI JOSHI and SAM SOLOMON

back to issue

THE Indian pollster is beset with many challenges. Designing and obtaining a sample of respondents who are representative of India’s population, assembling and training a team of field investigators who are committed to collecting accurate data, and effectively communicating the results of a survey is not simple. Being in the business of polling adds layers of marketing and selling to clients who are often unfamiliar with the proper interpretation of survey data. All this happens in an environment where one wrong call – such as predictions by many pollsters that the NDA government would be re-elected in the Lok Sabha elections of 2004, or exit polls projecting a tight race during the Bihar Vidhan Sabha election of 2015 – can lead to a public backlash against polling and damage reputations.

With the explosion in television media in the last decade, election polling proliferated across the Indian airwaves and commercial pollsters began to emerge in large numbers. The final day of any election now brings a flurry of exit polls which are discussed and critiqued by large televised panels in English, Hindi, and India’s many other languages. This article offers a brief tour through some of the considerations related to the costing, logistics, and sampling that lie behind much of the election survey research in India.

 

The entry of multinational polling companies such as Nielsen, TNS and GFK Mode in the 1990s gave an impetus to India’s polling industry. For most of these multinational agencies, studying elections was a small part of their portfolio, as their main focus largely remained market research. Election night forecasting for market research firms is primarily to showcase their capabilities to use sophisticated methods to forecast something as complex as elections. The biggest sponsor of election polls continued to be the media industry until the early 2000s, when political parties and individual candidates in India started realizing that surveys are an effective instrument to receive unbiased feedback from the ground that can then be used for electioneering.1 Over time, many domestic research firms sprung up to garner a share of the growing market.

The sudden mushrooming of survey research firms created its own set of problems, the biggest being a lack of focus on the methodology employed in election surveys. On the one hand, market researchers often inadequately sensitive to the nuances of political research – sample designs that are representative of India’s social and political demography, question wording, and so on – conducted election polls. Simultaneously, fly-by-night operators who otherwise functioned as manpower aggregators supplying field researchers to firms also started their own operations, expectedly without any understanding of methodological issues.

Even something as basic as a dummy secret ballot to satisfy respondents’ wish for privacy when disclosing their vote is not a standard practice used by most pollsters in India. Dummy ballots are essential in India because many social groups do not enjoy the liberty to publicly support or oppose a party or candidate of their choice.2 For instance, in rural India, women seldom go against the political choice of their families in public, and it is rare to find voters from weaker sections of society challenging candidates from dominant communities. Universal adult franchise came with the right to secret ballot in India precisely for these very reasons.

 

The industry mostly works on a cost-per-interview (CPI) model. Typically, if a sponsor receives ten quotations for a cross-sectional survey, these can range from below one hundred rupees to five hundred rupees per interview. What accounts for this wide range in pricing? The main reason is that fly-by-night operators only collect data from the field and have minimal overheads and operating costs, while more rigorous firms have sizable overheads towards infrastructure and staff that are over and above the field cost per interview.

Three variables determine the calculation of basic field costs. The first is sample size, the larger the sample the lower the CPI. This is not only because of the simple logic of economies of scale, but because a larger sample size ensures a greater concentration of respondents within a given geography. Whether by way of more respondents at a given sampling point or by way of more sampling points, the cost of travel and logistics involved in moving from one location to another increases the budgets. The second is weight of the instrument, or in other words, the number of items a questionnaire has. This has a direct bearing on efficiency and productivity of the investigators; the heavier the weight, the fewer the number of interviews per person day. The third is timelines, and when the turnaround time is less, a firm not only needs to employ more investigators but also to invest in their supervision and logistics management that pushes the costs northwards.

 

Election surveys require meticulous sample designs and reaching the last respondent can be an expensive affair, especially in remote locations. Such a wide range in quotations often confuses sponsors and this makes it challenging to convince them that good surveys come at a cost because they are expensive to conduct. The dilemma for sponsors, especially among the media, is to decide which vendor deserves a premium. It is not uncommon to hear voices from within the media say that every firm in the market has, at different times, got polls wrong and thus it is only for the viewership on election night that polls are essential. This is a problem related to the larger issue of expectations from polls. For over a decade, rigorous pollsters have tried to drive home the point that a survey only gives an estimate, which is subject to error. Unfortunately, the only point that determines success to media sponsors is getting the seat forecast right. Sample size rather than sample design has regrettably been the focal point.

The case with political parties as clients is different, since one of the main criteria for choosing a survey vendor is trust and confidence that data will not be leaked. Political parties do not use survey data as extensively as they should. Yet, the growing faith in sample surveys for effective electioneering is a welcome change for the polling industry as more and more politicians engage pollsters to receive unbiased feedback. And although surveys are not a decisive factor in crafting strategies, younger politicians, usually more data savvy, are positively inclined. As these politicians ascend through the ranks, their views acquire greater weight, presenting a great growth opportunity for the Indian polling industry.

 

Firms that depend largely on political polling and election surveys struggle with the seasonal nature of their business. Sustaining a business in days when there are no elections is a constant challenge, and diversification to other large-scale surveys has been a fruitful strategy for many companies. However, the seasonal nature also affects the way companies engage field investigators. Election surveys and especially exit polls require large-scale mobilization of manpower. This manpower is generally arranged by engaging freelance supervisors who act as contractors, managing anywhere from 10 to 20 investigators.

An exit poll in a state typically requires anywhere from 300 to 400 field investigators. Once the election is over, investigators rarely interact with the company that hired them. This system of mobilization, though cost effective, significantly compromises fieldwork quality as field investigators are insufficiently trained. Also, political polling requires a very different involvement as compared to consumer feedback surveys.

An alternative to the subletting model is to directly recruit field investigators by placing advertisements in newspapers and posters and notices in universities and coaching institutes. Walk-in interviews with large numbers of people, group discussions, and one-on-one interviews are carried out to recruit investigators. Experience of conducting surveys for different agencies is considered a liability rather than an asset. Investigators are trained in a day-long workshop and briefed about the hardships of fieldwork. Often many people opt out before hitting the field. This is a costly method of recruitment, but it does ensure better quality of fieldwork.

It becomes difficult to meet these costs when the duration of fieldwork is limited to a few days. But in cases where an election is tracked by different waves of polling, this method of mobilizing manpower is more cost effective. Here too, employee turnover is high; if one were to maintain a workforce of 150 for a duration of six months, it is likely that only twenty per cent of the investigators would continue until the end of the project. Finding women investigators who are willing to work in the field is another challenge. Moreover, ensuring their security when travelling to remote locations is difficult.

 

Recruitment of quality manpower is a costly and constant process, and the cost differences between the ‘subletting model’ and direct recruitment are significant. But hiring the right pool of field investigators and investing in their training and motivation is essential for getting good quality data. Using GPS trackers for field investigators, tagging GPS coordinates of sampled locations and moving from PAPI (paper and pencil interviewing) to CAPI (computer assisted personal interviewing) are some ways to keep the logistics in control and maintain quality. Indeed, running a survey operation in a large and diverse country such as India is more like running a logistics business.

While managing large-scale logistics and well trained manpower along with the use of the right technology keeps the operations smooth, this comes with its own set of challenges which in many ways are peculiar to the election polling business. The total engagement between a company and its field investigators for a typical exit poll project is not more than 72 hours, yet obtaining GPS tracking SIM cards for a few days is not possible. Transporting tracking devices or handheld devices for CAPI is expensive, and wear and tear is particularly high given the high rate of employee turnover among field investigators.

Clients often find it difficult to accept that good surveys come at a premium. Some pollsters have turned to CATI (computer assisted telephonic interviewing) as an alternative. CATI is much more cost-effective and as there are no fieldwork logistics, the errors that creep in at the level of data collection are also reduced. However, the sampling error is big, which is very problematic in the case of election surveys. Obtaining a representative cross-section sample with CATI as the mode of data collection is also challenging. Although mobile phones have penetrated rural as well as urban India, phone usage patterns among different sections of society vary to a large extent. CATI data often tends to over-sample upper castes and richer individuals as well as urban areas more severely than data obtained through PAPI and CAPI methods.

 

If representativeness is the key focus of an election poll, then compared to CATI, PAPI or CAPI seem to be more reliable ways of data collection. However, in terms of cost, the difference is huge. Although refusal rates are higher in CATI, costs towards field operations are none. The compromise on methodology also extends to the absence of a dummy ballot. But there are benefits that go beyond mere costs. All interviews are authentic and recorded. There is no fudging of data at the investigator level, which is a problem with contractual manpower. The firm is spared the sheer pain of large-scale logistics management, which in itself is a big relief. The infrastructure costs are high, but the same call centre seats can be leased out when there is no business and used in night shifts for conducting polls in other parts of the world. These benefits along with the bonus of cost efficiency make CATI an attractive method, but the overall compromise for election surveys is something to beware of. For now, CAPI with GPS tagging of locations and respondents seems to be the most suited method of data collection when it comes to election polls in India.

 

India’s dense demographic mix creates multiple obstacles in ensuring a sample that represents the Indian public, whether the methodology is PAPI, CAPI, or CATI. These obstacles are most pronounced in surveys not using probability based sampling.

Many researchers cite difficulties in conducting interviews with women respondents due to widespread bias against women participation in interviews and discussions of politics. In addition to the challenge of reaching women respondents, field investigators often must conduct interviews with women respondents in the company of respondents’ fathers or husbands. Women respondents sometimes look to their fathers or husbands for guidance; in other cases, a father or husband interrupts his daughter’s or wife’s response and answers questions for her. Field investigators must be adept at managing such situations.

This systematic bias in interviewing means that researchers often conduct a smaller share of interviews with women than intended. Researchers try to work around these challenges by employing more women investigators, who are better able to attain access to women respondents. This is especially the case for Muslim women respondents, who are especially difficult to survey. Even with the use of women investigators, research organizations often fall short of the proper share of women in the sample. Thus, they must often weigh up the share of women and weight down the share of men in the final sample.

Similarly, the upper class neighbourhoods in urban centres are among the most difficult to survey. Many of them have guards and gates to keep out people. On the other hand, people at the bottom end of the socioeconomic spectrum too are less likely to be selected for interviews. Moreover, contacting and interviewing selected respondents from the lowest castes can prove challenging. However, systematic bias in selection and participation on the basis of caste varies widely from state to state. Yashwant Deshmukh of C Voter explained how the politics of caste differs in different states:

‘In certain areas, the Dalit response rate is lower… it is directly proportional to how politically empowered the Dalits are in those areas. Before Mayawati, the Dalit response rate in UP was very low. But not anymore, because Dalits in UP are now wearing their identities on their sleeves.’

Addressing these local realities requires research directors as well as field investigators who are familiar with the social background of the different states. The lack of reliable data regarding the caste breakdown at the state and region level means that different research agencies will have different benchmark figures to weighting their data to adjust it to construct a representative profile of the population.

 

To the frustration of many pollsters, media coverage of polling is primarily focused on seat projections: how many seats will each party win? It is expected that sponsors and consumers of polls are most eager to know which party will win an election. However, this excessive fixation makes the jobs of pollsters more difficult, both because of the uncertainty surrounding vote share projections, as also the even more uncertain craft of converting votes into seats based on a survey sample. This uncertainty adds to the frustration, because all the valid political information that polls generate is written off if the final seat forecast is not accurate.

 

As examples, take the cases of the recent Delhi and Bihar elections where pollsters went wrong in predicting seats. The BJP was expected to win the 2015 Delhi assembly elections. A tracker poll series commissioned by the India Today Group to Cicero started in November 2014, when the BJP was in the lead. Eventually, as the election came closer, the gap between the BJP and AAP shrunk and signs of polarization on the basis of socio-economic class were reflected in the data. The BJP’s trump card of declaring Kiran Bedi as its chief ministerial candidate was expected to swing the election in its favour; initially it even made AAP strategists and leaders nervous. A snap poll was conducted within 48 hours and to everyone’s surprise, the move boomeranged and the BJP slipped behind AAP in terms of vote share.

As the election came closer, the pre-poll conducted by Cicero in the last week of January 2015 predicted a nine percentage point lead for the AAP. On election night, though Cicero and some other polling firms predicted a sweep for the AAP, no one came close to projecting 67 out of 70 seats. Unsurprisingly, though there were a few within the media who congratulated Cicero for capturing the trend on time, the overall sentiment was that the polls were off. Cicero was asked to give a public explanation of why its poll went wrong even though AAP was projected to win with a two-thirds majority.3

Similarly, in the Bihar assembly election, the BJP-led alliance was expected to give stiff competition to the Mahagathbandhan comprising of RJD, JDU and Congress. Initial polls indicated that the BJP was ahead in the race, but as the election came closer, the gap between the BJP and the Mahagathbandhan started to narrow. The talk in Lutyens’ Delhi was that Nitish Kumar was driving the rise of the Mahagathbandhan and that Laloo Prasad Yadav’s RJD was getting a new lease of life by piggy-backing on him.

Although the polls did not contradict the positive image of Nitish Kumar, survey data repeatedly indicated that the real backbone of the alliance was the RJD’s Muslim-Yadav support, which brought the alliance to a threshold of viability. However, most polls failed to capture the Mahagathbandhan’s lead over the BJP and projected a photo finish. Eventually, the result was very different from the forecasts as the Mahagathbandhan swept the polls. Undoubtedly, pollsters need to introspect as to why they failed to project the prevailing trend. But the fact that polls brought to light some key insights into what shaped the result has not been given any adherence because the final forecasts were wrong.

 

Forecasts can go wrong for many reasons. The process of translating the vote share of a survey sample into the seat share of a legislative assembly is a challenging one. Yashwant Deshmukh echoed this need for caution when reviewing seat projections:

‘Everybody has their own algorithm or mathematical formula to do that conversion. But that’s not foolproof. That’s not proven. That’s a work in progress always, even in the oldest of the First World democracies. We have seen in the 2015 UK elections that even they could not convert it correctly. They could not sense that the Tories would get a complete majority. And they have been doing polls for seventy or eighty years now.’

It is important for media to understand the limitations and complexities of polls. Changing vendors is never a solution; perhaps it would do more good if media sponsors devote some time in understanding the ‘power’ and ‘limitation’ of election surveys. More investment is needed in the area of research and development to minimize the errors in designing surveys, collecting, processing, and finally analyzing data. This is only possible if media sponsors are invested in their survey vendors – if not financially, then at least in the idea. Although sections of the media understand data and empathize with the difficulties pollsters face, the general sentiment is that polls are not to be taken seriously.

 

The growing number of polls of varying quality has often led to cynicism about the entire exercise among the general public. Periodic high-profile mispredictions, such as the incorrect call of an NDA win in the 2004 Lok Sabha elections, act as a platform for critical news coverage of polling. Against this backdrop, what is the future of the business of polling in India? What can be done so that sponsors and consumers of polls are better placed to sort the wheat from the chaff?

The threat of so many fly-by-night operators means that it is in the interest of polling firms which conduct methodologically rigorous surveys with well-trained and experienced field teams to cooperate with one another. The empowerment of a deliberative body of organizations which meets certain standards of methodology and transparency and can certify other such organizations has worked as a model for self-regulation of the polling industry in many countries. Examples include the British Polling Council in the United Kingdom or the American Association of Public Opinion Research’s Transparency Initiative in the United States.

 

Indian pollsters have recently come together to establish the Indian Polling Council (IPC), a self-regulatory body focused on fixing the fault lines within the polling industry. Promotion of best practices, consumer education, capacity building and safeguarding the interests of the industry fall in the mandate of IPC. To start, a format of compulsory disclosure is being designed, where some key information about the survey will be made public. This includes exact sample size, method of sampling, method of data collection, dates of data collection, sample profile, margin of error (computed on a standard formula), etc. Six leading companies and organizations in the Indian market: CFore, Cicero, Lokniti-CSDS, CVoter, IPSOS and Nielsen are working together on the format.

If too many people still see polling as a dubious exercise in obfuscation and influence peddling in India, then hopefully by establishing some standard protocols of disclosure for India’s pollsters, the IPC can change this perception. Greater transparency about how polls are conducted will not change everyone’s mind about polling. But it does provide the tools for curious consumers to more critically assess polls and sift the good from the bad. This should make the polling industry in India a more honest business.

 

* This article is based on the personal experience of Dhananjai Joshi, as well as interviews with experienced survey researchers of India conducted by Sam Solomon. Transcripts of interviews with these pollsters, which include Yashwant Deshmukh, Dhananjai Joshi, Sanjay Kumar, and Yogendra Yadav can be found at Polling One Billion. (https://pollingonebillion.wordpress.com)

Footnotes:

1. By comparison, the use of polling by parties and campaigns in the United States started during the 1950s and 1960s.

2. A dummy ballot should ideally ape an Electronic Voting Machine with the party symbols and names of candidates organized in the order that the Election Commission follows.

3. See, Dhananjai Joshi, Why We got the Delhi Poll All Wrong, 11 February 2016. http://www.dailyo.in/politics/what-went-wrong-with-delhi-polls-india-today-cicero-kejriwal-aap-kiran-bedi-bjp/story/1/1960.html

top