Rovell Super Bowl Ad Poll Highlights Challenges With Polls, Surveys
BY ADAM GROSSMAN
The Action Network’s Darren Rovell conducted a Twitter poll about Super Bowl ads after the game last Sunday. The text for the poll stated, “Companies spent approximately $460 million on advertising their products and services during Super Bowl ads tonight. Do you think you are now more likely to buy at least one of the products or services you saw?” Of Rovell’s two million+ followers, 16,186 votes were cast with 81% saying “No” and 19% saying “Yes”.
This would appear to be terrible news for the companies that spent the approximately $460 million in advertising for the Super Bowl. A deeper analysis, however, shows that the news is likely not as bad as it appears because of challenges with this poll’s validity specifically and survey work generally.
There are often three main questions to ask when completing data analysis particularly with polls or surveys. They are:
Are the data accurate?
Are the data reliable?
Are the data valid?
The data are considered accurate when they correctly reflect what is being measured (i.e. how percentage of people selected the “Yes” or “No” option). There should be no accuracy issues for the Rovell poll as it Twitter should easily captured the 16+ thousand votes given the simplicity of providing the answers through the platform (one click on either the “Yes” or “No” option).
The data are considered reliable when the results of what is being measured will be the same or similar over multiple trials. While Rovell cannot rerun the poll again under same conditions (i.e. re-run this pool right after Super Bowl LIV), one can safely assume that the vast majority of his audience would still vote “No” if he had using probabilistic analysis. Rovell did not provide a standard deviation or confidence interval for his poll. However, the sample size and percentages for each option means the differences between “Yes” and “No” would likely be similar if the poll was run on multiple times even assuming that the original poll has a very large standard deviation.
The data are considered valid when they accurately reflect what is being tested or examined. There are several reasons why this poll would not reflect if Super Bowl ads drove changes in actual consumer behavior. This includes:
Selection bias
Response bias
Confirmation bias
Selection bias occurs when a sample is not randomly chosen or reflect the population being examined. Block Six Analytics’ Audience Inference Platform (AIP) can determine the demographic and psychographic profile of a Twitter account’s followers with a statistically rigorous level of certainty. Our AIP analysis below clearly demonstrates that the Rovell’s audience does not reflect a more general population that is likely watching the Super Bowl by percentage of audience across several factors.
Response bias occurs when participants tell the person or people running the survey what they think they want to hear. Rovell has become well-known first at ESPN and now at the Action Network in large part for taking controversial and / or provocative stances on sports business issues. In addition, the phrasing of the prompt with the amount spent in the first sentence ($460 million) and the question starting with the “do you think” in the second sentence is suggestive of a “No” response. “No” means that advertisers are spending millions for little or no return on investment (ROI) based off the poll’s phrasing. Rovell’s followers could then easily determine that the controversial / provocative answer (likely the desired response) to this question is “No” based on Rovell’s background and the poll’s phrasing.
Selection and response bias are also likely one of the root causes of confirmation bias with this poll. Confirmation bias occurs when respondents “confirm” previous beliefs even if they differ from actions they will take. Rovell’s audience is much more likely to be interested in sports and business than the general population. Therefore, it is likely the voters of Rovell’s poll are saying “No” at such a high rate likely believe they would never be persuaded by advertising in sports at all (let alone a Super Bowl ad) given their perceived expertise on the topics.
We used AIP to test this hypothesis by examining keywords and topics emphasized by Rovell’s followers. We found that 76% of the top-25 keywords used by this audience related to sports meaning Rovell’s audience disproportionately talks about this subject. In addition, past research shows that Super Bowl ads can drive incremental revenue in ways contrary to the what the poll suggests. We have highlighted a study in past blog posts that demonstrates that both Budweiser and Pepsi generated new revenue attributable to Super Bowl ads with Budweiser creating $45 million in direct revenue from one game alone.
Addressing these biases and correlating results with revenue are two reasons that B6A’s developed our Social Sentiment Analysis Platform (SAP). By tracking what people say in owned and earned social conversation using natural language processing, SAP can track what people are saying across millions of posts without relying on “yes” or “no” responses (we use a percentage or dollar amounts). These posts are also in the “wild” with people stating their opinions in an organic context (i.e. no one is asking or shaping a specific question such as the one Rovell asked in his poll). In addition, B6A’s approach to calculating brand sentiment, engagement, and awareness has been shown to have a strong and statistically significant relationship with revenue growth.
There is definitely value in conducting surveys or polls. In particular, they can be accurate and reliable reflections of what people believe in the context of the questions be asked as demonstrated by Rovell. It also important to work with an experienced survey provider that understands and minimizes the biases describe in these posts as we do with Turnkey Intelligence. The challenge is what people think or say they think does not always reflect the actions they will take. Understanding the biases should make results Rovell less of a concern for brands that advertised during the Super Bowl.