Lies. Damned Lies. And Effectiveness Awards.
Recent global studies by System1 and Kantar estimate that of all the advertising produced in a given year, only 6%-13% is actually effective. Which means that, if you’re sitting in a post campaign analysis, reading a case study, or judging an effectiveness awards entry there’s statistically a very good chance that the work you’re being led to believe was stellar, was nothing of the sort. There’s a good chance the evidence being presented to you is based on faulty logic, misinterpreted or misrepresented data, or in the most egregious of cases, blatant lies.
The problem is, most of us are not statisticians so these errors could go completely undetected. But, once you know what to look out for then, with just a little bit of critical thinking about the methodology and presentation of any graphs and data you encounter, anyone can spot this dubious data whether it’s an innocent mistake or intentional deceit.
The first thing to look for is credibility - do these results seem plausible? To decide that, it’s worth remembering how advertising actually works. According to Les Binet, Head of Effectiveness at Adam&Eve DDB, “it increases or maintains sales and margins by slightly increasing the chance that people will choose your brand.” In fact, advertising is often referred to as a ‘weak force.’ That’s not to say it doesn’t work, it just means that in all but the most exceptional of cases, work that ‘shot the lights out’ most likely didn’t. As soon as you’re presented with remarkably large numbers, especially related to growth in market share or return on investment, your spidey-sense should start tingling.
Even if it fails that first credibility test, that’s no reason to reject it out of hand as ridiculous. It’s just an indication to keep your wits about you and proceed with caution and a healthy dose of skepticism.
While there are numerous flawed arguments and mistakes that may lead you to believe something that just isn’t true, they tend to fall into one of three broad categories: errors in the way the data or evidence were collected, errors in how the data were interpreted, or errors in the way the data are presented.
Probably the most famous example of an error in the way data were collected occurred in the US presidential election of 1948. Pollsters predicted a landslide victory for Thomas Dewey against the incumbent, Harry Truman. The prediction was so widely accepted that the Chicago Tribune prematurely ran with the headline ‘Dewey Defeats Truman.’ There was one small problem. He didn’t. Truman won.
How did the pollsters get it so completely and utterly wrong? They relied on telephone surveys to collect their data. Nothing wrong with that, except in those days, owning a telephone was a sign of wealth and higher socioeconomic status, which skewed the sample towards more affluent and generally more conservative voters who are, traditionally, more likely to support a Republican candidate, in this case Dewey. The vast number of more ‘blue-collar’ workers who were not surveyed because they didn’t have a telephone, favoured the Democrat, Truman.
Another error in collection occurs when not enough data is collected, or important data is omitted. Cherry picking. A few years ago a case study was published by a brand linking remarkable increases in Black Friday sales compared to previous years and attributed to their advertising campaign. Had the authors of the case study bothered to examine data from previous years leading up to Black Friday instead of just the period immediately before and after this particular year’s campaign, they would have realized that unlike in those previous years, Black Friday this time fell on a day after pay day. So people had more money in their pockets than usual come the big day. It doesn’t take a genius to figure out the real reason for their remarkable sales that year.
The most common omission you’ll find in effectiveness entries is a lack of context. It’s one thing telling me your sales increased by 10%. But when those sales occurred against a backdrop of a category growing by 15%, your dead-cert Gold award just went to barely scraping a Finalist. Which may explain why so many authors do it. Without context, it’s impossible for the reader to determine whether your results truly are remarkable, or just so-so. And it’s such an amateurish mistake to make that when it happens, it creates the suspicion that you’ve done it deliberately because there’s something you don’t want your audience to know.
A further error of collecting occurs when the wrong data is gathered for a particular objective; a mismatch between the goal and the metric used. The most common form this misalignment takes is when so-called ‘vanity metrics’ are measured instead of hard sales or revenue metrics. A massive increase in social media likes and followers is not in any way evidence of the effectiveness of your campaign if your actual objective was to increase your sales revenue. A far better argument for effectiveness would have been made had you measured increase in sales, the number of new customers acquired, the average transaction value, or the conversion rate from leads to sales.
Errors in interpreting data (assuming they’re not wilful) are easy mistakes to make. The most common example is mistaking correlation for causation. The mere fact that as variable X increased so too did variable Y, is not evidence that one caused the other. Many correlations you’ll be confronted with are actually spurious. That is, the relationship between the two variables appears to be statistically significant, but is actually caused by some other, often hidden, variable or even just due to chance.
There is a direct correlation between ice cream sales and incidents of drowning, but can eating ice cream make you more likely to drown? No. The correlation occurs because there is a third, confounding, variable; the weather. When it’s hot, people buy more ice cream and go swimming more often. The apparent relationship between the two variables is misleading because it doesn’t reflect a direct, causal relationship. You can find some of the most outrageous and entertaining cases of spurious correlations like the one below, at tylervigen.com/spurious-correlations.
A fundamental flaw in so many effectiveness cases is the failure on the part of the authors to eliminate all other potential causes of their important effects such as sales. Until you can rule out everything else (a good starting point is to ask whether a change in any of the other P’s of the marketing mix could have contributed to your results) and isolate your campaign, you have not proven that your campaign was responsible.
Related to spurious correlations is the ‘post hoc, ergo propter hoc’ fallacy which is a fancy way of saying ‘if this happened before that, this must have caused that.’ It seems a ridiculous error to make, but year after year it crops up in supposedly watertight case studies.
And then there’s the good old ROI calculation. Case study after case study concludes, in triumphant fashion, with a calculation of how much return the client saw in return for each rand spent, as if ROI were the be all and end all of effectiveness metrics. Invariably, it’s a double-digit ratio. Remember what we said about being wary of big numbers?
ROI mainly measures the financial return relative to the cost of an investment. It focuses on how well resources are used to generate financial returns, making it a metric for efficiency - how well resources / assets were used to achieve a financial return - not effectiveness, which is (or should be) concerned with how well marketing activities achieved their strategic goal.
Its appropriateness as an effectiveness metric is further undermined by the fact that it’s not even calculated properly. Prof. Mark Ritson lists numerous ways in which it is incorrectly calculated including: incorrect cost allocation where certain costs associated with the campaign are omitted from the calculation leading to an inflated result; attribution errors where revenue is attributed entirely to just one aspect of the campaign e.g. the TV ad; overly simplistic attributions of revenue where all sales are attributed to the campaign despite the fact that a large proportion of sales would have occurred regardless of the campaign; and basic miscalculation where ROI is calculated based on gross revenue rather than net profit, ignoring the cost of goods sold and other expenses.
And finally, we come to errors in the way data is presented. A.K.A. Graph crimes.
The first of these is choosing an average that while making your case look amazing, is not a clear reflection of the real story. When most people use the word ‘average’ they mean a particular kind of average, the mean which is calculated by adding up all the data points and then dividing by the number of data points. In many cases it is a valid and appropriate choice of statistic to give an idea of an overall picture. But because of that method of calculation, the mean ceases to be a useful statistic as soon as your data contains any outliers. These outliers drag the calculation in their direction and paint a distorted view of reality.
For instance, if you were to use the Mean as a measure of the average net worth of South Africans, extreme outliers in the form of certain families and individuals who, unlike the rest of us, measure their worth in billions, will inflate the mean and provide a value that is obviously not a true reflection of SA society. A far more realistic view would be provided by calculating the Median instead. This calculation accounts for outliers and provides a number that exactly 50% of people are above and 50% below and a far more accurate description of the true average net worth.
A brilliant demonstration of the importance of the choice of summary statistic is the fact that by opting to calculate the Mean, you can show that the average person has one testicle.
Pay attention as well, to the Y-axis of any graph presented to you. Manipulation of this axis results in the distortion of trends and differences in data, exaggerating or minimizing the true differences between data points. For instance, a compressed scale can make minor differences look significant, while an expanded scale can make significant differences look minor. With the right scale on your Y-axis say units of 1, a mere 5% increase in sales over time would appear as a very steep line graph. Far more dramatic and impressive than the exact same data plotted on a graph where the scale of the y-axis is in units of 10. Such a graph would depict a far more gradual increase or decrease. Similarly, manipulating the y-axis so it doesn’t begin at zero can be equally deceiving.
Consider a bar graph comparing sales figures for two years. With a Y-Axis starting at zero, and year 1 having sales of 100 units, while year 2 has sales of 105 units. The bars for both years would be relatively similar in height, accurately reflecting the small increase. But with a Y-Axis starting at 90 the exact same data would show a much taller bar for year 2 compared to year 1, exaggerating the difference in sales and giving the impression of a significant increase.
While this list is in no way a fully comprehensive list of all the data sins you may be exposed to - there are entire books dedicated to that - at the very least, it should make you feel better equipped to properly judge ‘proofs’ being presented to you.
Failing that, you now have a handy list of ways you can game your next Effie entry.
