Last week my post about research metrics and pre-testing drew Jacob Wright, senior strategist at Mother in London, into the debate. Kudos to Jacob for putting his viewpoint forward, even if I cannot agree with some of his assertions. But the real question is, why is he so convinced that pretesting is bad for creativity?
Thinking about the debate, it seems to me that Jacob’s commentary reflects a longstanding friction between researchers and advertisers that is as much due to mindset as methodology. It is one that I am familiar with from my time living and working in the UK. But now I wonder how many people still share Jacob’s viewpoint. Could it be that this antipathy to pretesting is uniquely British and perhaps outdated?
I believe that Jacob’s assertion that pretesting lacks either a theoretical or an empirical foundation is unjustified. Millward Brown’s Link pretest is founded on both. (And to be fair, I have no doubt that most other pretesting companies will claim exactly the same thing.)
One can argue about whether the theory behind a particular pretesting methodology is sound and complete, or whether the validation is adequate, but please be assured that we have made a genuine and disciplined attempt to ensure that our research is as accurate and diagnostic as possible. (We are researchers, after all). Over the years, Millward Brown has invested millions in Research and Development to improve the predictive and diagnostic capabilities of Link and to evolve it from its original TV incarnation to encompass other media as well as 360 campaign measurement. Validation efforts continue on a number of fronts (more on those here). Marketers don’t make decisions about advertising on a whim. Neither do we introduce new metrics to Link on a whim.
I wonder how many advertising agencies have made consistent investments over the years in improving their understanding of how advertising works. I suspect the number is very few. Instead agencies rely on the creative talents of individuals to produce success.
In his comments, Jacob describes the way those creatives learn and improve this way: “The process of learning a craft is one of trial and error. If campaigns are primarily assessed against in-market effectiveness, the people who produce campaigns will learn over the course of their careers to get better at making effective campaigns.”
To my mind, that is an inherently slow and risky approach. Marketers do not have a lifetime in which to seek perfection. They need something that will produce the desired effect now. Jacob’s statement also totally ignores the negative consequences of the individual’s mistakes. Weak advertising is not only a waste of media money, but can also exact a cost in terms of lost market share and sales.
The funny thing is that Jacob’s statement about individual learning could well be applied to how the Link pretest was developed in the first place. It was developed because we proved to our own (dis)satisfaction that the “reel of 10” test we used at the time was not predictive of in-market results. So we set out to create a more accurate and predictive approach, validating the new metrics against in-market results. We learned from our mistakes, and Link was born.
That was twenty years ago, but we continue to refine our approach in the light of new learning and data. There is, however, a major difference between the learning process I just described and the individual learning to which Jacob refers. Back in the late 1980s, Millward Brown tested and tracked perhaps a hundred or so ads each year. Today it is more like hundreds every month. Compared to agency professionals who work on just a few of campaigns at a time, we have a lot more opportunity to observe and diagnose the patterns in how people respond to advertising than people. And patterns there are—patterns from which we could all learn, if only there were a way to close the belief gap between us. This learning is not prescriptive; every ad is different and must be judged against its objectives. But this learning does help when it comes to diagnosing why an ad does or does not achieve the desired response.
So, to return to my original question: How pervasive is Jacob’s point of view? How many agree with it? How many dismiss it out of hand? How many accept it as a necessary evil? Or is it possible –could it be true – that there are planners and creatives out there who have learned to appreciate the feedback that pretesting can provide? Hands up please!
(5 votes, average: 4 out of 5)
Loading ...
Email This Post
March 17th, 2010 at 7:19 am
Hi Nigel, having joined Millward Brown from primarily an ad agency background — most of it in the US — I feel reasonably knowledgeable in responding to this post. There are two points I’d make:
Yes, overwhelmingly ad agencies are against testing. There is a feeling that it stultifies the creative process, lacks sensitivity to the creative product and can be characterized as “barbarians killing babies”. Part of my role in an agency research capacity was to don the mantle of the gladiator and go out to fight the “unbelievers”. Some fine fights were had, but mostly for show as clients invariably chose to do what they saw as fit.
On the other hand, there is a great deal more discipline on the creative front than you give credit. Part of my role was to develop a strategic approach for creative development. Done in conjunction with creative leaders, we distilled the essence of what we believed was great creative and set out strategic (not creative) guidelines for subsequent development. This process ensured rational and emotional touchpoints derived from consumer insights. So the creative process wasn’t just hit or miss, but based on a reasoned approach to creative excellence. (I do admit that our criteria for excellence was not linked to in-market effectiveness.)
So I see the inherent conflict between agencies and researchers as one of a clash of cultures. The agency has a process for arriving at great creative which invariably is different from the research company’s process of evaluation. Different processes, different values and different criteria for what constitutes great creative. Not likely to see it change soon.
March 17th, 2010 at 7:56 am
Hi Phil, if I managed to give the impression that there was a lack of discipline then I apologize, that was definitely not the intent.
I fully recognize that agencies spend a lot of time and effort to identify the right strategy and a great creative idea before trying to execute that idea. However, I would say that what finally becomes finished execution sometimes bears as much resemblance to the original idea as a camel does a racehorse.
We found this in work we did with BASES a while back. Ads designed to introduce new product ideas which tested well in BASES often failed to test well in Link. Often this mismatch seems to be due to an overriding belief in the need for “emotional resonance” at the expense of the product story and a failure to account for how ads are viewed and processed in different media.
March 18th, 2010 at 11:13 am
Hi Nigel,
I’m happy that my comments have led to a serious post (and hopefully soon a discussion). I’m a little less happy about how my viewpoint was represented but that is, after all, one of the joys of comment boxes.
My first point is that I am utterly in favour of testing and learning. But that I don’t feel that, in its current incarnation, pre-testing is a useful method for testing and learning. Pre-testing is terribly seductive - who wouldn’t want to be able to predict the future? As a standardised test it’s useful for large companies to use as a metric and to gain consensus among stakeholders, and to control against disasters. All of these things are fair reasons for pre-testing to exist.
I think that a thorough critique of pre-testing would focus on many things: its inconsistency with the science of human decision making; its inability to measure the social side of marketing; the problem of stimulus; the effect on the industry and the high hidden cost to clients of agency time.
Today, instead, I’ll focus on why we argue about it so much in order to address Nigel’s question about mindset.
Pre-testing is great at avoiding disasters - truly awful ads do not make it through pre-testing. I’m sure this is true, and that it is a large part of why pre-testing is useful.
Most of the data on pre-testing shows that ads which pre-test well (whether you’re looking at persuasion or impact) tend to perform well (although falling far short of the 95% significance level it must be noted). However the problem is with ads which pre-test poorly, a substantial number of which actually perform rather well.
e.g. 45% of those with a low persuasion score generate a short-term sales effect
33% of those with a low predicted sales effect saw a sales effect.
Pre-testing screens out the disasters, but it also screens out some of the potential successes as well. In fact I’d postulate that what pre-testing is very good at is removing the statistical outliers, both good and bad. Pre-testing, because it uses attenuated stimulus, because it asks people to explain what they think and because it lacks the social element so crucial to real-life success, effectively pushes up the average performance of ads, but mitigates against the possibility of excellence.
Agencies are not interested in producing ads that are “slightly better than average”. We want to produce great work. Our clients tell us to produce great work. And when we don’t produce great work, we tend to get fired.
That’s ultimately why we tend to disagree about pre-testing and why many company’s famed for excellence in advertising (most notably Nike) refuse to pre-test. My advice to clients would be that if you want predictable results, by all means pre-test. But if you want a breakthrough then don’t.
March 19th, 2010 at 11:26 am
Thanks Jacob, again, the commentary is much appreciated. Sorry, this is going to be a brief reply since I am trying to get out of London before you can hunt me down!
Just to clarify, the numbers you quote here are unadjusted for factors other than advertising so you cannot assume that the 33% of ads which enjoyed a rise did so because of the advertising. There is a far closer fit between Link results and in-market response when we take account of those variables.
Let’s agree on what makes a breakthrough ad. What examples would you suggest?
March 19th, 2010 at 11:56 am
Nigel,
Here are a few to chew on over the weekend.
The gold standard for the modern media landscape: Cadbury Gorilla; Whopper Sacrifice; Best Job in the World; Old Spice; Orange Gold Spot; Axe Wake-up service; Ikea Malmo promotion
Greatest hits of yesteryear: Coke Hilltop; Lucozade Daley Thompson; Levi’s Launderette; PG Tips chimps;
I pick these because they are great examples of advertising adding value to the client’s business without product innovation; and of spectacularly overdelivering in terms of impact and cultural resonance.
March 19th, 2010 at 12:13 pm
Oh and Nigel, as far as the figures go, doesn’t that then imply that the 80% of ads which tested and performed well might have done so for reasons other than the advertising?
March 20th, 2010 at 6:45 am
Yes, the 80% could all have been due to other factors but it is unlikely. The only consistent element to the analysis is the time comparison (average share in the 8 weeks post advertising compared to the average from the 4 weeks preceding the ad going on air). There is little reason to assume that changes in pricing, distribution and competitive activity vary in line with predicted ad effectiveness. So even with all that noise a clear pattern stands out. And in specific cases when we do allow for GRP weight, pricing and distribution in addition to copy quality we can typically explain over 75% of the variation in share change. Seems pretty clear to me…unless you really don’t think advertising has a short-term effect.
March 20th, 2010 at 12:16 pm
The fact is you can’t have it both ways. Either those stats are valid or they’re not. If they’re valid they highlight an assymetry in the predictive ability of the test - that a positive result is more likely to be right than a negative one. Similarly, my argument is not that the test isn’t right on average, but that the 25% of variation which you can’t explain hides a fairly significant problem with the methodology.
March 22nd, 2010 at 1:39 pm
This, as well as the comments on the previous post on the subject, is an interesting debate to be having but I can’t help wondering if we’re actually making any progress on the issue of reconciling the two viewpoints. Every now and then it rears it’s head on a blog (I remember it cropping up here before with Charles Frith putting the case for the opposition for example) and both sides welcome the debate have a bit of a ding-dong and nothing really comes of it. Part of the divide is almost certainly mindset, or even beyond that, ideology - like opposite sides of a political divide, we are looking at the same data and interpreting it in a different way to suit our own ends.
Whilst it’s interesting I don’t think it’s, ultimately, that useful. If we look at this from the point of view of the people who pay our bills, they generally sit fairly resolutely in one camp or the other - yes, there are some waverers, or some whose position might change such that pre-testing becomes a more or less attractive option at a given time, but overall there are clients that believe pre-testing is valuable and those that believe that it isn’t. Largely that is to do with the same different interpretations of the figures/methodological details we’re discussing here and the fact that it is (as most sciences are) imperfect.
If we accept that this is true (which, of course, you may not) then what we really need to debate is how we make pre-testing work hardest for the clients that do use it - assuming that the intentions of both parties are the same, i.e. to produce the most effective advertising for our clients.
We understand a lot more than we used to about how to make effective advertising but there’s a lot we don’t know - I can’t help but think that a lot of what researchers don’t know about advertising is in the heads of planners and a lot of what planners don’t know is in the heads of researchers. One giant leap towards perfecting the measurement of whether an ad is likely to be successful or not would be to use that complementary knowledge to help each other rather than to try to undermine/battle each other. Help us to help you help our clients.
Simply saying that creativity and advertising are too complex a beast to be predictable is, in my opinion, an intellectual cop-out. Saying we’re not doing that properly right now is quite different but if that’s the case, then let’s talk about the nuts and bolts of what we’re doing wrong/right and use our collective resources to improve things.
- what are we not evaluating that we should be - what are the key gaps in knowledge about producing effective advertising?
- what are we currently measuring that is inappropriate, imprecise or simply not useful to a creative team looking to improve their copy?
- what are the key factors for an effective campaign from an ad agency point of view? (I suspect if we both listed them, we’d have very similar lists)
- what is the most helpful possible output from a piece of copy evaluation?
- if breakthrough ads don’t make it through (which I actually don’t believe is true) how can we ensure they do?
- what elements, if any, of pre-testing have been most useful to you in the past?
These are just examples, I’m sure there are many more areas we could discuss.
The useful debate isn’t whether we should pre-test at all, whether ad agencies like it or not, we can’t un-invent the methodology and some clients are going to continue to use it. The useful debate is how, for clients who choose to do so, we can work together to make them helpful to all concerned.
March 23rd, 2010 at 6:56 am
Thank you, AJ, for a very sensible suggestion. I agree the key question is how we work together because it seems unlikely that the demand for pre-testing will go away. When the financial investment is high people want to assess the likelihood that they will get a good return and see if there is any way to improve their odds of getting one. And the clients who most value pre-testing are usually the ones facing the most difficult marketing challenges. And I do not think marketing is destined to become an easier task, simply more complex.
Having said that I do think that it is useful to understand the emotions and reasoning behind the positions taken by agencies and researchers. Because if you cannot address the root cause of the disagreement any methodological fixes will just paper over the cracks. Which is not to say we shouldn’t aim to fix them anyway but the argument will just shift ground to fix on a new perceived weakness.
March 26th, 2010 at 2:55 pm
Hey AJ. Thanks for remembering my unhappiness with pretesting. It didn’t change anything but the consensus is changing. It’s a problem out in the open rather than swept under the carpet.
Because it’s the clients who pay for research, an agency’s opinions on the matter are listened to politely and quietly ignored. Nigel effectively does this to Jacobs most damaging accusation that predictive testing produces asymmetric results.
It was in Beijing 2008 that I listened as Millward Brown presented their pretest findings for Halls cough drops that realised how silly it is to try and improve the advertising. There is no point.
That’s because it’s not creativity that is being developed.
It’s risk that is being managed.
Clients are under massive pressure to play it safe. It goes without saying that everybody wants better results but in this case better results means not making a mistake. Easily the best thing about pretesting is that it is the most comprehensive risk removal process in marketing.
Unfortunately it’s never described as that. So instead we sit around meeting tables using the word creativity when it’s not creative at all.
No more than 5% of advertising is creative but it’s the 95% that pays the bills.
If we all stopped pretending. We’d all get along a lot better.
March 27th, 2010 at 11:30 am
Hi Charles, long time no comment. Welcome back.
I cannot disagree with your point about risk management. Of course that is one role of pre-testing. It would be daft to deny it. The financial risk involved in developing a new campaign or ad is massive. Given the number of people involved in developing an ad even when there is a great idea to start with the chances of an execution staying true to that idea are low. So, yes, risk reduction is a key role.
However, I cannot agree that risk reduction is the only role for pre-testing. It is not about developing creativity it is about making sure that the audience recognizes that creativity. As to the asymmetric results that is just unsubstantiated BS. As you well know Cadbury’s Gorilla was pre-tested by Millward Brown and Hall & Partners. And as you may remember I am not allowed to discuss the results but one of my colleagues did pass on the following from Campaign:
“Deborah Mills, the European chief executive of the research company Hall and Partners, argues that the “gorilla” spot was improved by research (it was pre-tested by both Hall & Partners and Millward Brown).
Rachel Barrie, the director of strategy at Fallon, backs this up:” ‘Gorilla’ was researched again and again. In this case research didn’t only help build the confidence to do something incredibly courageous for the brand, it told us some things that we didn’t expect - like older audiences loved it as much as younger ones.”
So pre-testing may actually help agencies reassure clients that an ad is not too far out for the target audience.
At the risk of boring everyone silly with this debate I may write a post specific to Jacob’s claim that pre-testing cannot idenitfy breakthrough ads. It’s like claiming that you can’t predict what gives ads the potential to garner viral viewing (see my recent post).
July 6th, 2010 at 7:56 am
I come to this long-running debate rather late, but as a one-time Creative Director who is now involved (among other things) in running courses for clients, the subject of pre-testing often comes up.
Clearly, there is a different mind-set between agencies and clients: understanding this might help the relationship. It is also clear that pre-testing isn’t going away, so agencies need to understand it and work out how to use it for the best -or at least to minimise any real or perceived negatives. Of course pre-testing is about ‘managing risk’- the promise of prediction is the client drug and MB and others are the pushers.But ‘risk’ is hardly ever really defined, and the real risk is to produce risk-free work that does not stand out and does nothing for the brand in the longer term. As Gordon Brown said years ago, new news will get a short-term blip and doesn’t need much creativity. The benefits that creatives bring ‘often do little for persuausion but always do a lot for memorability’.
My first experince of the Link test was as ECD of Leo Burnett. We had an ad for a yellow fat which I felt could only be improved by better animation and a storyline. The Link test result was pretty poor. I asked what it was being compared against - all ads, I was told. When I asked if it could be compared to other yellow fats advertising the result was exceptional. So while I understand the value of averages and norms for the comparison of ads over time, how does this help a specific ad? Everyone involved has to ask questions.
I think you make the point -or your website does- that everyone needs to be involved throughout. If the agency and client are not sure about the objectives, or target audinece, or the main thing they need to communicate then pre-testing is going to be a waste of time. The researchers on all sides need to understand what the creative intention is. The ‘action standards’ need to be agreed if -despite all the ‘test tubes’- Link isn’t going to remain a yes/no device for the ’risk-averse’.
The debate you were having with Jacob about the statistics is unfinished and probably beyond me, but one conclusion seems to be that ads that test well tend to do well, but also that some ads that don’t test well can also do well in the market. From a creative point of view, many of the ads that test well are not particlularly ‘creative’ (subjective as that may be). The IPA recently announced the results of a survey correlating IPA Effectiveness winners with Donald Gunn’s records of awards. MB recognises that emotional advertising tends to do better than purely rational advertising (though a combination does better). This would indicate that engagement and enjoyment should always be criteria for success, but of course they overlap with relevance. The criteria will differ for each ad but need to be agreed by agency and client up front. For me this makes the point that, along with ‘norms’, the Link test is not as objective a test as MB or some clients would like to think.
This leads me finally to the point MB makes about ads like Cadbury’s Gorilla doing well in the Link test. A great ad doing well is not the same as saying that pre-testing can identify a ‘great’ ad (probably defined as effective and creative/original). This seems to me one of the things Jacob was saying. If you are going to quote Gorilla, it is a shame you cannot provide details. I know a bit about what happened with this ad, and the relevant questions I would ask are: in what form was it tested; when and what for? How was it ‘improved’? What was re-shot? The fact that it showed that older audiences loved it suggests to me that Link was used after the event to persuade senior people within Cadbury’s to run it. I’m sure it was not an easy sell. This use of pre-testing for internal reasons is a common one and I think a mis-use.
What is tested, when and why are critical questions for the use of Link. If it is to manage risk then work will surely be tested in rough form, so the choice of stimulus is crucial (and a cost). When and why it is tested depends on whether it is part of the development of a strategy and an idea that has been agreed, or whether it is simply being used to find a strategy or to make a judgement rather than really being a tool to help make a judgement.
A lot of whys and wherefores to be considered if pre-testing is to be accepted by all- and I haven’t even mentioned all the non-TV stuff that is part of many a campaign today.
July 9th, 2010 at 4:27 am
Hi Giles,
Thank you for a great contribution to this debate. I am sure the debate will last as long as advertising does!
Overall I would agree with much of what you say. Norms should be relevant and used sensibly for guidance not be used to make default decisions. Everyone needs to be agreed on what the ad needs to acheieve and how the creative is intended to engage the audience. And I love the fact that you quote Gordon Brown’s comment back at me!
I think the real issue is that you suggest that some ads may test poorly but perform well in-market. My suspicion is that this is more an urban legend than a fact. I am not saying it never happens. Simply that the cases when it is proven to happen are rare. In my experience cases like these are often the result in the brand’s specific context and envrioment than the advertising per se.
As to Gorilla, yes, mea culpa. Believe me I find this just as frustrating as you. For what it is worth I can tell you that Guinness Noitulove was tested as an animatic and performed well. I agree that the quality of stimulus is crucial and worth the investment to make sure an idea is represented well. A rough ad does not have to be a set of drawings. It should be whatever will best capture and communicate the idea of the ad.
I will no doubt return to this subject before too long and will try to do your comments more justice at that time.
July 13th, 2010 at 8:31 am
Hi Nigel - found your blog while researching input for a class I will be teaching on pre-testing and wanted to assure that I have not missed out on the latest news in this field (since I first prepared my paper about 5 yrs ago). A lot has been said about “why ad people do not like tests”. From my experience on the ad agency side I recall MB as a truly good example - I met some excellent researchers that where able to interpret and help push the creative work to an improved stage, not damaging the idea. But as in most businesses: it is people working with methods. And if they are not reflective or experienced enough humans sometimes tend to be - well, humans. And I think that is curical: to have a researcher that is truly weathered on behalf of advertising and how creatives and the creative process works. And that sometimes you just can’t tweak stuff to improve it. Well - just wanted to share this point - hope I did not miss it within the comments above. Great blog by the way!
July 13th, 2010 at 10:02 am
Hi Nina, thanks for the comment. I think your point is in keeping with the points made here and elsewhere. Great people make all the difference.