Outside it was hot and steamy. Inside, the audience at the AAAA’s Account Planning Group conference in Miami was nearly frozen in their seats by the air conditioning. The air temperature did not stop one delegate asking a heated question of the three pre-testing company representatives on stage. Likening their position to that of Smith & Wesson - suggested to be “we just make guns, we can’t control how people use them” - he asked what we did to teach our clients to use pre-testing wisely.
The question followed presentations on the topic “Can Pre-Testing Inspire Creativity?”. The presenters were Roger Baxter, Managing Director, Hall & Partners New York, Amy Shea, Research Director, Ameritest, and myself. Other pre-testing companies were notable by their absence, although I was told that at least one other major company had declined to take part.
Interestingly, the three presenters were unanimous in their opinion that the intent of pre-testing is not to inspire creativity. My summation of the three presentations would be as follows:
- Research conducted earlier in the ad development process rather than pre-testing should inspire creativity.
- Pre-testing is designed to assess whether the original idea has been executed successfully.
- Pre-testing should identify whether the ad inspires the intended audience.
- Pre-testing should assess whether the ad has either successfully supported the existing positioning, or positively stretched the audiences’ brand perceptions (rather than stretching them too far).
- Clear communication of the ad objectives before the pre-test takes place is critical. Unless the research team know what the ad is intended to do how can they assess whether it has done it?
(Those of you who are interested can download my presentation in pdf form by clicking here: APG presentation.)
So are pre-testing companies like gun manufacturers, allowing their clients to use their “product” without due regard to their own safety or that of others? Here is my take on the subject.
However inspirational the idea on which an ad is based, so many people have an influence on the subsequent development process that the finished film can often be as different from the original idea as a camel is from a race horse. All too often the finished film fails to inspire the intended audience, because the idea has morphed over time and is no longer relevant and involving. Alternatively, the idea might survive but the execution does not harness it to benefit the brand.
As a result, ads differ widely in terms of their impact in market. Clients invest millions in ad development and media spend and want to ensure that the ads which go on air will build sales and the brand. Unfortunately this desire often leads to the development of one-size-fits-all action standards which actually serve to limit creativity rather than encourage imagination.
Why do such action standards limit creativity?
First, a pre-test identifies how an ad performs today, not how it might perform in future with further development. A good pre-test will provide insight into how an ad might perform and guidance on what changes might achieve that outcome. Research results should not be taken at face value but interpreted in the light of the ad’s objectives to assess likely potential; otherwise potentially great ads may be discarded as ineffective, a tremendous opportunity cost that goes unrecognized.
Second, agencies often try and “game” the system, imagining that they know the secret recipe for success. The end result is creativity which is confined by limits which do not necessarily exist.
Third, and worst of all, some agencies simply throw in the towel and fatalistically accept the outcome of the pre-test. Again, potentially great ads may be lost simply because no one is enaged enough to fight for an idea in which they believe.
Do pre-test companies do enough to address these issues? I doubt we do.
In the first case, it is tough to argue that there should not be action standards, but do we do enough to persuade our clients that these need to be flexible and adapted to the specific category, brand and task? Advertising can work in many different ways to drive sales and build brands and our action standards need to reflect this fact. Do we ensure our presentations focus enough on the ad’s potential, rather than its current performance? Are we brave enough to engage in debates about ways to change the execution for the better if there is a real problem? The extent to which people feel they have the freedom to engage in these issues depends on the people involved, the culture of the client company, and the nature of the relationship with the ad agency.
In the second case, I can only say that agencies do themselves a disservice if they think they can game the system. If a pre-test is designed to measure the many ways in which advertising can shape peoples’ relationship with a brand, there is no recipe for success other than the following:
- An idea which is relevant to the brand and resonates with the target audience.
- An involving execution that will ensure people give the ad their time.
- Strong linkage between the idea and the brand.
In the third case, something is desperately wrong. Numbers are just numbers, no one should accept them at face value or allow them to determine the fate of an expensive investment without serious thought and debate.
So, is it all the fault of the research companies that pre-testing is often the bane of the ad agency’s life? No. Unfortunately, if it was, we might be able to solve it more easily. Rather the blame must be apportioned equally between research agency, ad agency and client. Only if all three parties take the time to engage with the process can pre-testing become more than an exercise in risk reduction.
(31 votes, average: 3.94 out of 5)
Loading ...
July 26th, 2006 at 5:01 pm
Nigel
Great summary. Your prespective on the role of copy testing and the entire creative development process will hopefully stir the pot some and cause the industry to begin to think about things a little differently.
July 27th, 2006 at 8:04 am
Nigel
An excellent and sensible summary. Although I do think that you have neglected to mention one issue, a pretesting system is only as good as the research team behind it. It is worth noting at this point that I think this is an issue for all the major research providers and is not aimed at any one company.
My experience suggests that most copy testing products will successfully identify the great ads and those that simply don’t resonate, but many ‘breakthrough’ ads (especially those in early stages of development) are likely to be polarizing resulting in ‘mediocre’ scores. It takes an experienced and inquisitive mind to separate the wheat from the chaff.
Unfortunately, increasing pressure to deliver ever higher profit margins seems to be encouraging the reduction of pretesting to an easily replicated formula that requires little thought (I believe that the ‘gaming’ you refer to is the inevitable outcome of this mindset).
In reality this means that in order to push more product in less time pretesting is often under thought and managed by the most junior staff.
For the long term good of the industry (not to mention those inconvenient clients) copy testing providers need to place more emphasis on quality, even if that means accepting slightly lower margins. It also means retaining those staff who have the experience, and rewarding those who really deliver (as well as those who provide high margins by over promising and under delivering).
If Smith and Wesson magnum 500’s were mass produced with the same laissez-faire attitude to quality control as most pre-testing decks then they would quickly be out of business.
Yours
Diamond
July 27th, 2006 at 8:40 am
Thanks for the comments Terry and Diamond.
Diamond, I would agree that the biggest challenge faced by any pre-test system is to identify which of the ads that achieve “mediocre” scores have the potential to be “breakthrough” with further work. And, yes, this requires not just good diagnostic questions but good people willing to explore beyond the topline numbers.
I doubt, however, that anyone believes they have a “laissez-faire” attitude when it comes to preparing presentation decks. Rather, I do know that the time pressure placed on pre-test companies to deliver results are often extreme, forcing people to default to standard templates and quickly thought out conclusions. One of the themes that came up at the APG conference was the need for more time to be creative, and this applies as much to interpreting research data as creating an ad.
I would agree that current business pressures have led to more focus on using pre-testing as a go/no go decision making tool, rather than an optimization tool, and regret it. Time pressure and the use of action standards encourages this mindset.
In most cases, however, action standards do lead to improved short-term sales performance, because the ads that air are more impactful and persuasive than might have run otherwise. What is far less measurable is what the opportunity cost of discarding ads and ideas that just need more work to make them really effective, not to mention the frustration on behalf of the agency team.
I believe that this can be avoided if all people involved in the ad development and testing process agree up front on what the ad is intended to achieve, and action standards are adjusted accordingly. That way, maybe we will see more ads that help clients “win big” instead of win by increments.
July 28th, 2006 at 4:25 pm
Hi Nigel,
I was just reading your comments on pre-testing and finding that you have spotted all the issues we all (research agencies, ad agencies, advertisers) have. I have mentioned three parties because in fact all of them are and should be involved in the process of building great advertising. Far too often we, as a researchers are placed in a position of “pretesters” where we are coming to the conference room with the test readings forced to confront them with a reading of “healthy” ad (action standard). Our judgement (we are judges in this situation) should be as objective as possible and figure based only. In far too little situations we are invited to co-operate in the early stage when the ad (and the brand) is build. I believe that only than we are able to understand the context, brief helps a lot but in many situations it is not sufficient to fully understand the context we are in. The importance of understanding the contex is especially visible where ad is not meeting its objectives, than it is often up to our “qualitative” judgement to recommend or not to recommend it to be on air.
All of us want to creat (or watch) A class advertising. By definition in most of the cases it is not possible. Therefore many times we are finding ourselves in suboptimal situations, where a given ad might not be great but in a short term is ok. On the contrary many times we are in a situations where ok ad is not enough. To make a right choose in this “grey area” examples we could not be only “pretesters” but people who understand and are involved in brand building but not be slaves of action standards.
Regards,
Pawel
August 1st, 2006 at 11:22 am
Thanks for the comment Pawel. To paraphrase, you feel that too often we are being used as the decision maker on whether or not an ad airs, even though we may not have been involved in the development prior to the test.
Clearly this is both difficult for us and potentially undermines the development of the best possible advertising for the brand. I think that many people assume objectivity and involvement are at odds with each other and so want to keep us out of the loop. Particularly if the objective is to diagnose and develop the best ad possible I think this believe is fundamentally wrong.
August 2nd, 2006 at 11:12 am
If the kind of involvement Nigel and Pawel are advocating is an entirely passive, observation based involvement (and one that doesn’t add any expensive hours to the final bill) then great. If you want to get your hands dirty, voice unsubstantiated opinions (the privilege of the client) based on a ‘philosophy’ of how advertising works and charge me for your time then ‘no thanks’.
No one would dispute that you only get out of copy testing what you put in, but a thorough briefing from an informed client will usually suffice (and I don’t mean a briefing form or a creative brief).
Copy testing and researchers do not inspire creativity or creative ideas, so they have no place being active participants in the early creative process. There are already too many voices in the mix, one more won’t help.
August 2nd, 2006 at 2:00 pm
Hi Nigel,
Could not agree more! Unfortunately business objectives ie share of research budget between “specialised” agencies, is working against us and against great advertising. Unfortunately that is the direction where the marketing business is heading to
August 2nd, 2006 at 8:41 pm
As expected from Nigel, this is a thoughtful contribution to the pretesting debate. It is also provocative, in that I have never heard anyone else from MB questioning the omniscience of their conclusions. The concept of a tripartite debate also makes perfect sense (though I’m sure people have been saying this for years.)
I have two comments.
Firstly the statement “Numbers are just numbers, no one should accept them at face value or allow them to determine the fate of an expensive investment without serious thought and debate” appears naive to anyone who has sat through multiple pretesting debriefs where research company and client smugly if silently appear to conspire to declare an ad dead on the basis of just one number. The truth is that clients have benchmarks and research companies have made a good living from perpetuating the one-number myth.
Secondly, and in this context, to suggest that the solution lies as much in the hands of agencies as with research companies and clients seems somewhat inappropriate. Sure, agencies can act as catalysts for change and should be more proactive in stimulating development of more nuanced methodologies. But ultimately this is a game played by research companies and clients and it’s there that the solution has to lie.
Rather than Smith & Wesson, a more appropriate analogy for pretesting companies may be Philip Morris. Clients are hooked but in the end it’s “habit” (i.e. poor creativity) rather than “cigarettes” (i.e. inadequate prestesting methodologies) that get the blame.
August 3rd, 2006 at 4:28 am
Hi there,
First of all: Nigel – great article, really hitting the point as far as my opinion is concerned!
Further I like to comment on what Diamond said about “most junior stuff”. I often experienced that especially at the clients site fluctuation of people working for a brand is high, which means that one is often confronted with brand managers or researchers that do not know much about a brand’s history, about past ads, pretests etc. As a result knowledge about a brand’s history get lost and with this often consistency in advertising.
As far as Nigels’ comment about flexible action standards is concerned, reality is unfortunately other way round. More and More clients request globally harmonised action standards, resulting in two numbers divided into new and established brands that serve as goals for each and every ad. And even the most experienced stuff will not hinder the client to circulate internally results for only these two numbers, even if they again and again claim that all other measures need to be considered to access the ads full potential.
All the best, Susanne
August 4th, 2006 at 9:25 am
Thanks for the comments Susanne and Jeremy, both of you have added new perspectives to the debate.
I guess my reaction to Jeremy’s comments is, “There speaks the well-indoctrinated planner.” Let me ask you a couple of questions.
When you worked for an agency (and assuming you had a set of objectives by which you were judged for promotion, pay increase and bonus) did you just blindly accept what you were given, or did you spend time figuring out what the assessment criteria were and whether or not you thought they were fair? If they were unfair, did you keep quiet or did you discuss the issue with your manager with the aim of agreeing a more appropriate set of goals?
Like it or not, goals and objectives exist in any aspect of business. If they are inappropriate you better engage with the process and get them changed or suffer the consequences. So, from my point of view, suggesting that agencies have no part to play in “the game” is ultimately a self-destructive play.
If anyone out there - planner or client - would like to learn more about the Link pre-test, validation of its “one number” (which is the output of four different measures of response to the ad) against in-market sales response, or the impact on business performance of setting pre-testing action standards, please just send me an e-mail.
Susanne, you touch on an issue that I suspect most of
us have come across but do not want to raise. Speaking as one who has been in the industry more years than I care to remember, I believe that there has been a general delegation of advertising-related tasks to more junior staff - on all sides.
The planner who asked the initial question at the APG Conference closed with the comment that he wished that more people at the pre-testing companies showed the passion for their work that the presenters on stage did (a comment that was applauded). I think it is unfair to level that complaint at the research agencies alone. As senior management faith in advertising has declined, so too has investment. The result, delegation to more junior staff, pressure on agencies to keep costs down, etc., etc. It is a viscious cycle that needs to be broken by a) better proof of ROI, b) a change in strategic focus from cost-cutting to revenue-expansion (and a recognition that this will require increased investment in talent and time).
August 7th, 2006 at 1:32 pm
If in doubt, refer to the APG’s top ten tips for conducting quantitative research, tip number 4:
‘4. RESEARCH THE RESEARCH
By this I mean understand, precisely and in detail, how the type of quantitative research you are using works. How the research is conducted, how the data is collected, and what is then ‘done’ with or to the data.
An advertising agency I once worked for had a strong antipathy to Millward Brown, and specifically to their quantitative pre-testing methodology : Link. The agency didn’t want to know, and vainly tried to wish Millward Brown away. They didn’t bother doing their research on Millward Brown. Millward Brown debriefs became an extremely uncomfortable head to head struggle ; a battle of wills in which the agency huffed and puffed and sulked and The Millward Brown researchers became more and more gnomic and monosyllabic and unhelpful. The agency never won. Unfortunately many of you will recognise this scene in your own agency. To get to a better and more productive place I recommend one simple thing which is almost never done: go and see the people who are going to do the research. Get them to present to you the theory on which the methodology is built. Ask them the pointed questions that are normally saved up for the debrief and used- when its too late – to bushwhack them. Find out from the people who run the company what they think. At least then you can have an informed debate from the position of knowledge, rather than ignorance’.’
August 8th, 2006 at 6:31 am
Once again Diamond, great comment. Thank you!
As you can probably tell from some of the comments above, I am sure that people at Millward Brown would welcome the chance to share information on our approach to pre-testing. It should lead to a far more productive relationship between ad and research agency.
November 8th, 2006 at 7:17 am
What about the situation of ad agency demanding a go/no go kind of analysis? Far too often we have an ad agency following ‘take it or leave it’ policy, protecting their creative with their teeth. What there’re saying is ‘even if ad’s got some downsides, its creative was produced according to the brief and won’t allow any changes, you either accept it or start making another one’. As copy-pretest isn’t actually about final decisions, it’s about evaluation and advise, we have hapless client listening to research agency listing ad’s flaws and ad agency refusing to change anything. And a company being part of international corporation may have hard time changing local ad agency to the one of their liking.
The talk of researchers and planner people working together looks fine on paper, but the reality oftentimes works the other way.