Charles Frith pointed me to this recent post by Jason Oke on the blog for Leo Burnett Toronto. Jason addresses two questions posed to his agency by a client who needs to develop a global POV on pre-testing. Since Jason explicitly invited further discussion, I thought I’d chime in.
Before we go further, let me state for the record (and I do hope you’re all sitting down) that I believe in pre-testing. The ad development process is fraught with difficulties, and all too often, an idea that starts out as a race horse is subjected to so many opinions and decisions that it is turned into a camel. Pre-testing can be used to check that the original idea still comes through and still resonates with the target audience—conditions which must be met if the ad is to have the desired effect on the brand.
I should also remind you that my point of reference is Link, which differs from some more traditional pre-test systems in that it does not literally try to test people’s memories or reactions. Rather, Link aims to predict in-market response by asking simple introspective questions about how the ad makes people think and feel.
The first question that Jason addresses is:
Given that copy testing in some form is here to stay, what do you believe is the best process for testing creative. If you could run it your way what would you do?
After reviewing a number of problems and challenges faced by pre-testing today, Jason frames his answer in terms of the need for copy testing to be flexible.
“If we had to pre-test,” he says, “we’d figure out what method suited each ad best.” After a discussion of different ad styles and content, he restates that idea. “The method should be dictated by the objectives of the advertising.”
I couldn’t agree more. Why subject an ad to tests which are not relevant to what the ad is trying to do? So if you don’t want people to watch your ad, don’t evaluate it on its ability to involve the audience. If you don’t have a compelling message to convey, don’t assess your ad in terms of relevance, persuasion, or credibility. If you are not trying to evoke a positive emotional response, there’s no need to figure out how an ad makes people feel. And if it’s not important to you that people know which brand is being advertised, don’t worry about how well your ad focuses attention on the brand. I think you’d be crazy not to want to see how your ad is doing in all these areas, but then I am not a “throw it against the wall and see if it sticks” kind of guy.
Jason is right to be critical of tests which are based solely on a rational model of persuasion. Because many researchers and clients found that literal recall and persuasion-shift tests provided only limited insight into likely ad effectiveness, pre-tests developed in recent years have tended to focus on the four key areas I just mentioned: involvement, rational and emotional relevance, and brand engagement. These dimensions are the bedrock of ad effectiveness, without which an execution is highly unlikely to benefit the brand. We would always want to measure responses on these dimensions, but there is still plenty of scope for customization in the areas of emotional response, likely impact on brand affinity, the effect of multiple media, and so on. Jason, if your client’s pre-test does not measure all of these things or provide enough flexibility to adapt to changing needs, then by all means ask them to get a new pre-test.
Now let’s turn to the second question asked by Jason’s client.
To what degree do you believe that the current testing processes (ASI, Link or similar) is [sic] an accurate predictor of in market success?
Jason backs up his “short” answer—”Copy testing is not predictive of in-market success”—with a number of quotes and assertions, most of which I would challenge.
For example, he quotes IPA board member Tim Broadbent as saying “There is no evidence in the public domain that [pre-testing] is predictive.”
In fact, most major pre-testing companies seek to validate their predictions against sales effects. At Millward Brown we have been documenting the relationship between the Awareness Index (our prediction of brand-linked memorability) and persuasion for over a decade.
We can offer a “gold standard” validation when we have an apples-to-apples comparison, with multiple tests for the same brand ranked against in-market sales response for the same ads measured by market mix modeling. In these cases, fits of 90%+ between pre-test results and sales response are not uncommon. Where market-mix modeling results are not available, we default to a “silver” standard, comparing raw share or sales figures pre- and post- advertising. Even though this data covers many different brands, categories and countries, and does not correct for the influence of other marketing variables, it still demonstrates a striking relationship between the evaluative metrics from Link and the probability and magnitude of sales response.
I would also challenge Jason’s assertion that a study of the IPA Effectiveness Awards showed a negative correlation between doing pre-testing and winning awards. Millward Brown has tested one-third of the IPA award-winning ads, and the average Awareness Index of those ads was twice as high as the overall UK average.
Does this mean that pre-test predictions are perfect? Of course not. I agree with Jason that such predictions are not acceptable substitutes for judgment—but in my opinion, they are a damn good place to start. While the biggest problems with pre-testing may arise “when research is used as a substitute for judgment,” I’d prefer to base decisions on informed judgment than gut feel.
I also agree with Jason’s comment on research: “the value depends on the attitude of those using it.” Last year, inspired by a question asked at the Account Planners Conference in Miami, I wrote a post provocatively titled “Is the Link pre-test the equivalent of the Smith & Wesson 500?” The ensuing debate touched on many of the topics addressed here, but might be summed up as “attitude is everything.” If people approach pre-testing results with open minds, they are likely to find that the results not only predict in-market effects, but also identify areas for improvement and provide learning to be applied to future development.
Having put the case in defense of pre-testing, let me just say a few words about one common misconception about emotions repeated in Jason’s post. Jason expresses concern that pre-testing emphasizes rational responses at the expense of emotions, referring to research which has demonstrated that “standard pre-testing methods under-represent emotional response.” Emphasizing this alleged bias towards rationality, he states “Advances in neuroscience and our understanding of the brain have shown that the vast majority of our decisions are emotionally based.”
That last point—that the majority of our decisions are emotionally based—has become the latest stick used to beat pre-testing. I don’t disagree with the statement; in fact, the latest scientific evidence suggests that all of our decisions are influenced by our emotional response. But to say that is to stop short of the full conclusion, which is that unless those emotions are experienced consciously, they will have little effect. In other words, decision making is influenced by emotion; it is not hostage to it.
Don’t take my word for it—read what Antonio Damasio says on the matter in his latest book The Feeling of What Happens:
- “Consciousness must be present if feelings are to influence the subject having them beyond the immediate here and now.”
- “When consciousness is available, feelings have their maximum impact”
And finally, as if he’s talking to Jason and others, Damasio says:
- “I did not suggest, however, that emotions are a substitute for reason or that emotions decide for us.”
So, in reference to Jason’s criticism of pre-testing, I would suggest that one reason it works is that people must be conscious of their reaction if an ad is to have a lasting effect on them. It’s true that people may find it difficult to vocalize their emotions spontaneously, but the Link test helps people identify their responses by providing them with a list of emotions. The approach works well, is customizable to a specific ad, and the results support the importance of emotional response in shaping our conscious reaction to advertising.
I’m thankful that I can end by agreeing with one of Jason’s last points. He says “Pre-testing can still be valuable in some cases as a diagnostic exercise - helping to understand what elements of an ad or campaign might be working and which aren’t, and whether they are working the way we thought they would.” In fact, I agree with this so strongly, that I’d go so far as to say that pre-testing can be valuable in all cases. But then I would say that, wouldn’t I?
Now, if you have read this far, it’s your turn. Does anyone agree, disagree or have anything to add?
(36 votes, average: 3.28 out of 5)
Loading ...
May 19th, 2007 at 12:14 am
Great stuff Nigel, thanks for picking up the ball and running with it. Really appreciate the thoughtful response. I think we agree in the end more than we disagree, and you’ve clarified a few important points that I didn’t express very well. I’m travelling and can’t respond right now but I’ll come back and take a crack at a response in a day or two.
May 19th, 2007 at 1:08 am
Very interesting post Nigel. Surely the benefits of pre-testing also encompass another set of goals:
1) provide due diligence - i.e. if my organisation is going to spend $3m on creative then i want to know that of all the ideas we’ve gone with, there is some 3rd party justification for the one we picked
2) When people argue against pre-testing and point to results which suggest little or no linkage with sales performance, they often neglect to mention that for each piece of copy which ran having been pre-tested, there are some pieces which never ran because the pre-testing was poor (do you have any numbers on this Nigel?)
3) If the IPA study showed a negative relationship between use of pretesting and IPA effectiveness wins it may also mean that these “amazing” campaigns were so obviously good that there was no need for pre-testing
One more comment – you’ve talked about pre-testing and post-testing (mix analysis etc) but advertisers also have the opportunity for in-market testing. It is possible to run different copy in different markets and analyse which is more effective. Yes there’s a cost to this but my guess is that digital technologies will reduce these costs in the near future.
May 19th, 2007 at 1:48 am
I think what comes out of this strongest for me on the first reading is the idea there’s no point testing for dimensions/objectives that are irrelevant. So here’s my heretical idea for the day. If a research report for an idea is more than 20 times bigger than the ad itself. It’s probably been tested to destruction, particularly given that a lot of interruptive advertising is 30-40 seconds of low involvement processing. Planners like research people are put under a ‘feel obliged’ pressure to ’show value’ to clients through volume of report output. This is not real value and as Nassim Taleb points out in The Black Swan (No. 26 NYT Bestseller). A lot of predictive data is there to reassure anxiety. Not predict the future.
What is great is that I’ve yearned for and never seen this conversation between research and advertising but that blogs are facilitating this discussion. The number of clients blogging on any subject, let alone important topics like this is a only a handful. There is a reason for this.
May 19th, 2007 at 8:49 am
Good morning. I am flattered and worried that you are posting comments to mb-blog.com on a Saturday! Thank you.
Jason, please do check back in with your comments. I hope this will become a truly productive debate.
John, welcome to the blog and thanks for your comments.
Yes, pre-test results can inform decisions regarding which ad will be the best one to run and can also be used to inform decisions on copy rotation, which one to run first and which ones deserve more weight.
I am not sure of the proportion of ads that get rejected at the pre-testing stage but my suspicion is that there are too many and too few! Let me explain.
Why too many? Because relying on pre-testing to weed out poor strategic choices or poor creative ideas is way too late. The earlier this can be done the better.
Why too few? Because we often see ads which test badly go on air because there is too much inertia in business process, i.e. we have air time booked we have to run something. Since poor pre-test results are pretty much guaranteed to track poorly in-market then it might be better to run existing ads again than use the new one simply because it is new.
To your point about the IPA, I have a slightly different but complementary suggestion. Many IPA Award winners are smaller company brands, which typically do not pre-test, but which have greater potential to grow than big, well-established brands. They are also more likely to have “new news” which is a proven driver of short-term sales response. I am willing to bet that if you analyzed the testers versus the non-testers you would find many more small brands in the latter group. This difference would be more than enough to explain the relative small difference reported by Peter Field.
Lastly, yes, in-market testing is a good way to test for copy effects provided the markets are really matched. Of course, the best choice is BehaviorScan where households are matched within market.
Charles, many thanks for facilitating the pre-testing discussion. To your point, I just wish we could train people to focus on what matters rather than report every (irrelevant detail). Believe me, it is not just the researchers that are at fault.
Ideally we would set some basic action standards for measures relevant to the ad objectives as follows:
Results confirm the ad resonates with the target audience and is likely to have the intended effect - don’t report the detail, don’t debate it, run it!
Results suggest problems but potential to address them - call a work session with key players from the agency, client and research team and dig into the results to figure out whether things are indeed off-track and how the situation might be remedied. Issues are often “structural” and can be addressed by re-editing, changing voiceover and even choice of music.
Results identify real issues in the way people respond to the ad - learn from it but don’t run the ad.
While the last recommendation may raise planners’ hackles I firmly believe that it is the right thing to do. Critical to success of my recommendation is that the agency team helps set the criteria for success and failure. Having them imposed without discussion is just a recipe for disaster.
May 21st, 2007 at 7:48 am
Nigel, I haven’t read Jason’s original blog entry (so correct me if I am wrong), but I think that one of the points that you are particularly critical of has some merit:
“If we had to pre-test,” he says, “we’d figure out what method suited each ad best.” After a discussion of different ad styles and content, he restates that idea. “The method should be dictated by the objectives of the advertising.”
While I certainly don’t agree with switching copy testing system on an ad by ad basis, I believe that it is true to say that no pre-testing system is perfect. Most copy testing systems prejudice some genre’s of advertising in some regard. The discerning research buyer needs to factor that into their choice of pre-testing system.
….and yes, this is even true of Link, I won’t abuse your blog by listing Link’s faults, but it is a little naive to assume one copy testing approach/company ‘fits all’.
May 21st, 2007 at 8:58 am
Hi Jim,
Thanks for joining in the debate. Much as I would like to I cannot disagree with your statement that no pre-testing system is perfect. That’s why it is so critical people engage with the ad objectives and test methodology up front and the findings afterwards. But does that criticism rule out the value that a systematic approach to pre-testing can bring? I don’t think so.
As to your issues with Link, I look forward to hearing your reaction to the latest enhancements which are being rolled out right now. They are designed to address issues like the latest learning on how emotions influence our conscious response to brands, the need to measure 360, integrated communications and so on.
Again, thank for the comment. It is great to see someone from the client side join in the debate.
Nigel
May 28th, 2007 at 8:39 am
I’ve finally got around to posting some more thoughts… interested to hear your take on it.
http://lbtoronto.typepad.com/lbto/2007/05/pretesting_part.html
J
June 14th, 2007 at 1:52 am
Nigel, I’m afraid your quotes from Damasio are a little misleading.
The first quote you use - “Consciousness must be present if feelings are to influence the subject having them beyond the immediate here and now.” - is simply meant to state the obvious, which is that if a subject is unconscious then they will not be able to think about and debate the feelings they have experienced. Clearly in an advertising situation if a subject is not conscious then the feelings will not be linked to anything. But it does not say that a high level of consciousness is necessary, just that some level of consciousness is necessary. It is perfectly possible to have a low level of consciousness, process emotion in advertising, link it to the brand, and change your buying predisposition without ever thinking hard about or recalling the emotion you experienced.
The second quote - “When consciousness is available, feelings have their maximum impact” – appears to suggest that feelings are dependent upon high levels of consciousness in order to have the greatest effect. Read the full quote and you will see that this is not true. What Damasio is talking about is how we can best use our reason to combat what he calls the ‘pervasive tyranny’ of emotion. Here it is in full:
“The pervasiveness of emotion in our development… connects virtually every object or situation in our experience. Whether we like it or not that is the natural human condition. But when consciousness is available, feelings have their maximum impact, and individuals are also able to reflect and to plan. They have a means of to control the pervasive tyranny of emotion: it is called reason. Ironically, of course, the engines of reason still require emotion, which means the controlling power of reason is often modest” (Damasio 1999: 58).
So what Damasio is saying is that it is only when we are fully conscious that we can counter-argue the effects of emotion. In other circumstances – for example, when low levels of consciousness are present, as in most TV viewing – emotional influence runs riot.
The third quote - “I did not suggest, however, that emotions are a substitute for reason or that emotions decide for us.” – seems eminently sensible. After all, it is conscious thinking that controls the muscles that pick the pack off the shelf, not emotion. But again, what Damasio is really talking about is the critically important power of emotion in decision-making. Here is the full quote:
“I suggested that certain levels of emotion processing probably point us to the sector in the decision-making space where our reason can operate most effectively. I did not suggest, however, that emotions are a substitute for reason or that emotions decide for us. It is obvious that emotional upheavals can lead to irrational decisions. The neurological evidence simply suggests that selective absence of emotion is a problem. Well-targeted and well-deployed emotion seems to be the support system without which the edifice of reason cannot operate properly. These results and their interpretation called into question the idea of dismissing emotion as a luxury or a nuisance or a mere evolutionary vestige” (Damasio 1999: 42).
Incidentally, ‘The Feeling of What Happens’, published in 1999, is not Damasio’s most recent book. His most recent book is ‘’looking for Spinosa’ (Damasio 2003), where on page 149 he shows a diagram of decision-making with a path A which is rational and a path B which is emotional. On the same page he states quite categorically: “On occasion, path B can lead to a decision directly, as when a gut feeling impels an immediate response.” The ability of emotions to direct decision-making has been validated empirically both in experiments and in real life (See Heath Brandt & Nairn, Journal of Advertising Research, 2006). Not least you should note that the recent IPA report by Les Binet and Peter Field, referencing 880 successful advertising case studies, concludes that ‘overwhelmingly the most effective campaigns are those that focus on the emotional rather than the rational’ (Marketing magazine 13th June 2007: 28).
You say in your blog ‘So, in reference to Jason’s criticism of pre-testing, I would suggest that one reason it (pre-testing) works is that people must be conscious of their reaction if an ad is to have a lasting effect on them.’ I have to say I think that is not supported by what Damasio is saying. If anything, Damasio is saying that if people don’t pay attention then emotions run riot and rule their subsequent behaviour. After all, it is emotions, not reason, that mostly underpin consumer decision.
Dr. Robert Heath
University of Bath School of Management
r.g.heath@bath.ac.uk
June 14th, 2007 at 8:09 am
Hi Robert,
I think you must be very careful when quoting Damasio and applying the same quotes directly to both consumer behaviour and advertising objectives.
I would tend to agree with your statement “If anything, Damasio is saying that if people don’t pay attention then emotions run riot and rule their subsequent behaviour. After all, it is emotions, not reason, that mostly underpin consumer decision.”
This, however, relates to the purchasing decision. The less conscious the purchasing decision (i.e. the less attention you are giving the decision) the more opportunity for emotions to ‘run riot’.
Viewing an advertisement (with little or no attention)is an entirely different kettle of fish. The decision you are making at the point in time is not whether to buy the brand, but whether to give the advertisement attention. I don’t offhand recall Damasio saying anything pertaining to this. Here LeDoux is applicable when he says that the function of emotions is to make you give more attention to the ad (or snake).
So testing an ad for emotional content is perfectly valid from two perspectives: the emotion will increase the attention the advertisement gets, hence its memorability; and then the emotion will impact on the brand decision at a later stage. At both stages the function of emotions might be different because in the buying stage you are recalling feelings about the brand, at the exposure stage you are interpreting the emotions projected by the ad.
Regards,
Erik.
June 14th, 2007 at 8:54 am
This is fabulous. It is fabulous for Robert to join the discussion here. And it is fabulous because his posting absolutely gets to the meat of our agreements and disagreements.
Robert writes,” “It is perfectly possible to have a low level of consciousness, process emotion in advertising, link it to the brand, and change your buying predisposition without ever thinking hard about or recalling the emotion you experienced.” I think we’d completely agree with that! However, what Robert doesn’t say here, but did in his 2007 MRS paper: “people can be powerfully influenced by communication that is processed with low attention and of which they have no conscious recall.” The Damasio quotes are in reality hugely damaging to this idea.
Watch how Robert tries to bluster his way through; I don’t know if he is trying to kid himself or us.
First of all he builds a straw man. Robert writes: ” “When consciousness is available, feelings have their maximum impact” – appears to suggest that feelings are dependent upon high levels of consciousness in order to have the greatest effect.”
“High levels”? Who said high levels? This is a good example of how Robert ascribes to us a position that advertising needs to achieve a high level of conciousness. Damasio didn’t say high levels, Nigel didn’t say high levels. Back in 1994 Gordon Brown was writing, ““All the evidence is that consumers are very reluctant to ‘engage brain’ while watching TV”. High levels would be great, but we’re all being realistic about the likelihood of this. At Millward Brown we are talking about paying sufficent attention so that, at a short time later, consumers can report that they remember having seen something about the brand recently. That’s all.
Having tried to paint us into an indefensible box, Robert then tries to twist Damasio’s words. Robert writes, ” “Consciousness must be present if feelings are to influence the subject having them beyond the immediate here and now.” - is simply meant to state the obvious, which is that if a subject is unconscious then they will not be able to think about and debate the feelings they have experienced.” Well, read those sentences again. Is Damasio talking about thinking and debating? No, he is talking about feelings. Nothing more than feelings.
Robert also summarises; “What Damasio is talking about is how we can best use our reason to combat what he calls the ‘pervasive tyranny’ of emotion.” This is wonderful, because he goes on to repeat the full quote. Damasio says, “when consciousness is available, feelings have their maximum impact, AND individuals are also able to reflect and to plan.” The emphasis is mine, but the point is clear - Damasio is not only talking about needing conciousness to counter-argue the effects of emotion: he is talking about conciousness boosting both feelings AND reason. Robert is staunchly trying to ignore the fact that in both quotes, even in context, Damasio is arguing that, without consciousness, feelings have a low, short lived impact. Because this would undermine what he has argued for since 2000.
For the third Damasio quote; “I did not suggest, however, that emotions are a substitute for reason or that emotions decide for us.” Robert initially seems to agree - yet three paragraphs later flatly contradicts this with, “It is emotions, not reason, that mostly underpin consumer decision.”
To be clear - we’ve monitored advertising working emotionally, rationally, and emotionally and rationally. We are not debating the impotance of emotions in decision making. Nigel argues somewhere above that “all of our decisions are influenced by our emotional response”. Where we fundamentally disagree with Robert is when he argues (as he did in his MRS paper) that “Most important of all, emotional content in advertising is significantly correlated with lower, not higher, levels of attention”.
June 14th, 2007 at 4:30 pm
Well, this is an interesting and unexpected addition to the discussion on pre-testing!
First, Robert, thank you for posting your comment. I do appreciate you doing this, even if I beg to differ with your interpretation of the Damasio quotes. I stand by my interpretation of them, even with the expanded quotes you provided.
The amusing thing is that I have just finished writing the first draft of a POV on search advertising in which I argue that search is not the rational process many people think it is, because it is subject to people’s preconceptions and emotional responses to brands. Then I check in at the blog to find I am being accused of downplaying the role of emotions in decision making! And one of my colleagues seems to have got carried away by his emotions to post a rather robust defense on my behalf.
Unlike Dom, I am not going to argue interpretation with you. Only Damasio really knows what he was trying to say and whether it is applicable to our discussion, and, with that in mind, I sent an e-mail asking for his interpretation. I will let you know if I hear back.
Meanwhile here are my thoughts in response to these three comments:
As I said, Robert, I stand by my interpretation of the Damasio quotes. I firmly believe that what he suggests is that unless an emotional response is available to introspection it will not have a lasting effect on a person’s predisposition toward a brand. I do not say that someone must reflect consciously on that response at the time of viewing. Rather I think they need to be conscious enough of their feelings that they could reflect on it if prompted to do so. That’s a good thing for the two of us, because both of us have recommended the use of words as an aid to peoples’ introspection, helping them tell us how they feel in response to seeing an ad.
To Erik’s point, I think we must make a distinction between the emotional response that directs our attention to an ad and the emotional response to a brand at the point of purchase. The stronger the emotional response, the more attention we pay to the ad. The more attention we pay, the more memorable the content will be. Provided the memories and feelings evoked by the ad are firmly linked to the brand in people’s minds, then they have a chance of influencing subsequent purchase decisions. As Dom suggests, I think it is perfectly possible that people cannot then recall the source of that impression at a later date, even though with the right prompt they may remember that they had seen the brand advertised.
Do I believe that emotional response has a direct influence on peoples’ brand choice? Absolutely, and I said so in the original post. To suggest otherwise is disingenuous, and I would recommend reading some of the Millward Brown papers that confirm that emotional campaigns tend to be more effective than rational ones (for established brands). I think Dom is right to suggest that the most fundamental difference between us is the issue of how much attention is enough for advertising to be effective. I do not believe high levels of attention are necessary for TV ads to have an effect, but I do maintain that the higher the degree of attention an ad gets, the more memorable it will be, and therefore there will be a higher likelihood of observing a subsequent effect.
August 20th, 2007 at 6:01 pm
Very interesting discussion and glad that it’s remained civil. Having sat in rooms with pre-test practitioners and agency planners and creatives, that’s no mean feat.
I have a question and a comment.
There are certain ads - arguably for a lot of people, the most powerful ads - that I can’t imagine having got through pre-testing, including Link testing. For example if we wound back the clock and put the fabled 1984 ad (http://www.youtube.com/watch?v=OYecfV3ubP8) into pre-testing I’d be surprised if it would have made it through. There are a number of reasons I think this: 1) the gap between animatics and final execution would have made getting a real sense of the ad problematic (something that may apply to a lot of ads) and it was talking of something that consumers could only imagine (the Mac didn’t appear anywhere in the ad, and even if it did, giving a sense of how revolutionary the experience would have been would be difficult). And yet the switchboard at Apple lit up and the ad is the stuff of legend.
Even for less revolutionary products I have similar reservations that a pre-test would have smiled upon them, for instance (to stay in Appleland) the original iPod ads.
I’d be interested in Nigel’s view on this, particularly given your intimacy with Link testing. Do you think these would have made it through?
And the comment: A report has just been released by the IPA, “Marketing in the Era of Accountability” which was compiled by Les Binet and Peter Field, two men men who’ve had a longstanding relationship with the IPA generally and the IPA DataBANK in particular. Under the heading “Beware of pre-testing” (p99) they’re pretty clear: “Cases that reported favourable pre-testing results actually did significantly worse than those that did not.”
Sorry to come into this so late, but only just came across this link.
August 28th, 2007 at 10:23 am
Hi Gregg, sorry not to have got back to your comment before now but here are my thoughts on your question and comment.
First the question would the fabled 1984 ad or initial ones for the iPod have made it through Link pre-testing?
I am absolutely sure that these ads would have been found to produce a strong, positive response from members of the relevant target audience. I am sure people would say they found them intriguing, involving, interesting, enjoyable, and different. So the first hurdle of effective communication would be passed - the ads would get peoples’ attention and be remembered.
What is less clear is whether or not people would understand what was being advertised. Typically, the lack of clear brand-linkage would be a problem, but we always need to bear in mind that advertising communication does not happen in a vacuum. What else might mean that the ads would achieve their intended purpose? Here are three suggestions:
1) The creative itself may be so intrinsically interesting that people want to find out what the ad was for.
2) People are informed by friends or the media or make the linkage through other means, e.g. a TV campaign for Tetley tea in the UK took off once the characters in the ad were featured on the package.
3) The product featured is obviously interesting and relevant to the intended audience - they don’t need to remember the name they just need to remember what it is and where to find it.
The Link test does include questions that ought to allow us to anticipate whether the ad is going to make people try to figure out what is being advertised and want to talk to other people about it. We also assess the degree of active interest in the product. The big challenge is to anticipate whether other non-advertising driven factors will play a role and, of course, there is nothing to stop us adding additional questions to address such issues.
The real question to my mind is whether or not the client’s action standards are flexible enough to allow for possible exceptions.
Turning to your comment, I have to admit to not being familiar with the analysis in detail. I do know that the majority of IPA Award winners do pre-test their ads. This leads me to wonder whether the analysis has been balanced to take account of the nature and size of the brands that did not pre-test versus those that did. Non-profits and lesser-known brands or new products often return much higher returns to advertising than established brands. The former may be less likely to pre-test than the latter, resulting in the difference observed.
What I can say is that IPA Award winners typically achieve tracked Awareness Indices three times that found in our overall UK database. That confirms to me that high brand-linked ad awareness is something which is related in-market success.
December 4th, 2007 at 6:57 am
OK, this isn’t about Link testing, but it is about researching the 1984 spot:
http://scampblog.blogspot.com/2007/10/research-is-waste-of-time-and-this.html
December 4th, 2007 at 7:07 am
Hi Jason, yes, I guess I missed that, mea culpa!
This is an interesting example and I thought about writing a post about it. Every qual researcher I have talked to about this case study suggested that it was an example of “bad qual” and no good researcher would accept peoples’ comments at face value. I am not sure that I agree. Researching rule breaking ideas at an early stage is tough for the reason you suggest. That’s why we are exploring different ways to help people visualize what something might be like and then share and explore their own interpretation and thoughts in a web 2.o environment. Hopefully that might get past the fact that people can only react based on what they know and understand now.
Will it be any more successful? I guess we will find out!
Cheers, Nigel
December 5th, 2007 at 6:26 am
I’ll be glad to see anything which helps resolve this… although I wonder how we’re ever going to be able to test for the effect of something becoming popular and thus overcoming the don’t-know-so-don’t-like barrier… (I’m thinking the Cadbury Gorilla or the Aeron chair or cash machines or the Walkman…)
(also, I’ve posted a reply to your comment over on my blog)
March 1st, 2008 at 5:56 am
I’ve just been going through Link Tests for an ad that did well in Shanghai where the product sells well and performed poorly in Beijing where the product is less popular.
I’ve made the point that its not unnatural to expect cities that aren’t familiar with a product to have a lower enjoyment of their advertising.
But here’s what I’d love to see Nigel. Why don’t we find a bunch of Brits from around the world who have missed the Gorilla ad and Link Test it. Then put it through all the amendments that the process might suggest and see the results.
Are you up for it Nigel? It would be an investment for Millward Brown if it made the ad better
March 1st, 2008 at 1:27 pm
Hi Charles,
I would agree that it is not unusual to see differences due to familiarity but I guess the question is whether or not the ad assumes a certain degree of familiarity with the brand. We often see differences because people have not been properly introduced to the brand and what it stands for - particularly in countries like China.
As to the challenge…mmm…why Brits? Are you suggesting that you have to have familiarity with CDM in order for the ad to “work?” If so, I suspect that is probably the case. As for testing the ad, why do it twice? I’ll try getting hold of the results again now that the ad is old news.
Failing that, I’ll see if anyone with a budget is up for this. You don’t honestly think they let me spend money do you? Even this blog is a skunkworks project.
March 2nd, 2008 at 10:56 am
OK. Gorilla is a British ad. Most people will have seen it so the sample is to find Brits that haven’t seen it. It wasn’t tested as I understand it and my point is that it should be tested and amended as per MB’s feedback to see if we can at least qualitatively determine if Link Testing makes a better ad. At the very least it would make a fascinating report to read on recommendations for improvement.
But seriously. MB should do this and see what happens. It’s as close as researching the research methodology as we can possibly do.
As for your last comment I laughed out loud
March 2nd, 2008 at 1:35 pm
Working on it. If I get anywhere I will put up a new post to set the ball rolling. After all, we need to have some ideas on what to look for, e.g. are we concerned that people might not remember which brand is featured, what emotional response are we looking for, is it just designed to create buzz, etc. That in itself should be an interesting exercise. Cheers
April 3rd, 2008 at 6:52 am
Nigel - Am I being unfair if I see a disconnect between your earlier assertation (Link measures are measures everyone should want for every ad) and later comments about small brands, new products, not-for-profits and qualifying remarks about brand link scores not being able to take into account other factors?
“What have the Romans ever done for us? Well, except for the roads, sewers, education, law & order…..”
Overall, thanks for starting this discussion and particularly for addressing two things that annoy me no end: One is cookie-cutter testing with little context (admit it - it happens waaaay too much and testing companies don’t always kick up a fuss.) The other is the attitude of the people recieving the data. I have worked with too many client who just want a single ppt slide with their final score. “Cut to the chase good sir - are we making this ad or aren’t we?”
Perhaps some loose code of conduct could emerge from this for all agencies, clients and researcher to follow. Heck, the person who poured my coffee this morning gets checked by his manager to see if the cups are clean and the coffee is fresh. Yet, when it comes to expensive research and million dollar campaigns we - all of us - often just go through the motions.
April 3rd, 2008 at 7:24 am
Hi David,
Well it would not be the first time I have been inconsistent but I don’t think this is the case this time around!
All I am saying is that a pre-test can only predict based on the conditions that exist at the time of the test. To Jason’s point, if an ad gets tons of coverage in the press and social media then it will have far more impact than the initial pre-test responses might suggest. We can report that people say it is an ad THEY would talk about but cannot judge whether the pundits will do so.
Again, the point about small brands etc is that they are innately more sales elastic than big brands. The point I was making relates to the IPA Award analysis. Small brands tend not to test at the same rate as big. Even if they did, exactly the same response on all pre-testing metrics would still yield a bigger percentage sales increase in-market, making it look like the ad for the smaller brand was far more effective. 1% point increase on 5% share is a 20 percent increase, whereas it would be 2 percent on a 50% share. We calibrate our sales response predictions to take account of this but effectiveness awards tend not to do so! The IPA sought to deal with this by creating a “longer and broader” sales effect category - if I remember that correctly - but judges are only human, big numbers are more compelling than small ones!
I think your suggestion of a code of conduct is really interesting. A joint group of agencies, clients and researchers ought to be able to hammer something sensible out. Does the MRS have anything specific to pre-testing?
Cheers,
Nigel
June 21st, 2008 at 1:05 pm
[…] da MwB, rolaram duas discussões muito interessantes que em certo ponto acabaram neste filme. A primeira falava especificamente sobre a validade de pré-testar material como 1984 (lembram-se disto?) e […]