If you think having your ads pretested is bad (and you can tell me all about that below), it could be a lot worse. In an internal memo picked up by the Fallon Planning Blog, the president of Cramer-Krasselt, Chicago, reported that their client, CareerBuilder, put the account up for review after the Super Bowl ads developed by the agency failed to rank in the top 10 in USA TODAY’s increasingly influential viewer poll.
So what does this agency-busting poll comprise of?
According to this article in USA TODAY, 238 adults in Texas and Virginia used hand-held meters to register, second-by-second, how much they liked each ad. USA TODAY polling editor Jim Norman admitted that the poll’s objective was to produce a simple measure of ad liking and nothing more. In this article in adage.com, he explained, “All we’re trying to measure is whether people like an ad or not. … We’re not claiming to measure effectiveness or sales or anything other than whether they like it.” (So - if the intent was just to get a liking score, why did they need such high-tech measurement tools? Why not just ask people how much they liked each ad? I can only assume that the survey’s authors wanted to use a method which would sound scientific and accurate to their readers.)
Now, I’m sure we’d all agree that, whatever their individual merits or flaws, all of the major pre-testing systems on the market today are more sophisticated than the USA TODAY poll. For example, here are just a few of the many ways in which Millward Brown’s Link test differs from the USA TODAY poll:
- The sample composition reflects the target audience
- Evaluation focuses on whether the ad will create the desired response from that audience
- Diagnostic questions help us understand how an ad might be improved
Since Link’s introduction in 1990, over 45,000 TV ads have been tested in 79 different countries around the world. Initially a test with a TV focus, Link has evolved to encompass print, online and multi-media campaigns as well. New measures of engagement, persuasion and emotional response have been added and refined over time.
For all that, Link remains a work in progress. While I believe it is light years ahead of the USA TODAY Poll as a means of evaluating ads, I know it’s not perfect.
Now I’d like to hear what you think. Tell me how Link could be improved by adding a comment below. No need to publish your real name or where you work unless you want to. All I would ask is that you tell us whether you have used Link or not, and how you think it could be made better.
In particular, tell us how Link needs to change in light of:
- growing consumer control
- increased emphasis on engagement
- new and diverse media opportunities
So here’s your chance! Tell us what you really think—but keep it clean, please!
Tags: Nigel Hollis, Millward Brown, Link, copy test,
(32 votes, average: 3 out of 5)
Loading ...
March 13th, 2007 at 5:18 am
Hi Nigel,
This particular episode exemplifies (exaggerates) the tension between agencies and clients over the role of copy testing. Agencies appreciate guidance and feedback to aid the creative development process: clients generally seek decisions. To my mind this represents a competition between art and science. In this case the barbarians over-ran the museum.
March 13th, 2007 at 9:53 am
Hello Nigel –
I agree with Philip.
This being said; I also see a disturbing trend in our business. The time allocated for the execution of our research work is growing shorter and shorter. Hence, solutions such as the one presented here become more attractive to our clients. As we struggle to deliver the turn-around times they want, they look at pseudo-scientific solutions.
Is ‘hard’ copy-testing goint to die as more ’soft’ solutions appear? I don’t think so, but we are pressured to evolve. We have to continue to deliver all the results from our methodologically sound tools in the times of the artsy quick solutions.
March 14th, 2007 at 6:29 am
Hi Nigel
Can I quote myself? Here’s what I’ve said in a previous blog entry about what personally I foresee for consumer insight and pretesting in the new media environment:
“In a world where on-demand, interactive new media is the only media, it might not be necessary for us to bother trying to research consumer attitudes at all, when we would have a near-perfect picture of what media people are exposed to over time, and the effects of that exposure on media consumption and purchase. In such a world, in the terms of consumer insight, attitudes become consumption itself.
“I would probably guess also that advertising pretesting, as we know it - out of ‘field’ - will become a redundant practice. What would the point be of having a separate out-of-field process for pretesting a piece of advertising, when it’s actual performance in field, amongst a perfectly controlled audience, can be monitored, perfectly?”
I’d love to pursue what I’ve begun suggest….
My whole article addresses in part the themes of growing consumer control, increased emphasis on engagement, and new and diverse media opportunities. You can read it here: http://www.kevinjoyner.com/blog/2007/02/but-with-great-power-comes-great.html
March 14th, 2007 at 7:44 am
Thanks for the comments so far.
Welcome to the blog Kevin. I would love to hear other peoples’ views on your comment (preferably those outside of research) before I give my response.
So what do you think? Is Kevin right that live-testing will make pre-testing redundant? Is behavioral data all you need to understand whether your marketing activity is effective?
March 14th, 2007 at 6:12 pm
What’s wrong with pretesting? Well it’s unfair to blame just pretesting, Lord knows advertising has an almost kamikaze obsession with it’s own navel, but it’s a start. Pretesting is almost a methodology in and of itself for a vaguely defined target called authenticity. Which suggests that the industry is looking; counter-intuitively I might add, the wrong way down the telescope. Quality blog I might add Nigel
March 15th, 2007 at 6:25 am
Just wanted to pick up in this: Is behavioral data all you need to understand whether your marketing activity is effective?
Surely behavioural data just tells us what has happened in the past (historical) and doesn’t answer why that behaviour happened?
I’m personally fed up of web retailer trying to up-sell me something based on my previous behaviour.
It usually shows that they don’t know me and have taken an estimated guess at what my interests are.
To quote an adage regularly used in the financial world: The past is not always the best indicator of future success, but at the moment it’s all we’ve got!
So surely attitudinal insights are key to creating greater success if behavioural data has it’s limits?
Perhaps technology can help combine both attitudes and behaviour for more powerful learnings.
March 15th, 2007 at 8:40 am
Thanks for the comments Charles and Jez.
Jez, thanks for picking up on the behavioral issue. Let me add a couple of thoughts to keep the ball rolling.
I agree that the fundamental problem with behavioral data is that it is backwards looking. It tells you what happened not why. And that’s a big problem if things do not go as planned. Not every ad has the effect we might hope for. Some may even have negative effects.
If we rely only on behavioral data to test ad effectiveness, albeit with small test and control groups, don’t we run the risk of diluting the brand’s equity by running lots of different executions, some of which “work” and some of which don’t? What does that do to a brand’s positioning and clarity? And once we realize what has happened it is too late to put the toothpaste back in the tube.
So while I believe pre-testing will change in the coming years I do believe its forward looking nature will mean it still has an important role to play.
Thoughts anyone?
March 15th, 2007 at 9:35 am
Nigel, you might be right I think, except for very big brands where the market is huge: new media technology, if not now then in the future, will allow us (won’t it?) to control the number of people who see whatever execution; and then track whether or not they clicked on it, what they looked at next, and what they went on to buy. This is really good; and what’s 500 people for each execution out of a market of hundreds of thousands or more?
March 15th, 2007 at 12:57 pm
Hi Kevin, yes, the risk for bigger brands will be lower, and I do see value to the test/control approach.
MB has used it in one form or another to test online ads since 1996. The HotWired Ad Effectiveness Study that year exposed one group of people to the test ad and a matched group to a control ad. Once people unloaded from the page carrying the ad both groups saw a request to take a survey. Differences between the two cells were due to the ad exposure. Dynamic Logic uses a more sophisticated but similar technique today.
One of the key findings from the HotWired study, however, was that the test ads created a predisposition to buy the brand that was incremental to the immediate clickthrough. Just going on behavioral response would have missed that effect. Check out the Journal of Advertising Research, June 2005, for a full length paper on the topic.
So, yes, an experimental design with a limited number of people is a good way to judge the short-term incremental effect of an ad, BUT it will not tell you why the ad worked or not, it may underestimate the longer-term impact on people who are not ready to buy at the time of exposure, and may not allow you to distinguish between user and non-user reactions.
Overall, I think the combination of behavioral data and attitudinal would give you the best of both worlds.
March 15th, 2007 at 1:14 pm
Hi Nigel, it’s easy to be polemical, but a blog certainly enables a conversation that is so rarely had between advertisers and research people. It’s tempting to discuss at length why pre-testing kills off brand energy in the long run by putting short term efficacy over long term interest and involvement. However it would be hypocritical for me to say it doesn’t work at all, and is the most inappropriate tool to use in all circumstances. The product, category, market, competitive set, customer and communications literacy of the customer are just the beginning of the discussion.
So rather than say the methodology is no good, I’d be more inclined to ask, what are the advantages of this process against the disadvantages. Where I believe the research industry has built a reputation for mercenary cheque taking is that I don’t I’ve ever heard any of a single large research network ever say in a meeting that there is little to be gained from a particular request to research. It’s a little bit like the Advertising agencies that have failed to be courageous and pointed out that perhaps a TV execution isn’t the right path to go for a client.
I could rattle on I suppose about the relevance of pre-testing advertising expectations from pre-post-modern consumers in developing economies but most of that has been covered by Mary Goodyear’s great work on marketing and provides a useful reason for doing ’safe’ executions over more complex stuff which while infinitely more effective, is also more difficult to achieve. It’s for this reason that most marketeers err on the side of safety while ultimately diluting their brands potential.
I hope my reasoning isn’t indecipherable. It’s a huge topic and one I don’t think a blog can untangle although it certainly is a welcome start.
March 15th, 2007 at 1:30 pm
Hi Charles, I think you just took us from a big topic to a vast one! The original intent of the post was to focus more on methodology than philosophy - if we can call it that - so I would like to focus on your question, “what are the advantages of this process against the disadvantages?”…except that Outlook has just reminded me I am late for a meeting. I’ll have to return to this comment later!
March 15th, 2007 at 1:54 pm
Yes agreed!
My ideal by the way of behavioural data would be that which is collected from ‘respondents’ who don’t even know they’re being observed. With the use of internet cookies, I can see being able to gather longer-term data, continuous with historic purchase data. The internet is very big, a really massive data resource; but of course there would be limitations… maybe I’m thinking of a huge number of people agreeing to keep a ‘cookie’ or two for, say, 12 months… or longer.
In this model of research the function of attitudinal pretesting becomes sort of closer to that of qual now - it’s an ideas tool. Really sophisticated behavourial data provides the proof for things.
I will certainly read that journal edition if I can.
March 18th, 2007 at 5:21 pm
OK, apologies for the lack of response for the last couple of days but it has snowed two days in a row in Vermont, the second day giving us a foot of perfect powder. Those of you lucky enough to ski out West or in Europe may not think much of this but on the US East Coast that’s a big deal!
Back to business.
Kevin, I will try to find out what the current state of the art might be but the “mega-panel” idea is one that some companies have been examining for a while. Might be worth a post.
Charles, one of the objectives of this post was to get feedback on exactly that issue: what are the advantages of pre-testing and what are the disadvantages? What does it do well today and what does it not? I would love to hear from others before I try to address those questions.
April 12th, 2007 at 7:59 pm
Hi Nigel,
I would just like to enquire a really quick question. Many pre-tests utilize self-reports, but how accurate will self-reports be in such cases? Wouldn’t on-the-spot experiments like these be more “accurate” per se?
April 13th, 2007 at 8:02 am
Yes, experiments like these ought to yield a more accurate picture of how people respond to ads and campaigns. But, as suggested above, they will not tell you why people responded as they did. The information could be very useful for optimizing the ad/media combination but may not be as useful for refining executions and providing learning for the future.
April 20th, 2007 at 5:27 am
Its fine to test an advert to see how it can be improved, however you can not test the life expectency of it. It may be a great advert when it first airs (look at the Trident ad, people thought it was smart - until someone called it racist), however they can quickly get bored of it - no one has developed a test for that - well no agency wants to know that detail
April 21st, 2007 at 12:16 pm
Not so fast David. Decay in likability, interest, relevancy et al are areas that many of us are interested in. There’s lots of ways that a commercial can be refreshed, and given the remix/recombinant culture that exists, I’m surprised that this topic isn’t on the table for the interruption communications model.
April 23rd, 2007 at 7:55 am
Hi David and Charles,
It is interesting that you should be raise the points right now because I have been engaged in a similar debate with some folks at Millward Brown.
The accepted ‘wisdom’ at Millward Brown is that TV ads do not wear out - at least in terms of creating the impression that the brand has advertised. In general I think that is probably true. What is less clear is whether or not the continued exposure to an ad that has been seen several times has any further benefit in the short-term, either in generating more sales or loyalty maintenance. What actually happens seems to depend on the brand, context and advertising task. There is good evidence from sales modeling that diminishing returns generally means ads produce less change in image and sales with increased exposure over a short time frame BUT if you bring the same ad back after a hiatus it may once again have an effect. Maybe people have simply forgotten the brand impression the ad created or new people have entered the decision making window for an effect to take place. To me this is about relevance of the impression.
With regard to response to the ad, e.g. likeability and engagement, then most ads do not wear out (at least based on our tracking data). A few ads, however, like ones based on a joke, do seem to become less engaging once people get the joke, and a few are start off irritating but on further exposure become really irritating. However, these seem to be the exceptions rather than the rule. The Trident ad that David cites sounds more like a context issue rather than an engagement issue, i.e. people only changed their reaction once someone had called foul?
May 14th, 2007 at 5:34 pm
Hi Nigel. Jason Oke of Leo Burnett Toronto has picked up on pre-testing and I’m hoping we can stir the debate up a bit once more either here or there.
http://lbtoronto.typepad.com/lbto/2007/05/pretesting.html
May 16th, 2007 at 7:47 am
Hi Charles, thanks for pointing this one out.
As you might expect, I do have a rather different view of the value of pre-testing and I certainly believe Jason’s point of view is partial at best and biased at worst. You can expect a new post on the subject on mb-blog.com before too long!
Cheers,
Nigel
May 16th, 2007 at 10:03 am
Anything to keep the discussion going Nigel. It’s an important subject.
May 16th, 2007 at 11:07 am
Charles, thanks for the hat-tip.
Nigel - looking forward to hearing your thoughts. I was hoping to get some debate going as I don’t think there’s enough constructive discussion on the subject (more often it seems to just be ad agencies complaining). And as a fan of your blog I very much respect your opinions on the subject!
June 9th, 2007 at 1:23 am
I had an idea last night that one of the weaknesses for pretesting is that there is no case study or memorable campaign whereby the research industry can proudly claim ‘advertising improved by pretesting’.
Furthermore I don’t think I know of any award winning work that has gone through the process. Of course winning awards is a bit of back slapping for the ad industry but its fair to say that although subjective there are still a lot of great executions that are subsequently recognised through the process. Am I completely wrong or even better has research ever had or intend to have anything to celebrate guidance great work. I do feel a post about this brewing on my own blog
June 11th, 2007 at 7:06 am
Hi Charles,
A prime example of an award winning ad that was pre-tested would be Guinness’s “Noitulove”.
I have a couple of other examples in mind but perhaps I will keep them to myself until you have written your post!
Cheers,
Nigel