Yesterday I attended a session hosted by the ARF Council of Advertising Effectiveness. Entitled “Is My Ad Engaging? The Value of Biological and Cognitive Copy Testing Methods,” the session seemed to promise some interesting debate from the lineup of speakers. But while the presenters expressed widely diverging views, little real debate occurred. Maybe this post will stir some up.
Rather than review the entirety of the proceedings, I will provide an overview along with my key takeaways. I am not sure that I heard much that was new, but the proceedings did help me to clarify my own views on the topic.
Before I get into my comments, here is the panelist lineup:
Caroline Winnett, VP, Marketing, NeuroFocus
Lee Weinblatt, CEO, PreTesting
Graham Page, Global Director of Innovation, Millward Brown
Kirsten Zapiec, VP, Advertising and Brand Performance, TNS
The original ARF invite to the event listed five questions which each presenter was meant to address in a 20-minute presentation. My comments will be organized around those questions, along with my take on how well each question was answered.
What are the key measures used by each approach?
The companies represented covered an interesting continuum, from a new company which relies solely on biometric measures to assess an ad’s performance (NeuroFocus) to an established one which relies solely on direct and projective questioning (TNS). The other two companies, Millward Brown and PreTesting, occupy the space in between. As we learnt from Lee’s presentation, the standard procedure of PreTesting combines biometric and behavioral metrics with traditional survey questions. Graham stated that Millward Brown recommends a “horses for courses” approach, suggesting that, in addition to standard introspective, projective questions and non-verbal tasks, we would recommend additional biological measurements when client’s objectives warranted the additional cost.
Caroline offered us the interesting analogy of a doctor who asks you what your symptoms are, but also measures your temperature and pulse. I assume she meant that both questioning and biometrics are required to make an accurate diagnosis. After all, there could be several reasons why your pulse might race in the doctor’s consulting room: a) you are really ill, b) you had to run to make the appointment, c) you are being examined as the prelude to a vasectomy, d) the nurse is rather attractive …
Takeway: Although biometric measures are considered to be scientific and more accurate than those based on traditional survey questions, they are still subject to interpretation. In fact, what we know about biometrics is either derived from studying abnormal responses, or from what people tell us, consciously, about how they feel. Caroline suggested that learning about how the brain works is inferred from studying people with damaged brains or illnesses like ADD, in other words, the comparison of abnormal response to normal response. Graham suggested that our understanding of what brain response means is also highly dependent on conscious interpretation of a stimulus. For instance, people usually report that they fear snakes. Showing people a picture of a snake evokes a certain response in terms of brain activity, which we subsequently label “fear.” Given that our conscious mind interprets that response as fear after the event, how can we be sure that the interpretation is correct or consistent?
What evidence exists to validate that each approach can predict advertising effectiveness?
Only Graham addressed this question directly. He showed that the combination of persuasion and predicted brand-linked memorability (or “recall”) from Millward Brown’s Link pre-test was related to changes in market share, as measured by companies like Nielsen and IRI. However, neither metric relies on biometric response measurement.
Caroline offered that NeuroFocus used the responses to ads which were reported to be effective “in market” to derive their three cognitive requirements of effectiveness: attention, engagement and retention. While Lee suggested that recall alone was not predictive of effectiveness, neither he nor Kirsten offered any validation of their approaches (beyond specific cases).
Takeaway: As yet there is no compelling proof that biometric measures such as Electro-Encephalography, Galvanic Skin Response or Saccadic eye movement relate to sales response.
How does each method facilitate the creative process and the optimization of ad effectiveness?
Caroline addressed this point directly and highlighted several benefits of the NeuroFocus approach:
- The ability to identify the cause of a particular response.
- The ability to identify ways to compress an ad, i.e., in order to transfer a TV ad to online.
- The ability to predict “wear-out”
Lee also provided examples, focusing on print ads and identifying several instances where ads failed to achieve their objectives simply because of how the content was laid out.
Takeaway: Yes, biometric data can help identify precisely when people respond to an ad on basic dimensions, e.g. increased or decreased attention, visual versus auditory focus. Eye tracking can also help identify the focal point in an ad. Accurate coding of open-ended recall and tasks like Millward Brown’s interest trace may get similar results in most cases, but direct measurement is undoubtedly better.
However, predicting wear-out on the basis of random exposures across a few minutes, versus real-life exposure over weeks or months? Sorry, but without real evidence I don’t buy it. What I think NeuroFocus is really measuring is the familiarization resulting from frequent short-term exposure to the ad. Does this mean that the ad will be ineffective when it is seen again several weeks later? No, the ad will be recognized, and that fact alone will stimulate memories and associations at a preconscious and possibly conscious level. The effect will shift from adding or changing brand associations to reinforcing those impressions.
How does each method measure consumer engagement with the creative concept and brand idea?
This is where the audience questions really hit home.
First Scott McDonald, senior vice president of research at Conde Nast, asked whether it was possible to distinguish the response to bad strategy from the response to poor execution, particularly given the time pressures involved in pre-testing. Lee focused his response on the fundamental issue of whether anyone would listen to the feedback, but Graham suggested that poor strategy often resulted in an ad that was engaging but failed to be persuasive. Graham also recommended testing strategies and ideas as early as possible in the ad development process, an idea which seemed to meet with general agreement.
Marissa Sison, director of analytical methods and operations for General Motors, followed Scott by asking if a “leap of faith” was required to understand whether an ad that evoked a positive biometric response communicated the desired brand positioning or not. The answer seemed to be that, like the doctor, you need to both ask and measure response to provide a proper diagnosis.
Takeaway: The conclusion that you need both verbal and biometric measures is fine, provided the situation warrants it and the additional cost is justified. Many times we visit the doctor only to be told “Take some ibuprophen and get some rest.” Nobody wants to pay for an MRI when all they have is a headache.
Are cognitive and biological methods measuring different aspects of how consumers respond to ads, or are they measuring the same responses in different ways?
Suggesting that Graham’s presentation had provided evidence that biometrics and survey response provided the same insights, Don Diforio of the ARF posed this dual question directly to the panel at the end of the session. Lee, speaking on behalf of PreTesting, gave an unequivocal “yes” to the first question, stating that he did not see the same results from the two different approaches. (But then I would assume his tests are designed not to provide duplicative information). Caroline suggested that the NeuroFocus approach added to traditional techniques rather than replacing them.
As the session closed, Don asked the panelists why they thought so many marketers expressed concern with pre-testing, even though the practice is widespread. He asked, “What are you not measuring?” Perhaps not surprisingly, no one really answered the question, although it is clear that in their own way each of the companies is seeking to identify something new that will add value to the practice of pre-testing.
To my mind, the important issue is not what we could do better. Of course we need to continue to try different approaches to improve prediction and diagnosis. However, pre-testing – no matter how it is done – is not going to guarantee “success.” My suspicion is that people are not really complaining about the techniques as much as they are complaining about the process. They want success to be easier, with less hassle, and fewer hurdles to clear along the way.
Inexperienced people who botch a task often blame the tool they used, failing to realize that they were at fault for using the tool inappropriately. In the same vein, I believe that some marketers look to pre-testing to solve their difficulties inappropriately: convince the agency to do what I tell them, convince the boss to give me the budget, prove it was not my fault the ad got panned by the media.
If that’s the case, then demanding a better pre-test is not going to help. Producing successful advertising is not easy. There are many “moving parts” which must come together to if an execution is going to be faithful to the original idea. If anything, pre-testing makes life difficult by giving you a clear idea of how an ad is likely to perform: well or poorly.
So what do you think? Is biometric data a worthwhile addition to a pre-test? What should we be focusing on in terms of future development?
(29 votes, average: 2.86 out of 5)
Loading ...
January 22nd, 2007 at 1:50 pm
Excellent synthesis of what was discussed at the ARF Ad Effectiveness meeting! While the panelists did a great job providing lots of insight in a very limited time, the meeting was most interesting to me for the questions that it did not answer. First, are there preconscious emotional responses to advertising that do not pass into conscious experience but strongly influence purchase behavior? Second, are we able to measure these preconscious emotional responses to advertising and understand how they influence behavior? I believe that, in a minority of cases, emotional responses to ads stay preconscious yet influence purchase behavior. I also believe that, for these unique cases, current copy testing measures do not adequately predict ad effectiveness. Going into the meeting, I thought that biometrics were the key to measuring preconscious emotional responses. Graham, however, rather convincingly suggested that biometrics were really measuring the biological correlates of conscious experience. Graham discussed how most biometric tools were developed by examining the biological responses occuring during certain types of conscious experience. If current biometrics are primarily another means of measuring our conscious response, then we are back to square one in developing preconscious measurement techniques. I think a lack of validated preconscious measures is a significant problem for copy testing. It leaves us open to the oft-cited criticism that our tools sometimes inappropriately reject effective creative executions. Our response to this criticism is usually lackluster because we show how our measures succeed “on average,” without being able to explain the exceptions when our measures don’t work well. My assumption is that preconscious emotional responses lay behind effective ads that current copy testing measures reject. As I said, though, this assumption was not discussed adequately by the panelists, leaving these important questions open for further exploration.
January 22nd, 2007 at 2:29 pm
Hi Don, thanks for this interesting and challenging comment.
You voice a concern that I suspect is shared by many. I do not pretend to have a definitive answer, but have spent some time thinking about this in relation to Robert Heath’s low involvement processing. Here are my thoughts.
I do believe that advertising can have a non-conscious effect on behavior because there was a preconscious response (in contrast with a conscious response had no direct link to purchasing behavior which I think is reasonably well accepted, e.g. a positive and conscious emotional response to a well-branded ad predisposes people toward the brand, even though they think no differently about the brand as a result of seeing the ad.)
I believe that pre-conscious effects these rely on repetition to be effective. Why do they need repetition? First to ensure that any new association with the brand becomes entrenched in memory, and, second, to overcome any pre-existing and conflicting associations. If an ad only works on this level it is probably a) very inefficient, requiring high ad spend to generate sufficient repeat, b) hard to detect in the short-term - not just in a pre-test but in tracking or sales monitoring. The effect would be far stronger for an ad that starts off by being processed at a higher, conscious level, so that subsequent exposure, even if not attended to consciously, ‘refreshes’ the initial impression.
Can I prove this? No. It is only my interpretation based on what I have seen and read. If I were an advertiser would I bet on this type of pre-conscious only effect to grow and maintain my brand? Hell, no!
January 28th, 2007 at 5:36 pm
Useful synthesis and interesting topic
I think the whole biometric or not debate is a bit of red herring. Understanding of how the human mind and memories work has moved on considerably and left our industry struggling in its wake. We are still largely trying to use tools designed on rather simplistic outdated understanding of the human mind – regardless of biometrics approaches.
Psychologists and neuroscientists now widely recognise that people’s knowledge and ideas about things are formed through two key types of memory, Explicit and Implicit. It is also worth reminding ourselves that advertising and brands are just a couple of varieties of things or ideas and not some special unique subset, so there is no reason to suppose that this would be different for them.
One explicitly memorises something when you know and can identify the source and the detail of the thing in question. Thus accurately recalling an ad, identifying the brand it was for and its content are all largely measures of explicit memory - whether measured through pre-testing or tracking methods. Ads that do well on measures of cut through etc all tend to be working through explicit memory.
Implicit memory is where you have some feeling for or idea about a thing/brand etc - but you can not identify the source or place that that idea emanates from. Ads that work mostly in this way will not do well in most pre-testing or tracking methodologies.
One does not necessarily follow the other – i.e. implicit memory is not the “pre-conscious” lead in to the “conscious” explicit memory. Over time, with or without additional stimulus, explicit memories may become implicit and vice versa.
Now think about the human mindset and attitudes towards commercial messages. We live in an age of marketing savvy consumers and if people know that the idea they have about a brand explicitly comes from an ad (explicit memory) how much credence do they give it? Probably not a lot, more likely they would typically think something like “well they would say that wouldn’t they” and discount it accordingly - unless it tallies closely with their existing preconceptions about the brand, good or bad and so does not change anything.
In contrast, if consumers have a feeling or idea about a brand that comes from an ad, but they can not identify that it does (implicit memory) it is far more likely to become embedded into their current view of the brand and influence their future preferences and behaviour. This may be thought of as the more “emotional” aspect of how advertising works, but the implicit memories are as likely about fairly hard nosed attributes such as price and quality, as about warm cuddliness.
Ads that are good at the latter but bad at the former will generally score badly in pre-testing and on traditional tracking measures reflecting cut through, message understanding, appreciation etc. However, they may well be very effective communications.
Likewise ads that work mainly the other way round will do marvellously in pre-testing & tracking - but quite likely deliver very little real brand benefit.
I believe that it is largely for this reason that pre-testing has proven to be a notoriously indifferent predictor of the real effectiveness of an ad on sales or consumers attitudes to the brand itself (not the advertising). This disconnect between what we measure and what really is effective is also why the really important brand measures such as preference and loyalty etc rarely shift in response to campaigns covered in ad tracking - even if the ads are ticking all the appreciation & cut through boxes, etc.
Pre-testing is undoubtedly good at identifying which ads will score well in traditional tracking metrics though - so it is a neatly self-fulfilling circle of evaluation - whilst it probably does little to make advertising any more effective in reality.
At the moment our best hopes are ads that work well on both explicit & implicit memory - at least they may score well and potentially also actually do something of value for the brand. But if you think about it, it is probably quite hard for an ad to be good at generating both forms of memory.
Furthermore implicitness may not be helped by too much repetition. The real source of wearout of an effective ad’s effect may well be that if you keep repeating it more and more people eventually remember the ideas it contains explicitly - and hence discount the message content.
As I said at the beginning the biometric debate is a bit of red herring as it is not clear that it is any better at identifying ads that work well than other methods.
Therefore, it does not address the main issue advertiser’s face - how do you identify ads that are good at generating the right sort of positive brand enhancing (ideally implicit) memories that really change the way consumers feel about and behave towards a brand.
Crack that and we can really move the game on. If either biometric or other new survey based techniques manage to achieve this then they can genuinely claim to be better than what we have now.
Just as it is very hard to turn a Philips screw with a flat head driver, sometimes a new tool is required to do the job properly afterall ……
January 29th, 2007 at 5:07 pm
Hi James, thanks for adding your comments to the debate.
I think we agree that biometrics are largely a red herring - until they can be proven to tell us something that existing techniques do not - but I can’t agree about your comments on explicit and implicit communication.
Why? I think your focus is on how memories and associations get formed. The question that pre-testing needs to address is will memories and associations taken from the ad stick to the brand in a way that comes readily to mind at a later date, irrespective of whether they were learnt implicitly or explicitly.
I would agree that people are less likely to give spontaneous feedback on implicit communication but aided questions tailored to the ad’s objectives ought to be able to do so. If people are unable or unwilling to report that an ad made them feel a certain way or implied something about the brand when aided with the desired impression immediately after seeing the ad a couple of times, I doubt it will have the desired effect.