I suspect that many people view me as a stick in the mud—as someone who’s unwilling to change and continually defends the status quo. Why else would I so often be asked to defend the value of existing methods of measuring advertising effectiveness? For example, I have been asked to speak at the Canadian MRIA Conference 2010, and though the final details are not in place, it looks like I’ll find myself on stage with my old sparring partner Jason Oke, debating the topic of advertising effectiveness research against the backdrop theme that research need to change radically.
Could research practices be changed for the better? Of course. But my view, which some people mistakenly interpret as resistance to change, is that to singularly focus on new methodologies is to miss the real problem. New tools are no good if people don’t know how to use them. And too many people seem to lack the skills and the will to maximize the value of existing research tools.
So, without getting into the specifics of ad effectiveness research, I’d like to explore the idea that research in general needs to change.
Ray Poynter provided a good place to start with his recent posting over on his blog thefutureplace. He focuses on a couple of key issues for quantitative research: the issues involved in generating a random sample, which are becoming more numerous and complex, and the assumption that people can answer direct questions (which, according to recent developments in the sciences, may be a bad assumption.) Correctly, however, Ray points out that researchers have long known that the use of norms, relativities and test and control methodologies will reduce any potential research bias.
In contrast to Ray’s focus on quality of samples, Stan Sthanunathan, insight boss at Coca-Cola and co-chair of the Advertising Research Foundation’s Online Research Quality Council, believes the problem with research is that it is too focused on understanding consumption and shopping behavior. Stan proposes “we need to understand the human condition, which you’ll only know by observing, listening, synthesising and deducing.” (Click here to read the full article.) What clients want, Stan claims, is not a better mousetrap, but research that will inspire them and help them take transformational action.
I tend to side more with Stan’s viewpoint than Ray’s. The quality of research samples has been an issue since I started in market research. Get over it. We need to be pragmatic about when it is worthwhile to pay for a really representative sample and when a convenience sample will do just as well. We need to look for convergence from different data sources to see whether they tell us the same thing. And we need to apply our judgment to whether or not the results are likely to be representative. As Alastair Gordon suggests in his comments on a previous post, we cannot simply take research findings at face value. We must ask what the data really means and what we should do as a result. It is a skill that may have fallen into disuse, particularly in the face of today’s time pressures.
At the end of his post Ray highlighted the three other issues facing market research; speed, cost, and lack of insight. Of the three, I am sure that Stan would tell us that lack of insight should be the one we try to fix.
But I am afraid that many clients will demand that new and more relevant insight must come out of research that is done quickly and cheaply. And that, I believe, is the fundamental issue we are facing in market research today. Increased speed risks focusing on the trivial, not the strategic. The online and CRM data “exhaust” promises vast amounts of free data but ignores the cost of data handling and the time required to analyze it. Will even more data and even less time make producing insights any easier? I doubt it.
Before I close, I want to touch on Stan’s call for market researchers to inspire their clients. There is nothing wrong with that desire. But in my experience, the best and most inspiring findings come when all parties involved collaborate in investigating the results from different research sources. The synergy that results from disparate sets of information and minds can produce transformational results for a business. But those experiences have been rare. All too often I have found an extensive analysis and presentation boils down to, “So what does the research tell me to do? Is it a yes or a no?” And whatever you say, the results often seem to be ignored. It makes one wonder why the research was commissioned in the first place. In cases like that I think the client gets the research they deserve. Maybe the missing element from market research today is lack of willingness on behalf of clients to engage with research and take actions based on the findings. Maybe clients should be inspiring market researchers as much as researchers should inspire clients.
OK, enough from me. What do you think? Why does market research fail to inspire? And where does the root cause lie? Please share your thoughts on this important issue.
(6 votes, average: 4.83 out of 5)
Loading ...
Email This Post
December 28th, 2009 at 2:57 am
Hi Nigel,
Excellent and a very relevant post. Wishb you a great 2010.
All 3 of you (Stan, Ray and you) make very valid points.
Stan has a very valid and often repeated point by the clients that research fails to inspire.
The root cause is quite complex and the blame lies at both the agency and the client end.
Do we have people who know the client business or rather have the desire & aptitude to understand the client business problems? The focus is too much on tools and turnaround with quality.
You have answered this quite well by saying that research cannot operate in a silo and come up with insights in isolation. There are very few clients where the research agency is a genuine business partner (Coke in China is actually one of them though MB HK team services them) and only in that scenario is research going to get outside the box.
In a lot of clients CMI teams are the sole interaction points for research agencies and they like good clients keep on driving home the point that agency needs to deliver better value. This is maybe the most demotivating for the research agency as they don’t even meet the marketing folks.
I am posting a link to an excellent post by Mike Browning & Bert Brower on how to make presentations interesting. http://www.research-live.com/4001301.article The question is how many CMI teams in Asia would let us do these kinds of presentations? Not many that I know of.
Cheers
Sandeep
December 28th, 2009 at 7:20 am
A thoughtful post indeed.
Perhaps the root of this is the question of what is the nature of the market researcher’s job. In a sector where companies compete on the basis of methodology, practitioners become experts in a methodology, not in human behaviour. Moderators know what works in groups, not what works in the real world, Millward Brown know what works in Link tests etc. etc.
It’s in the nature of measurement that a measurement is not the same as the thing being measured. And therefore, I agree with you Nigel, that in many ways it’s pointless to believe that there is a magic bullet methodology out there - every measurement is a one-dimensional reflection of the three-dimensional reality, and the job of the interpreter of research is always to bridge that gap.
However is it the job of the researcher to interpret the research? Should the practitioner just be an expert in his/her chosen methodology or should they, as Stan seems to suggest, be an expert in life, and how it relates to that methodology? Or is that the job of the client who buys the research? After all, that client is the person best placed to understand his/her own business and with the context and perspective to analyse a piece of research against business goals and previous research.
I think clients in large multinational organisations have to look to themselves before casting the stone at researchers. One man in charge of one market often has the time, the power and the perspective to really understand his research. However a bickering committee of 15 local marketing directors discussing a global research project are always going to lower the debate to a discussion around whether action standards have been met, and to reduce complex issues to a single number.
Finally, as has been pointed out, expertise and a larger perspective on research cost money. If clients want insightful (and robust) research then there is a cost attached to that. In a world where the focus group has become the universal panacea due to its low cost and quick turnaround, the actions of most clients say loud and clear that it’s not deep insight, or indeed robustness that count, but simply having research, any research.
Research has grown its share of total marketing budgets over the years, but at what end result? Have we become better at marketing? I’d argue not. Is that the fault of researchers? Tough question, but the analogy I’d draw is whether you can blame arms manufacturers for wars…
December 28th, 2009 at 6:19 pm
Nigel, good post as always. Four thoughts:
What you dismissed as “online and CRM data ‘exhaust’” could be the elephant in the room. If something seems to be “really” wrong with market research, then changes to competitive substitutes may explain more than internal issues with sampling, speed, or lack of insight. As you say, the sampling issue is old-hat. So is the issue with speed. And, even if market researchers do under-deliver on insights into “why,” they easily outperform the “data-exhaust” analysts’ ability to explain why; so insight construction is not a relative weakness of market research. So it’s time to consider external competitive threats to market research.
Here’s one threat. When analytics are applied to “online and CRM data exhaust,” the relative advantages of this research method immediately appear. Convenience sampling is unnecessary and the issue that people have difficulty correctly recalling or describing their behavior almost vanishes. Hence, regardless of how old-hat market research’s worsening sampling issue seems to be to you, it is that very issue that provides fodder to every marketing analyst out there who wants to see budget dollars shift away from market research to census-like databases that they can now analyze with increasing ease and effect. In the absence of ever-improving analytical tools for census-like databases, I imagine that clients would be far more accepting of market research’s sampling issues.
In showing how Ray and Stan’s views (apparently) differ over what is currently wrong with current research, I think it’s worth mentioning that they both seem to be recommending a similar shift: toward market research that generates more insights through more listening, observing, synthesizing. In Ray’s posting that you cited, he indicated that the posting was a preamble to “Why NewMR?”–an approach that seems to center on the triangulation of various types and qualities of sample that you seem to support.
The trick will be getting market researchers to learn enough from the competition, if they try to make listening-driven insight-generation a new focus. If market research can stay open to analytics and to new opportunities for random sampling (e.g., of content), while strengthening its qualitative zeal into grasping the human situation, then I think its focus on insights will pay off.
December 29th, 2009 at 10:14 am
The post and the responses are excellent. I have been pondering this issue for several years now and am currently engaged in training our folks how to analyze. I believe a root cause of the problem (insight paucity) is that researchers find comfort in their craft. That as researchers they are far more comfortable making narrow observations based on findings and contextual comparisons (norms, historical performance, competitive benchmarks), but distinctly uncomfortable in going further to attempt to suggest deeper implications and actions.
My advice is to have an “out of data” experience. Have the courage to go beyond the findings. Integrate the findings with external sources (the online and CRM data “exhaust”) and add the magic ingredient — human experience and intelligence. Of course, far easier said than done in that this moves us from the “science” of research to “art”. Going beyond the mechanics of process to adding depth and insight that takes us to a new way of thinking about the issue. In my opinion, much more creative than what we hire and train for.
December 30th, 2009 at 4:32 pm
Nigel
This is one of the very best posts I’ve read in 2009. So many important points; beautifully considered and articulated.
Katie
December 31st, 2009 at 10:21 am
Great post indeed. The key takeaways for me is that a) clients often get what they need (which is often the quick/dirty answer) but b) (to Phil’s note), the great market researcher should be ready to give “all the rest” if needed.
To end on a humorous note, if my wife asks me to wash the dishes, I can either a) wash the dishes or b) throw out the dirty ones and buy her the finest new china available. I think most of our clients ask us to do the dishes, and while they might be happy with new china, it might not be the best strategy.
December 31st, 2009 at 1:50 pm
Thank you all for your comments. I really do appreciate the feedback and different viewpoints you have presented.
To clarify, I do support Ray’s belief that convergence is an important part of the solution to what ails market research be it in sampling, data sources or analysis. So maybe we need to apply the same logic we use in analyzing one research project to all the research questions we face. You would not rely on just one question to give you the right answer from a survey, so why assume one sample, data source or research methodology will do so?
Anyway, Happy New Year to all of you celebrating tonight. Here’s to a great 2010!
January 3rd, 2010 at 8:31 pm
I think the “convergence” issue is a key one. As an industry (users and agencies), we seem a bit “faddist,” always convinced that the holy grail, the silver-bullet insight system, is to be found in the latest product or methodology. In many cases though, the answers aleady exisit in the masses of under-analysed, unintegrated research information that already exists. In the last few years I have been involved in developing sophisticated Shopper Insights/Brand Health systems that drew on data from Panel, Retail, Survey, and Quali data to provide some pretty terrific and very rich marketing insights. No revolutionary techniques here: the “new” aspects of these were largely a matter of creating theoretical frameworks that made sense, and taking the time to integrate the data sensibly.
What made these interesting was the unxpected degree that something like consumer panel data could offer insights into Quali findings (and vice-versa) once you developed a systmatic approach to comparing them. What this kind of convergent approach to research design/methods does require is MR companies willing to invest in conceptual (as opposed to technique driven) R&D, development of consultants who can span methodologies, and serious commitment to getting clients to buy into research programmes, not just projects. Not easy, but somehow it seems to me that the future of our industry must lie at least as much in getting more out of what we do already as it does in developing ever new techniques. I’ve done a few posts on this issue (”Are Surveys Heading for Extinction?” etc.) at: http://wp.me/pBmtI-2u if you are interested.
January 4th, 2010 at 5:09 pm
Hi Nigel - great post, and some good discussion here in the comments too. I’m glad to see you’re resisting being painted as the defender of status quo. Similarly, I hope I don’t get painted as the guy who hates research. Some of my best friends are researchers…
The interesting thing for me, even in this thread, is that everyone (clients, agencies, researchers) seems to be tempted to point the finger at everyone else, when this is a problem that can only be solved when we ALL work together. None of us alone can increase the value of research. Researchers have a hard time fighting for better research when they’re (rightly) afraid to rock the boat and risk a contract; client CMI groups are often marginalized within their companies and budget and time constrained; and agencies have long lost our credibility on research issues because we’ve selectively used the results (loving research when it supports our POV or our creative, and fighting tooth and nail when it doesn’t). Likeminded people need to reach across disciplines if there is to be any change.
Really looking forward to speaking with you at the conference, and from the sound of it to being in loud agreement with you on many things. And maybe even to push the MR peanut forward a little bit.
All the best for 2010.
January 4th, 2010 at 5:21 pm
[...] This post was Twitted by berniemalinoff [...]
January 4th, 2010 at 5:24 pm
Thanks Alastair and Jason for adding to the debate. Interestingly it seems to me that there are two valid and complementary tracks in your comments. 1) Real synergies can be realized when different data sources are compared on a systematic basis, but, 2) that requires commitment from both the client and their agencies. To which I would add a willingness to find ways to overcome issues regarding proprietary methodologies, different data and terminology data plus good old fashioned distrust.
Jason, the conference session should be fun. It would be great if we can come up with something that does push things forward. Cheers!
January 5th, 2010 at 4:55 am
As an adjunct to the convergence issue, I tend to think that the ‘insight from data’ industry has a problem knowing what data source (or data collection tool/methodology) to select for a given task. CRM data is a great source of data and insight for certain task, but unsuited to answering other requirements. Similarly, the things we think of as ‘market research’ are better suited to some briefs than others. Being truly mode-neutral is very difficult.
Related to this, am I the only one who finds that the real requirements of a piece of work are often hidden somewhere outside the actual brief? It’s a real challenge sometimes to make sure programmes are really addressing the deeper insight needs and not just the ones articulated in the hastily written brief.
More constructively, I’ve been lucky to spend the vast majority of my career working pretty much exclusively with one client at a time. This puts me in the luxurious position of having time to get immersed in client’s business and get to know them individually and as organisations much better than colleagues who are continually juggling several clients (often across quite different product fields).
For all the reasons above, I’m hugely sympathetic to the view that it’s far harder for ‘outsiders’ to recognise the truly insightful than it is for clients on the inside. Isn’t it encumbent on us as providers to make wise judgements about what bits of information clients need to see and explore and discover for themselves to understand the real insights? Assuming we can always unearth the insights ourselves is arrogant on the part of providers and unrealistic on the part of clients: it’s a team game as Jason points out above.
January 5th, 2010 at 9:36 am
Thanks Trevor, your comment makes me think that I should have ended the post by asking what clients are doing to inspire collaboration among their agencies and to explore data in conjunction with them and their marketing teams. If progress is to be made it needs to be a win, win, win, just as it now is in the realm of packaged goods where manufacturer, retailer and consumer need to win in order for a new brand to be successful. In this case the winners would be the client insight team, the research agency and the marketing team.
January 18th, 2010 at 1:06 am
[...] is really wrong with market research? (Nigel Hollis) Nice, balanced overview of the challenges of market research. He writes: “I am afraid that many [...]