AMSRS 2011 Highly Commended Paper by Erica van Lieven- The Caterpillar Becomes the Butterfly
The Caterpillar becomes the Butterfly;<br />The inevitable metamorphosis of Market Research<br /><ul><li>Overview</li></ul>“It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change.” Charles Darwin.<br />Against a background of questioning about the future of the Market Research (MR) industry, this paper seeks to understand how satisfied clients are with market research (in Australia and other developed economies) and how the industry contributes to client business decision making. We use a client focused view on how relevant and fit for purpose market research agencies are against their perceived consulting competitors currently and, understanding this, and having extensively reviewed the current literature on change in business and technology, propose what kinds of changes the industry may be facing.<br />Industry future thinkers and leaders have been quite vocal about their doubts for the industries future. <br />Over 180 responses were gathered during March-April 2011, from clients in Australia, the UK and the USA in a quantitative survey. Once these were understood, additional in-depth conversations were had with a number of senior insights directors in Australia. This paper shares our findings from the research and our thoughts for the future.<br />The results tell us that cost and time challenges are top of mind for clients, and because of increasingly competitive markets, the need for innovation and consumer insights remains critical. Clients use a range of consulting agencies for different reasons; with objectivity a key underlying reason for seeking out an information provider, whatever their speciality. Being a specialist is also important, either in technique/approach or in category or business knowledge.<br />Management consultants appear to be primarily chosen for their business acumen, their ability to understand broader commercial objectives and their ability to challenge clients’ thinking. In comparison, the larger, full-service Market Research agencies offer benefits stemming from their size, breadth and global presence, such as their ability to leverage learnings, provide consistency across markets, and their ability to specialise in multiple areas (both across categories/industry sectors and techniques/approaches) and invest in and make effective use of cutting-edge technologies/sophisticated tools.<br />Buyers are less likely to use smaller, more senior-led Market Research agencies if they want consistency with what they’ve done before and “traditional” and “proven” techniques — and are more likely to use smaller Market Research agencies for their specialist knowledge and ability to tailor solutions.<br />In-between these, the medium-sized, local Market Research agency appears to be able to provide both tailored solutions, interpretation/insights, in-depth customer understanding (not just data) and specialist knowledge, as well as speed/efficiency and cost-effectiveness (over other information providers). <br />When asked how differing consulting agencies are performing, Market Research agencies are performing as well as other consulting suppliers to our clients; no one consulting supplier stands out for performance ahead of Market Research. <br />We conclude that overall, there are currently no obvious shortfalls or standouts for Market Research providers when compared to competing consulting agencies, although we have seen that there are gaps and opportunities to improve our performance going forward. As Darwin noted, it is the one that is the most adaptable to change that survives; the industry has adapted and survived in the past, but cannot afford to stop here. Clients are under pressure to innovate in ever tighter timeframes and with lower budgets; data is becoming a commodity and perceptions of the value of Market Research are claimed to be declining, whilst client needs and expectations are also changing. The business imperative opens the door to innovation; new competing approaches and technologies that help clients answer their business challenges and move their business forward with success and speed. Not all clients are able to enunciate the changes they need, a leading few can see the coming speed, innovation and clarity that is required.<br />It is up to the Market Research industry if and how we want to be part of this future.<br /><ul><li>Introduction </li></ul>“I suspect that the most successful companies will cease to be market research companies, and Market Research itself may cease to exist as a defined profession.” Ray Poyntner <br />UK market research is “inadequate...reactive, and...out of touch.” — Steve Gatt, Volkswagen Group’s Economic and Insight Manager, referenced in an article from Research, August 2010.<br />These are just two viewpoints on market research today (both originating from the UK), but both raise concerns if they reflect broader opinions and the market research industry generally. <br />Add to this the recent findings from the GRIT Report which state:<br />Major structural and systemic changes being faced by those in the marketing research industry, mirror much of what we have been seeing in the US economy as a whole. These structural changes have a multi-pronged effect, including (1) more worrisome attitudes and beliefs about the marketing research profession, (2) more concern about the ability to keep up with the rapid pace of technological innovation, and (3) a growing tension between quality of work output and the demand for speed. Part of the increased pessimism in the marketing research industry is fuelled by “systemic stressors” that make it increasingly difficult to deliver on the shared goal of high quality, highly valued research. Attitudinally, two-thirds feel that research buyers are less able to tell the difference between high quality and mediocre research now. <br />We are living in an unprecedented period of change for the market research industry. Technologies are emerging and boundaries are blurring between traditional one-way market research surveys and co-creative solutions, such as community panels, social media and other “listening” and mining research approaches. This inevitably affects the way we do business as market research agencies.<br />While we focus on asking questions of our survey respondents, and fiddling round the edges of finessing the questions, how often do we stop, and ask questions of ourselves and our clients, or even gather insight from a broader audience in the world of consulting and advisory services to see whether there is a better way of working that will not only improve the value and outcomes for clients, but will help ensure our future success as an industry?<br />The quote from Ray Poynter challenges the way we think about our industry and also the future of market research; will it survive through this period of change and, if so, in what shape or form? Will market research adapt or transform in scope during this period of exceptional technological change?<br />This paper seeks to understand how the quality, effectiveness and value of our work in the market research industry measures up against other consulting businesses and information providers. We then seek to understand in the context of the changing business environment that Market Research Industry exists in, and how the expectations and needs of our clients, themselves in changing business environments will affect or impact industry futures.<br />We were intrigued to understand how satisfied clients are with market research and how the industry contributes to client business decision making. Our focus was to get a client view on how relevant and fit for purpose market research agencies are against their perceived consulting competitors.<br />This paper uses both quantitative (online survey covering a range of businesses that use market research and other professional consulting businesses) and qualitative methods. Views have been gathered not only from Australia, but also the UK and US industries, to provide a broad perspective, as we were mindful that the Australian perception of pressures may lag behind our overseas colleagues. <br />Further, in order to understand the broad pressures on the market research industry, we have reviewed a wide range of literature on its future, technological changes affecting the industry, and the history of the Australian Market Research industry, to bring the views of current industry clients into sharper focus.<br /><ul><li>Tick tock, tick tock... times they are a-changing...
3.1Overview </li></ul>Whilst it feels like we are living in a time of unprecedented change, the history of market research in Australia suggests that each generation evolves through a period of turmoil brought about by new technologies. While technology has been evolving and influencing how we have been conducting market research for the last 80 years in Australia, the most recent technological evolutions and the invention of the internet have accelerated the speed of change. As Ray Kurzweil observes in his analysis of the history of technology:<br />Technological change is exponential, contrary to the common-sense “intuitive linear” view. We won’t experience 100 years of progress in the 21st century — it will be more like 20,000 years of progress (at today’s rate). The “returns,” such as chip speed and cost-effectiveness, also increase exponentially. <br />Not only are we experiencing rapid technological spurts, but underlying this are economic, financial and social changes which impact consumer behaviour and thus by default the market research industry.<br />New technologies give us access to more respondents. This leads to the availability of more data while the emergence of new approaches to capture the data opens doors for new competitors or DIY research methods.<br /><ul><li>3.2Through the Looking Glass: technology is changing fast and becoming “transparent”</li></ul>Technology is allowing people to connect and express their views in ways not possible before. The size of Face book is unprecedented. In January 2011, Australia Face book was reported as having 9.8 million unique Australian visitors going to the site every month1.From early beginnings in the 90’s the internet has reached saturation levels in Australia, with an estimated participation of 80% by 2010. Mobile access, the uptake of Smart Phones facilitating mobile and media convergence, the shortcutting of search to personalised apps on smart phones, all enable connection and communication like never before. <br />Technology is facilitating what Mike Earls has described in his book Herd, as our innate “super social nature” which is a key differentiator in our evolution, allowing us to work and live together in ways not previously possible. Mike Earls suggests that the explosion of online sharing is tapping into a fundamental human need. Should he be correct, the ever-evolving technologies of connection will continue to supply a data tsunami.<br />The platforms that support social networks like Facebook indicate another key aspect that should be considered in driving change in our current business environment. In the past, economic theory has been based on a theory of scarcity. The internet and our ability to store increasingly exponential amounts of data changes all this. Businesses now making headlines in growth and bottom line profit are not necessarily in the traditional model of controlled delivery, and short supply; holding product in a warehouse is no longer the only way to be profitable in the digital era. <br />Chris Anderson has explored this change in his bestselling books The Long Tail and Free: In Free, Anderson suggests Free is proving to be a profitable model for newer businesses and a damaging one for older businesses that remain fixed to older technology, structures and distribution systems, particularly those that prevent consumers being in closer contact with products. <br />Further, technology innovation has allowed the platform to disappear. As Jack Dorsey, the creator of Twitter, said in his interview with Charlie Rose, “My goal is to simplify complexity. I just want to build stuff that really simplifies our base human interaction”.<br />The technology disappears, allowing the human connection to be facilitated; it’s like the glass wall in the qualitative viewing room. With intelligent “questioning” a lot of behaviour can be observed with less and less intrusion into the room with the respondent.<br />3.3The Data Tsunami or a Wonderland of Information?<br />More is being written than ever before, offering the opportunity to observe and capture this information to meet the emerging needs of our clients daily work pressures. As one academic has written, “We’re in the midst of a literacy revolution, the likes of which we haven’t seen since Greek civilization”.<br />The period of change we are now experiencing in the market research industry gains impetus from the new technology that facilitates new ways of connecting with consumers. Connecting with consumers is central to our industry, which was built in Australia in the early 1930s when access to people was a large part of the challenge. Now, with the benefit of 80 years experience, the web has created unparalleled access to respondents. Online panels create ease of access to respondents; asking questions or observing responses is easier and faster than ever before. <br />One of the first consequences of this access to consumers is the mounting volume and sources of data that are and will be “available”, sometimes called the “data tsunami”. <br />The data tsunami has arrived. According to recent research from the MIT Center for Digital Research, digital information is doubling every 1.2 years and will exceed 1000 Exabyte’s in the next year. The size of the largest data warehouses is tripling every two years. There is so much data now related to our “own lives” that it is overwhelming and almost unmanageable. This is especially true for the social customers who are becoming more powerful every day through the new word of mouth marketing platform called the social web. <br />Added to this, we now have emerging approaches for “accessing” and synthesising consumer information, which have the ability to blur the boundaries between market research based on traditional one-way market research survey methods (which the Market Research industry was built upon) and newer “research” approaches facilitated by web 2.0 and increasingly 3.0 technologies, which are based on co-creative solutions such as community panels, social media and “listening” approaches. <br />This brings into question the framework of who are we competing with in market research and what is the essence of market research’s competitive advantage as an industry over other possible sources of insight for our clients as critical issues for industry participants.<br />And finally, not only are the possibilities of the techniques available to us being expanded, access to data/information is exploding, and traditional Market Research methods and rigors are being questioned, bringing about debate about the very scope/jurisdiction of “market research”. This challenges the very core of the way we do business. <br /><ul><li>History and evolution in Australian Market Research
4.1Overview</li></ul>To help us look into the future of market research, we examined how we define the industry now. We then examined the history of the industry, particularly in terms of the impact that technology has had, and the way market research has responded to these changes.<br /><ul><li>4.2What is the Market Research industry?</li></ul>Defining the industry in terms of sales and types of research is relatively easy. As reported by ESOMAR in their Global Market Research Reports, Industry turnover in 2009 was $28.9 billion US, a 4.6% (inflation adjusted) decline year on year. The total turnover figures suggest the beginnings of a shift in spend in research methodology. Currently, Quantitative research makes up 80% of research being commissioned, down from 83% in 2006 .Qualitative Research accounted for 13% in 2009 (down from 14% in 2006).The newer online Qualitative had become 1% of total industry spend in 2009.<br />Online research accounted for 10% or more of the overall research spends in 22 countries. Australia ranks eighth in the world for spend on online research; with 28% of research spend being online.<br />Whilst the top 5 companies globally have consolidated to cover 48% of total industry turnover (compared to 40% in 2005), there remains diversity in the remaining 52% of the industry that is possibly its greatest strength in this period of change.<br />However, defining the industry in terms of the breadth and depth of what it does is far more challenging. From the above Industry figures, it is clear that the majority of the industry is anchored in quantitative questioning, and the industry is dominated by 5 large players, with indication that further consolidation is imminent amongst the 5 major players. Beyond this we know that there is a diversity of agency sizes and approaches, methods and techniques that are used. Within this diversity of approaches will lie the seeds of innovation. Should the technological changes that are apparent now in facilitating access to observing and asking questions, as well as changing business processes continue to impact the industry, it will be either from the diversity that supports innovators in the industry or the early adopters who see an opportunity in another industry and bring it into the market research business that we can expect the changes to begin. <br /><ul><li>4.3An industry history of change</li></ul>The first reference to Australian market research was by Herbert Casson in 1928, when he announced for the first time in Australian literature that every business needed a market investigator to be successful. <br />In Australia, the industry is built upon a proud history of “questioning” which began in the early 30s when the first pioneering researchers (thought to be Berlei in the late 1920s and 30s) collected and analysed the measurements of 6,000 women as they were being fitted for underwear in Australian department stores. The measurements were analysed and calibrated to produce the Berlei Type Indicator which placed all women in one of five size groupings, from type A: Average to type BH: Big Hip!. <br />Since its origin, the industry has required innovative and dedicated people who were prepared to pioneer methods of garnering what the consumer was thinking. In these times (the heyday of the wireless radio, before telephone, fax, TV and web 1.0), in order to find what someone was thinking, panels were built and invested in to facilitate the questions:<br />To establish a permanent home grocery purchase reporting panel and control it, was a very expensive business. At the time the idea was too far ahead of the Australian market. What we had done was to enrol through our normal consumer surveys a large reservoir of persons who were willing to answer questions by post. This was 1965 and we had developed a panel of 3,000 families....With this group we carried out an extremely comprehensive group of studies for H.J Heinz Company which was reviewing its soup marketing in Australia. We carried out so many tests for Heinz in that first 6 months that our panel members sampled between them 5 tons of soups<br />The first computers entered the offices of the far-sighted and dedicated in the early 60s, and were unproven in their ability to contribute to the work. Innovation brought growth through people taking risks, trying new methodologies and honing their questioning skills. The focus was on being able to create a method where questions could be answered:<br />One of the healthiest offspring of this new science (market research) is the Consumer Panel, an efficient means of arriving at continuous accurate assessments as to why certain products succeed and others fail....This continuous flow of “marketing focus” spotlighting directly on the consumer.<br />In Australia, the market research industry is barely 80 years old, and there is a sense that we are again at a crossroads; again we are questioning how the industry will move to the next phase of evolution. Change has been a common theme in the development of the industry. A reading of the industry history is rich with stories of previous challenges brought by technological innovation, as evidenced in the following quotation from Researching Australia:<br />The fifties brought one of the biggest shakeups in media ever that was the advent of the television. That was a horrific thing for media. Everybody thought they were going to be put out of business, nobody knew where to next. Therefore, it threw off a whole lot of new ideas and new requirements.<br />Technological change has historically caused the industry to question its purpose and yet has ultimately facilitated innovation. Whilst the speed and capability of the technological change is increasing, a review of the industry history proves its capability to adapt innovations and evolve the method. So when industry observers suggest,<br /> “Marketing research, as an industry, is faced with having to adapt to environmental changes (mostly technology driven).”,<br />My suggestion, based on industry history is that the industry has a track record of building on these periods of change to take the next steps forward.<br />4.4 Evolving Business models<br />The speed of change in the business models surrounding the industry is not to be ignored. Information businesses like newspapers, books and photography have all had to deal with the shift created by the on-line world, the shift from scarcity to ubiquity. <br />Can market research learn from the new emerging corporate winners? The new winners are not so much the well established companies of old, but the new platforms that seamlessly provide services “free” and fast, whilst accessing the critical mass to provide value added services elsewhere. Examples include:<br />Google versus Newspapers — the business model is changing so fast for traditional newspapers that as they lose vast amounts of traditional advertising revenue to emerging online retailing platforms (Seek, online retailing) they cannot adjust their business model fast enough. <br />Borders versus Amazon — Borders was losing $US 2 million a day with 30% of its stores being unprofitable while Amazon reported sales of $9.51 billion in sales in 2011, 33.4% higher than the previous year. This is whilst Amazon is investing further in cloud computing and a “music locker" that allows people to store files on its servers. Apple and Google are believed to be working on similar projects.<br />Kodak versus Flickr — Kodak was so focused on making money from old technology (photographic paper, chemicals, and retailing this through traditional mechanisms) that it forgot it was in the business of images and memories and completely missed the digital step.<br />The recent announcement that YouTube will be renting out 3,000 movies from major studios represents a move by Google to become the “TV channel of the future”. Skype now owned by Microsoft opens up the possibility of Microsoft becoming the largest communications company in the world.<br />Google is worth studying for what it has achieved in growth and profitability through a focus on quality. Companies like Google, have changed the rules, putting innovation at the centre of what they do to innovate and grow. The ‘Google Way’ challenges all who work with innovation to question if they should consider structural and cultural change in their organisation to implement successful innovation. <br />These new business models tell us one thing: If Market Research ignores changes, to protect old revenues that we may sense are under pressure from new ways of connecting with consumers, the history of companies past (newspapers, books, music, photography, to name a few) would indicate that this is not a recipe for survival.<br />The ‘Free’ view as express by Chris Anderson is not universally agreed with. Malcolm Gladwell in his critique of Anderson’s views as published in The New Yorker, highlighted that in the new on line age, the only rule that is true is that;<br />“The digital age has so transformed the ways in which things are made and sold that there are no iron laws”<br />The key point is that all business are now alert to the transformative nature of the digital age and the Market Research industry would be well advised to be open to the opportunities that evolve in this period of change.<br />5.0Survival of the adaptors<br />“It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change.” Charles Darwin.<br />Change and the need to respond to new technologies is part of the history of Market Research. Historically, it has been the industry’s ability to adapt and evolve that has kept it relevant and fresh. The data sourced and the questions answered have oiled the wheels of profitable innovation for over 80 years. The presence now of a data tsunami evolving from platforms that allow increasing consumer interaction facilitated by technological platforms will mean more innovative opportunities for our industry and its future. But what will these futures look like?<br />All these new technologies work off a key insight — they seek to become invisible. They seek to involve the user in an experience that takes the presence of the platform away. <br />This invisibility is one of the five megatrends shaping the future of market research as observed by Robert Moran in his paper “The Future of Marketing Research in 2021: <br />1. Data abundance<br />2. Asking-observing shift: utilising platform invisibility<br />3. Democratisation<br />4. Convergence — data to insight to foresight to strategy<br />5. Strategic Imperative — evolving in a data abundant world<br />Could transitioning from unsolicited and ‘invasive asking’ to co-creation, to more listening and observing techniques mean we evolve as an industry into less structured asking and ultimately to utilising the vast amount of data generated by social networking to enable data mining of the style described by some industry observers as ‘fishing in the river’ of information, as suggested by Robert Moran in his Insights Future paper:<br />“The need for closeness to the customer will drive innovation, especially in unprompted consumer research. Corporate leaders will develop fluid, searchable knowledge collection capabilities — an insights-on-demand capability that will not require interruptions to initiate individual studies to answer business questions. By 2021, we think that leading-edge companies — probably led by consumer packaged goods and technologically driven enterprises — will look for answers to 80% of their marketing issues by “fishing the river” of information.”<br />“In 2011, 80% of global market research spend is dedicated to quantitative research. By 2021, “the survey will not die, but research will move beyond the survey to social media listening, observational and co-creative MROCs (Market Research Online Communities) tools”. <br />From our reading both of industry history and current discussions about business models and futures of the industry we would summarise the past and possible future direction in the following table.<br />Table 1: Summary of trends and potential futures:<br />HistoricallyPresentFutureRole, PurposeOne-way data collection; asking questions of target audience and gathering information (‘objective reporting’ – objective truth/single reality)Two-way data collection and integration; co-creation, listening, looking for answers and seeking insight; building ‘narratives & implications’Information synthesis and mining; moving from insight to foresight to strategies **Methods‘Methodological rigor’; robust, representative sampling, statistical validity ‘Methodologically curious’ - alternative view of validity, e.g.; crowd sourcing ‘Methodologically agnostic’ – ApproachNew, disruptive and often unsolicitedAccepted, mixed approaches and rise of online panels, online communities “Invisible”; new ways of listening, observing in social media (without interference); tapping into the sub-conscious and emotions, eg; neuroscience, biometrics, eye-tracking<br />Table 1: Summary of trends and potential futures (cont):<br />HistoricallyPresentFutureDataDiscrete data sourcing – ad hoc research creating “data silos” Multiple data ‘streams’ or sources, e.g.; ad hoc research plus social media “listening” starting to be combined (by ‘data warehouses’)The “river” of data - “a continuous and organic flow of knowledge” with “1,000′s of tributaries” or ‘data oceans’Relationship with research buyersService, supplier oriented; ad hoc data providersBuilding closer, longer-term relationships that aid delivery of insight & recommendations, not just dataMutually beneficial partnerships, integrating with clients or becoming an extension of client side MR department to enable commercially relevant foresight & strategiesTechnology available1920s-30s: panels begin, paper-based postal surveys;1950s-60s: first generation computers, arrival of TVs (media shake-up)1960 – CATI1990 – online, WEB/internetMobile phonesWeb/InternetFacebook and other social mediaSmart PhonesGPSCommunity panel platformsNeuroscience, biometric & eye-tracking toolsGamification“technology agnostic’ will allow fluidity of adaptation as technology will move rapidlyTechnologically enabled “disappearing platforms”<br />5.1 Becoming Invisible<br />The essence of our industry has been built on asking questions and creating platforms that have allowed questioning to happen in a controlled manner, as cost effectively and speedily as possible. It took pioneering spirit to find ways of connecting 80 years ago, and the key skill was “asking questions”. However, considering the access new platforms give to vast amounts of original data from consumers, where structured questions are replaced by finding answers, a new future may be evolving — a future where we need to harness the platforms to give the best possible answers, rather than planning and asking questions in a controlled process. Asking questions makes Market Research very visible in the process, whereas getting answers on platforms that disappear into the background provides new opportunities for getting answers.<br />Just as our clients are realising they don’t own their brands, their consumers do, Market Research may need to understand that no one ever really wanted to answer our specific questions. Realising this, new technology creates the opportunity of doing research without actually asking questions. Ray Poynter made this point in his blog in April of this year;<br />My starting position is that nobody ‘wants’ to do market research. People want to have information about markets that will help them make better decisions (in the same way that nobody wants drills, they want holes in walls). Market research is what people do when there isn’t a better, cheaper, faster alternative. Similarly, within market research, surveys should be what we do when there isn’t a better, cheaper, faster alternative.<br />If we were to go back twenty years, then the number of occasions where a representative, quantitative survey was the best answer to a problem was large. However, over the last twenty years, and particularly over the last ten, the world has moved on. For example, online access panels are not representative but they are cheaper and faster and have been deemed good enough. More recently, listening research has begun to tap into customer conversations, DIY options are beginning to challenge supplier models, and online research communities (MROCs) are providing qualitative answers that often outperform quantitative (e.g. survey-based) methods. <br />Joan Lewis, Procter and Gamble’s global consumer and market knowledge officer, was quote by the Advertising Age in an article summarising views from the US Advertising Research Foundation Think 2011 Conference in New York, <br />“The industry should get away from believing a method, particularly survey research, will be the solution to anything...we need to be methodology agnostic”. <br />She then said in a further interview after the panel, <br />“Social media listening isn’t only replacing some survey research but also making it harder to do by changing consumer behaviour and expectations”. <br />These comments focus our thinking on futures for Market Research. Industry thinkers have caused some discussion about the industries future; however, we wanted to frame the future from the view of our clients. Our focus for learning is primarily from our clients as they are the ones to best judge what they need but cannot get, their need is highly likely to drive our innovation cycle, which will be facilitated by the technology. So how is Market Research industry performing what is the perception of our clients? Is it fit for purpose, helping them in these times of change and meeting their changing demands?<br />How will client pressures drive the shape of things to come for Market Research industry? <br />6.Our research - Quantitative<br />6.1Purpose and approach – what we did<br />A short online quantitative survey was built with the objectives of understanding:<br /><ul><li>What are the biggest challenges clients are currently experiencing in their business in the innovation and marketing areas in the next 3 years, both spontaneous and then prompted?
How do various consultants, including Market Research perform in helping Clients make successful commercial decisions, given the rapidly changing environment we are working in?</li></ul>We targeted opinion-leaders and decision-makers in innovation and marketing — both locally and overseas, and across a range of industry sectors.<br />We asked questions about eight different consultancy agency types that service clients. The agencies chosen were deliberately broad and chosen to give context to how Market Research was performing against other consulting suppliers our clients would typically use.<br />Agency types included were:<br />Table 2: Agency type included in quantitative survey approach<br />Agency type included in quant survey phaseAdvertising agenciesSmall senior led consultative research agenciesMedium sized, local market research agencies offering full service across multiple sectorsLarger, global market research agencies offering full service across multiple sectorsMedia agenciesManagement consultantsSales or trend data specialists (e.g. Neilson, Atzec, Mintel, Euromonitor)Specialist new media/software companies( e.g. community panel providers)<br />We planned to keep the analytical approach quite simple as we were unsure from the outset as to the number of completed questionnaires we would achieve, and did not want to compromise any analysis undertaken with low numbers of responses.<br />Given the complexity of the subject, and the strong likelihood that our survey audience would be extremely time-poor, our goal was to gather broad quantitative inputs via the online survey - which we endeavoured to keep as simple and short as possible, while also aiming to make it relevant and engaging to the audience. We then followed up in more detail with 1-1 interviews. The discussion board was initially selected as the method most likely to fit our respondents’ preferences, allowing privacy and ability to comment in a time poor environment; however we found this approach did not appeal as we had very few clients take up the option. We therefore searched for qualitative insights/feedback elsewhere. This resulted in reviewing comments from looking to social media and web-based resources, as well as a number of in-depth interviews with senior Insights Directors in Australia and overseas. In-depth interviews were conducted either face to face, or in the instance of overseas input, via a series of written questions.<br />Initially, a charity donation and access to the findings were offered as incentives to complete the survey, but with low response rates, this was modified to offer a more traditional and financially-based reward.<br />5.2Who we interviewed<br />Through the online survey, we succeeded in reaching n=189 buyers and decision-makers in innovation and marketing from Australia, UK, USA, covering a broad array of industry sectors, including FMCG, Government, Financial, Utility/Services, Retail, Telecoms, IT, Pharmaceutical/Healthcare, Manufacturing and more. See table below of sectors included. In-depth interviews were with Insight Directors in FMCG organisations, one in-depth interview was with a major European FMCG business.<br />5.3Research Findings<br />To set the context and better understand the business environment that Market Research and other consulting businesses are currently operating in, we first asked the research buyers about their anticipated and current critical business challenges.<br />5.3.1Cost-related Challenges Top of Mind<br />Spontaneously, by far the most top of mind business challenge buyers are expecting to face in the next 3 years related to budget and costs (29% of comments overall), indicating, not surprisingly, that industry is feeling the pressures of needing to address their business issues “against a shifting backdrop of budgetary constraints, doing more with less, sample and data quality issues, declining response rates and the drag of the larger economy as a whole” — reference GRIT Report.<br /> ‘Cost’ challenges (in our survey) took many forms (across the diverse range of industries we spoke to and US, UK and local markets), including:<br /><ul><li>tighter margins and maintaining profits
budget restrictions or getting budgets agreed on.</li></ul>Not surprisingly perhaps, cost was front and centre across the board, in Australia as well as UK and US and for small, medium and large-size businesses in all sectors.<br />Additional pressures are being felt amongst the FMCG industry in Australia on margins and profits given that two grocery retailers dominate the market, as highlighted in a subsequent qualitative interview):<br />“Our biggest challenge is the pressure we’re getting from retailers due to their pricing strategies which eats into our profits” – FMCG, Australia<br />Innovation and Competition go hand in hand<br />Secondary challenges anticipated, based on the spontaneous responses, related broadly to innovation/NPD and competition (15% and 14% of comments respectively). Regarding innovation/NPD, research buyers indicated that they are facing various challenges across the NPD funnel, ranging from being creative and generating new ideas, as well as identifying the best new ideas and most marketable products (for the target audience), to doing innovation or NPD with less budget and less time, and getting new ideas to market faster, as highlighted by the following comments:<br />“Building true long term innovation pipeline (marketing and R&D/Supply chain capability)” – FMCG<br />“Innovation usually means more expensive products/processes at a time when commodity costs are rising and businesses need to find ways of making products cheaper” – FMCG<br />“We need to really step-change the way we approach innovation. We need bigger ideas and need to find ways to move quicker through the funnel” – FMCG<br />“Commercialisation of ideas into top selling products” – FMCG<br />“Creating innovations that are aligned with customer needs” - Telecoms <br />“Finding the time to devote to innovation and research” – Equipment business<br />In some cases, engaging staff in the innovation process internally was also an issue.<br />“Working with existing resources, both staff and funding to progress innovative ideas, encourage and progress staff towards new uncharted ground whilst maintaining business as usual and delivering the same level of service to our customers” – State Government<br />“Getting staff to become involved in the innovation process” - Government<br />While many of the quotes above are taken from FMCG businesses, “innovation” tended to be a common theme across industry sectors and markets (in the spontaneous comments), but was significantly more mentioned by medium-larger businesses compared to smaller ones (1-20 staff) – and this is reflected in the quantified results.<br />Why would this be such a focus?<br />To sum up, as one FMCG respondent pointed out: “without innovation we won’t go anywhere”.<br />Connected with this, and a key driver of the need for innovation (or product improvement), many research buyers (from all industry sectors and markets) are also telling us they are facing increasing competition, saturated markets and the need to better understand how to differentiate themselves and stand out from the clutter.<br />This competition is coming from various sources — from overseas (cheap imports from China mentioned several times – mainly by Australian-based companies), from Private Label products, and the rise of e-commerce, and also sometimes referred to competition between businesses/in business. <br /> “The increasing level of competition in the market place” – Financial/Banking<br />“Distinguishing your brand / product / offering from those of your competitors” – Financial/Banking<br /> “Increased competition from small niche players and in China the increased presence of local players” - FMCG<br />“...the ever increasing pressure from private label” – FMCG<br />“Competitive bidding required to win contracts leaves little or nil for marketing” – Communications business<br />“Loss of exclusivity” – Consulting business<br />“Large competitors supported by the Federal Government” – Financial/Banking<br />The issue of Private Label (in Australia, FMCG sector) is further highlighted in an article in the Sydney Morning Herald, in an interview with the Unilever’s new Chief Marketing Officer, Peter Boone:<br />“...Boone believes that it is innovation that will be his greatest defence against the seemingly unstoppable march of supermarket private labels”<br />And of course, the market saturation and increased competition, including from the recent upsurge in private label products in Australia, is further impacting on companies’ profitability.<br />Overall, what we can see from these top mentioned challenges is that they are intrinsically interrelated, with the market and increasing competition impacting companies’ success and profit, thus driving the need for innovation. Innovation can also help counteract the potential downward spiral of margin erosion and commoditisation and therefore curb competition and increase profits long-term.<br />Connecting with Consumers remains a focus for business<br />Undoubtedly, underlying the challenges mentioned above is the need to understand the consumer. Identifying, reaching and connecting meaningfully with the customer (or client) were also recurrent themes mentioned by our audience, looking to better understand customer needs and expectations with a view to enabling innovation, counteracting competition and providing better, more targeted offerings to ultimately help retain or grow their customer base and improve profits or revenues.<br />Spontaneous mentions were somewhat higher in Australia and amongst Government, Manufacturing, Public Services, Utility, Construction and Residential type businesses, but do not appear to differ by size of business – large or small, all need to understand their customer. The variety of issues relating to consumer understanding is highlighted by a few comments below:<br />“Connecting meaningfully with clients and maintaining client value” - na<br />“Keeping informed of changing customer demographics, having accurate information on customer expectations” - na<br />“Retaining customers” - na<br />“Reduction of customer base” - Utility<br />“Being able to reach, connect and meet the needs of our target audience” – Construction<br />“Identifying untapped consumer needs” – FMCG<br />“How to remain relevant to consumers who now have more choice than ever before” – Banking/Finance<br />The Social Media conundrum<br />The next most anticipated challenges related to social media and customers (12% comments). One of the key challenges of social media for clients appears to be how to harness this resource and ensure sufficient return on investment. This suggests to us that there is still a certain level of caution in the market, with buyers needing some convincing yet or further evidence of the value of social media as a selection of comments reveals:<br />“How to make the most of social networking” – Government/Public Services<br />“Getting value for money out of investment in social media. Not getting hijacked by social media” - Government/Public Services<br />“Working out how to use social media to our advantage and minimise the negative impact that it can have” - Healthcare<br />“Proving an ROI on social media” - IT<br />““Adapting social media as an effective tool (using it to our advantage)” - Healthcare<br />In one in-depth interview (FMCG), the respondent noted that their business had used social media recently, but felt that it wasn’t giving them any new information over and above their current brand tracking data. <br />“We’ve done an online community recently. Lots of work and things to consider though, like keeping questions relevant, concepts coming and a lot to keep people engaged. It’s a debate whether it is worth that extra work.” FMCG: In Depth Interview (IDI)<br />Clearly some clients are still questioning how to effectively utilise Social Media and how to weigh up the value of what it can bring in comparison to perhaps more ‘tried and trusted’ methods. <br />Spontaneously, comments about social media were significantly higher amongst Australian respondents than US/UK clients, and were more likely to be mentioned by Government, Manufacturing, Public Services, Utility, Construction and Residential businesses. This suggests either their application in the Australian market research environment is still developing and/or clients are still struggling with the fit to purpose, cost and resourcing requirements. Typically Australian Market Research budgets are tighter than their overseas counterparts, possibly making Australian clients slightly more risk averse when trialling new and emerging methods.<br />We would suggest that it is up to us, indeed our obligation in the Market Research industry, to objectively investigate and challenge the value of social media (or any other new information source, technique or tool), but believe that we should also keep an open mind, helping clients understand the opportunities and benefits of these tools in terms of how they can help to solve their business challenges and move their business forward (at the appropriate time).<br />Other challenges<br />Other challenges mentioned by 10% of survey respondents were technology and change related, often in terms of keeping pace with or adapting to technology and change in general, or, like social media, maximising the use of technology and proving its ROI. Likely related to customer understanding, various ‘marketing’ issues were also cited as business challenges including raising consumer awareness and attracting new customers.<br />Adapting to change was mentioned more often by Australian-based businesses, while technology and marketing challenges tended to be UK/US driven.<br />From this we can probably infer that the rapid pace of change, whether driven by new technologies or social, economic and political shifts is affecting most businesses. <br />Less frequently mentioned business challenges in our survey included: staff recruitment/training and motivation, internal issues, such as management team, or software/systems upgrades and costs – which tended to be more internally focused issues – or external issues in relation to the current economic climate (e.g.; global recession) or political/governmental policies (e.g.; carbon tax).<br />5.3.2Most pressing business challenges currently (prompted)<br />When we asked about the current challenges, with a prompted list, we found that the top, key challenges didn’t change much: consumer understanding, cost, competition and innovation, although the apparent priority or ranking shifted slightly.<br />Understanding your consumer remains critical<br />While the cost of business was top of mind in the spontaneous mentions, the most often selected prompted business challenges that buyers are facing currently related to better understanding and connecting with the customer (across the industries we talked to) – see Figure 1. Clients are particularly keen to understand how to better connect with their consumers now as well as being able to predict their future needs. The changing consumer media landscape makes this harder, as one client interviewed said:<br />“The changing media landscape and the fragmenting and changing media habits that we all have. Making sure we are still engaging with our consumers. And utilising the different media types in the most effective way. Making sure we do get our return on the media dollar” “This is a newer challenge that’s happened in the last 2 years” FMCG IDI<br />This quote is reflected in the predictions of the authors from the Engagement and Talent Committee of the Research Transformation Initiative:<br />95% of ‘standout’ leaders believe that getting closer to the customer is a top business strategic initiative in the next 5 years.<br />Figure 1: Prompted business challenges<br />Every business faces financial challenges<br />Concerns about costs (specifically reducing operating costs/avoiding margin erosion), while not the top-most selected across the various industries included, do remain high on the prompted list of current challenges and are unlikely to go away anytime soon. <br />We would expect that the challenge of reduced operating costs is one likely to be passed down to Market Research and other information/consulting providers, although it was noted in the GRIT report that most businesses expect to have equal or higher spend on research in the coming year, although the spend might well shift. Either way, being aware of these business pressures is surely something that the industry needs to be mindful of, as innovation in methods that provides cost effective solutions will have increased value perceptions.<br />Client need for innovation: increasing competition and saturated markets<br />Innovation, as noted in the spontaneous comments, is also a key challenge for our audience, most particularly in terms of identifying and creating genuinely innovative ideas or products in markets which are often saturated already, and with the increasing cost and time pressures as a backdrop. While most of the themes were relatively consistent across AUS/NZ and UK/US markets and industry sectors, more substantial differences were driven by size of business. It was the medium to larger organisations that were more likely to rate innovation as a key challenge, but especially in terms of increasing openness and supporting risk-taking, compared to small businesses (with 20 or fewer staff).<br />This may reflect the more established processes that larger businesses need to run effective innovation processes. Whilst larger businesses tend to have more established or entrenched processes and procedures, they can therefore be more conservative in their adoption of new processes, making it more difficult to move away from long-term established and ‘safe’ methods or culture to one of risk-taking. This does mean that clients can be more risk averse to move to newer approaches, unless the commercial imperative is very strong. This is reflected with what happened with the move by clients to on-line research in Australia when the speed and cost saving far outweighed the change in field methodology.<br />Innovation acts as both as a driver of the need to get closer to the customer, to better understand the direction and demand for innovation, it is also be driven by it, i.e.; the need for consumer insights that lead to a competitive advantage drive market innovation in terms of tools and techniques to deliver better insight, as was highlighted in our Qualitative interviews:<br />“Operating in a saturated market, it’s difficult to do well.... without innovation we won’t go anywhere” FMCG IDI<br />“In the FMCG world where we have big brands that have got a long history we also have to ensure we continually evolve those and renovate them so that we get continual growth.” “We’re also looking for the innovation that will give us a step change in the future so these are our biggest challenges.” FMCG IDI<br />The need for competitive differentiation was consistent across business sectors, markets and size of business:<br />Thinking about the traditional role of the Market Research industry, it is reassuring to hear that businesses continue to rate the need for consumer understanding/customer insight so highly, given this underpins the ‘raison d’être’ of market and consumer research. It goes some way to supporting the current relevance of market research, however we do not believe the current status can be taken to assure our future in the role of insight generation when data becomes a commodity.<br />Less often mentioned challenges<br />Leveraging and improving sharing capabilities of existing knowledge/information is less frequently perceived as a key challenge by our audience overall, although medium-larger business are finding ‘internal knowledge sharing’ a tougher issue to deal with than small businesses with fewer than 20 staff.<br />Potentially smaller companies are already doing this relatively well, the small size facilitating information sharing, while size may be a barrier for the larger organisations. However, the fact that sharing and leveraging information more effectively does not appear to be a very strong concern or challenge in general may also be suggesting that there is perceived to be less value to be extracted from existing information. Could this bode an end to large databases of historical data? Is the shelf-life of data/information (in its raw form) getting shorter and shorter as people look to the future to predict the next trend that could impact how they need to do business?<br />Or, does it simply mean that many companies are as yet not registering the shift from data scarcity to data abundance, and are not yet concerned about how they will cope with the data tsunami when it hits? Looking to the future, we believe that, while ‘understanding’ the consumer will (continue to) be vital for consumer-driven businesses, actually unearthing commercially useful and actionable insights from the mountains of data (from different sources and in different formats) that will be available, will in itself be an enormous challenge – both for client-side and agency companies.<br />In contrast, it is the small businesses which are finding ‘adapting their business to the online age’ tougher, potentially because their business has relied on face-to-face contact and relationships for their success in the past, as this quote shows:<br />“Maintaining a personal relationship with customers as I move more & more to internet selling and contact - from a business I had built locally based on closeness and availability” – Small retail business (1-20 staff)<br />Least often mentioned challenges from our list were:<br /><ul><li>making sense of what consumers are saying in real time
embracing a faster, real-time way of decision-making</li></ul>However, we suspect, both from our reading and in depth discussions with clients, that speed and the need to have research turned around quickly and insights delivered promptly (often without the need for a full report/lengthy debrief) is certainly an emerging issue and one that is likely to become more pertinent in the coming years.<br />The GRIT research noted that “ever-shorter timelines are preventing the delivery of quality” and “nearly 60% (of surveyed respondents) agreed that the ‘quality of work is becoming less important than the speed of deliverables’”. Furthermore, these were expected to become more not less important in future<br />In-depth discussion for this paper confirmed this view that client’s need insight with speed:<br />“We need research that is as dynamic as our business, which is where the gap is at the moment. The traditional research has not moved to be as dynamic as the business” No one has 12 weeks spare in their launch plan to do research, we need much faster ways than the traditional techniques, plus we want the really good impactful technique where the researcher can come in the next day to download on us, as we sometimes just need to know were heading in the right direction...and that sometimes doesn’t need a big report” FMCG IDI<br />Comparison and Summary<br />We felt it was relevant to look back at this point and reviewed a paper written by Susanna Roth for the Conference ‘Adaptation 2004’ in which she conducted in-depth interviews to understand the changing role of Market Research in major Australian corporations – looking to the past, present and future. <br />It is interesting to note that many of the business issues being faced today are similar to those uncovered by Roth in her research in 2004, namely:<br /><ul><li>Competitive differentiation
Counteracting emerging threats to the continued relevance of core products and services</li></ul>However, 7 years on, we have also seen some new issues and ‘cost’ has become a central concern with the intervening global economic crisis not yet forgotten. While Roth’s interviews with senior management revealed that ‘identifying new communications opportunities outside traditional channels’ and ‘understanding the.... brand disloyal youth segment’ were key issues in 2004, our respondents are now rating understanding and connecting with their customers as the number 1 business challenge today, and are actively looking to harness the relatively new phenomena of social media as a tool to help provide better value information or insights.<br />Thinking about the traditional role of the Market Research industry, it is perhaps comforting to see that businesses continue to rate the need for consumer understanding/customer insight so highly, given this underpins the ‘raison d’être’ of market and consumer research. And it does go some way to reassuring us that market research is still a relevant industry and has a current role to play – not only therefore in delivering consumer insight but in relation to helping businesses with innovation, competitive differentiation and margin erosion – all key challenges. <br />The fact that we are able to help clients meet their key challenges would likely have assured our future historically, but in 2011, Market Research is no longer the sole vehicle for delivering consumer understanding. With new competitors on the block, we believe that the industry cannot afford to be tackling this challenge for clients simply through our established research practices and methodologies. We need to be ready to adapt and embrace alternative approaches for gleaning consumer understanding and providing clients with commercially-relevant insights that help them innovate ‘better, faster and cheaper’.<br />Looking forward, there may also be opportunities for Market Research to work with clients to tackle the currently less important, but likely emerging challenges, such as leveraging existing information in better way or making sense of what consumers are saying in real-time. These are areas in which the Market Research industry ought to have a role to play, but perhaps is not currently responding to fast enough.<br />“... we need research that is as dynamic as our business, which is where the gap is at the moment. The traditional research has not moved to be as dynamic as the business”. FMCG IDI<br />5.3.3Which information providers are being used to help with important decision-making in innovation and marketing, and what factors are driving their selection?<br />The next section of our Quantitative survey looked to identify which information providers are being used most often to help businesses with these challenges, and why.<br />Overall, our research indicates that buyers are using various information providers and consulting agencies for different reasons, each appearing to have different strengths and weaknesses. <br />The apparently good news for the Market Research industry is that we are already not a single entity but a “broad church”, with small, medium and large-sized businesses offering different strengths and benefits and being valued for our different areas of expertise. <br />Comments from our qualitative interviews also go some way to confirming this:<br />“For the research industry, I think it does suit our needs, there are different agencies out there for different purposes” – FMCG IDI<br />Cost vs. Specialism<br />While we saw that costs and budgets are clearly top of mind for our audience, ‘cost-effectiveness’ in itself does not appear to be the key/strongest motivation for choosing one provider over another, although it ranks highly for Sales & Trend Specialists and New Media/Software companies equally.<br />There appears to be both good and bad news attached to this finding. The good news is that, while businesses are suffering cost-pressures, Market Research and other information providers are being used and are not being judged purely on cost –.<br />“We don’t go for the cheapest either, it’s the individuals...” – FMCG IDI<br />The bad news for traditional Market Research agencies is firstly that New Media companies and Sales/Trend Specialists are being selected for being ‘cost-effective/providing great value’ more so, creating a potential future threat.<br />The potential risks of the new data abundance is that data or information do become de-valued because the information is virtually free and widely available (such as we are now seeing happen with traditional newspapers and availability of online news content), which prompts research buyers to expect lower costs from research or information providers. <br />“On the one hand information wants to be expensive, because it's so valuable. The right information in the right place just changes your life. On the other hand, information wants to be free, because the cost of getting it out is getting lower and lower all the time. So you have these two fighting against each other” <br />This phrase was first captured in 1984 and describes the tension that is still playing out in the digital age. Information that used to be expensive because it was hard to get is getting easier and easier to access. Two years later Stewart Brand also added these comments to the phrase:<br />“If you cling blindly to the expensive part of the paradox, you miss all the action going on in the free part”<br />This is central to what the Market Research industry needs to address now.<br />One way to counteract this might be to shift the focus from data/information to commercially relevant insight, or even foresight and strategising, as have been suggested by industry observers and bloggers such as Lenny Murphy.<br />Table SEQ Figure * ARABIC 3: Main reasons choose to work with one information provider over another (base n=184)<br />Figure 2: Main reasons choose to work with one information provider over another<br />Market Research vs. Other Information Providers/Consulting offers<br />Part of our aim in this research was to understand how the core Market Research industry shapes up against other information providers/consulting agencies. Where are we performing well and what might the threats or opportunities be for us? Several industry commentators and clients have stated Market Research needs to be more like Consultants, but what does this mean?<br />Looking at each of the provider types (Table 3 and Figure 2), we can see that while Management Consultants offer a number of strengths, they appear to be primarily chosen over other information providers/consulting agencies based on their ‘business acumen’, their knowledge and ability to understand the broader commercial objectives, and ability to challenge buyers/clients’ thinking. <br />Advertising agencies, on the other hand, are less likely to be used for their commercial know-how, and more for their specialism, ability to provide tailored solutions and their objectivity.<br />Sales & Trend Specialists are valued for providing objective, cost-effective information and insights (interpretation, understanding – not just data) as well as benefitting from being specialists in their area, but are not seen as being strongly proactive, challenging buyers’ thinking or helping to provide clear prioritisation of buyers’ business issues.<br />New Media agencies, such as online community panel providers, are seen to be providing great value for money as well as being technology/approach specialists, but are less likely to be used in the capacity of management consultants providing commercial expertise/business acumen.<br />We included three types of ‘traditional’ Market Research agencies in our survey, with the expectation that responses would differ for each – and that Market Research is not a single entity. While these three types may not truly represent the differentiation in the market, we felt it was more important to understand how Market Research businesses compare with other similar information providers/consultancies, not just amongst our most immediate competitors. We also had to be mindful of the number of agency types we could cover in the survey.<br />As we suspected, the “traditional” Market Research industry is multi-dimensional, likely driven by its need to adapt historically, in step with technological and market changes and this appears to be to our advantage now. We can see from our research that Market Research agencies are being used for a variety of reasons, and appear to have a number of strengths, depending on their size and nature. The larger, full-service Market Research agencies offering benefits stemming from their size, breadth and global presence, such as their ability to leverage learnings or provide consistency across markets, and also in terms of their ability to specialise in multiple areas (both across categories/industry sectors and techniques/approaches) and invest in and make effective use of cutting-edge technologies/sophisticated tools.<br />Buyers are less likely to use smaller, more senior-led Market Research agencies if they want consistency with what they’ve done before and “traditional” and “proven” techniques, they are more likely to use them for their specialist knowledge and ability to tailor solutions.<br />In-between these, the medium-sized, local Market Research agency appears to be able to provide both tailored solutions, interpretation/insights, in-depth customer understanding (not just data) and specialist knowledge, as well as speed/efficiency and cost-effectiveness (over other information providers).<br />So, can we grin like Cheshire cats, give ourselves a big pat on the back and carry-on as we are? Or would this just be Market Research in Wonderland?<br />While our findings show that Market Research has a number of strengths, history, method and great diversity, there are nonetheless gaps meaning possible threats, or opportunities, depending on how you want to see them. We believe therefore that the industry will need to continue to adapt to external and internal challenges, and believe that the need for transformation may never have been greater than now. <br />Value Perceptions<br />As noted earlier, New Media companies are being used to provide great value/cost-effective solutions and more so than traditional Market Research companies. This could represent a threat, given the current climate and aforementioned financial business pressures, as well as a potential shift in clients’ mindset with less focus on quality and perception of value from research, as noted in the GRIT report.<br />Perhaps the most disconcerting is the erosion in self-perceived respect for research...with a 20 ppt jump in the percent who feel that research is less valued than 5 years ago.<br />Market Research may need to work harder to increase value perceptions in future (e.g. through dealing in knowledge and insights not data) or find alternative, more cost-effective solutions or approaches; the risk of not doing so could mean more companies using other, newer types of information providers or more DIY research, already common in some businesses:<br />“We do very little ad hoc Quant. Maybe because it’s more expensive and we rarely need it” – FMCG IDI<br />Commercial Focus<br />A further gap, and a potential opportunity to tap into, relates to commercial focus. Our clients are choosing to work with Management Consultants when they need business acumen (as well as specialist knowledge of their category/business) —along with “providing clarity and prioritisation of business issues” and being “proactive in offering advice/suggestions on business”. <br />This finding supports an industry need to include the broader business objectives in our work, have a commercial perspective and not just a project or process perspective, and we need to be more proactive in suggesting ‘solutions’ and ways to help them move their businesses forward, rather than sitting back waiting for clients to come to us with brief. <br />“I believe the market research industry needs to evolve from providing methodologies to providing holistic answers to business questions, how to get there, is of course the specialist area of agencies, but they also need to improve on business understanding” FMCG IDI<br />To quote from Diane Shelton, General Manager of Customer Knowledge and Insight at Coles, in response to a question in Research News February 2010 – If you could change one thing about the research industry, what would it be?<br /> “The overall lack of understanding of business and commercial realities”<br />Techniques, Techniques, Techniques<br />‘Use of traditional or proven techniques’ and ‘use of novel/innovative approaches that help provide competitive edge’ – both factors reference techniques or approaches, although may be perceived as opposites in terms of currency.<br />Interestingly, none of the three Market Research agency types represented here appear to be valued for either of these factors particularly highly. Media agencies were more likely to be chosen for innovative approaches, but using traditional techniques did not appear to be highly valued or a reason for choosing any agency type. This intuitively feels true – and we tend to believe that clients prefer new techniques and innovation to traditional ones, particularly given the current rapidly changing consumer and technological landscapes. <br />However, from our Qualitative research, we know that novel/innovative approaches or new techniques are only relevant if a) they help clients do things better, faster or cheaper, and thus help them achieve a competitive edge, for example, or b) meet the research or business objectives. At other times, it may be the traditional or proven technique that better meets the objectives or is more fit for purpose:<br />“In regards to new methodologies, I sometimes think that agencies propose new methodologies because they are new and not because they are actually right for the objective” – FMCG IDI<br />5.3.4How are these information providers performing on these factors (and overall)?<br />A word of caution. Our survey was structured to only include perceptions of agency performance amongst the more regular users of these agency/provider types (to help ensure results were relevant and representative of users, but also to reduce survey length). <br />The overall picture is generally very positive, but this may in part be due to the fact that buyers are simply working with suppliers that they are happy with (and not the ones they are less happy with) rather than representing a sweeping endorsement for each of these businesses from our audience. <br />The other possible conclusion is that there may be many times when agencies do not perform well, or there is a diversity of performance within each agency type. Certainly, when we look at the Overall Performance (this was measured amongst the total audience, with about ¼ saying ‘don’t know’ for each agency type,) we see a big difference in perceived performance (see Figure 5). <br />Overall ‘typical’ performance for each of these agency types appears spread and somewhat average on the whole, rather than ‘exceptional’. There is no one agency type that stands out as being exceptional or scoring significantly higher than all others in terms of being able to ‘help clients make successful commercial decisions in innovation and marketing’. On the other hand, we might also conclude that Market Research businesses are performing just as well, if not marginally better than other types of information providers/consultancies.<br />With this caveat in mind, we have looked at the overall performance of each agency type and performance against specific attributes, and have conducted Factor and Regression analyses to help determine which factors are driving industry perceptions of these agencies. (Note – analysis conducted excluding small research businesses). <br />Table 4: Overall Performance (amongst total sample): How do these agencies perform overall in helping you make successful commercial decisions in innovation and marketing?<br />Table 5: Perceived Performance across factors (amongst regular users of each agency type)<br />Overall, Market Research businesses are currently performing relatively well against other consulting agency/information providers, and across most of the areas measured, looking at small, medium and larger, global offerings, which provides further evidence that the industry is still relevant and in a good position to thrive to tell another tale or two.<br />However, looking at the performance of other businesses on specific factors, and taking into consideration the overall performance scores, there is also no room for complacency; a number of threats and opportunities are apparent.<br />Management Consultants provide business focused insight and value<br />While the overall top 2 box performance of Management Consultants is slightly lower than that of small and medium-sized Market Research businesses, contradicting to some degree our expectations, there are a number of areas in which they clearly perform very strongly, including ‘not being afraid to challenge you’ and ‘having knowledge of/being specialists in your category/business’, these are areas in which Market Research agencies are not performing as strongly.<br />Our (multivariate) analysis suggests that the value of using Management Consultants is driven primarily by their ability to create strong business relationships that allow them to be proactive in offering advice, challenging clients and offering tailored solutions, their objectivity and ability to focus on the business issues, as well as being able to leverage resources or learning from other markets and offer innovation/novel approaches. Their business acumen underpins all factors.<br />Secondly, they are specialists who deliver insight and understanding, and this is backed up with effective use of sophisticated techniques and efficient turn-around times.<br />Performance perceptions tend not to be driven by cost for management consultants, nor use of traditional techniques and consistency with what done in the past.<br />Figure 3: Performance Perception Drivers – Management Consultants<br />It is worth noting that the R2 is not very high, indicating, we believe, that there are likely many other factors that drive perceptions of Management Consultants (that were not included in our survey). However, it is clear based on the attributes and factors included that drivers are related to the core reasons why clients use this type of agency and are prepared to potentially pay more for their time – for their specialist expertise and business acumen. Their positioning allows them to create value for clients.<br />Keeping in mind the financial challenges and current climate, as well as perceptions of commoditisation and decreasing value from research, the industry may well find it needing to re-build value perceptions... and might take learnings from other consulting businesses such as Management Consultancies.<br />It is interesting to note that a similar conclusion or similar recommendations were made by senior managers interviewed in Roth’s 2004 paper on how to ensure research companies remain useful and relevant to business. It appears that these words of advice are still relevant today:<br />“Think business first, research second” <br />“Identify business opportunities yourself” <br />“Understand the clients’ business and build business partnerships”<br />“Clients like to be challenged – don’t take briefs on face value”<br />Noteworthy also, Roth also talked about reviewing cost structures in the industry given that “data collection is the most difficult feature to differentiate companies on because it is seen to be a ‘commodity”. While we agree with this in our research, data collection is not necessarily any longer the “item that costs clients the most”, and moving to a different costing structure may no longer be sufficient to increase value perceptions. Returning to the traditional news/press analogy, we believe people will be prepared to pay for commercially relevant knowledge and insights where they would not be prepared to pay for information without analysis and consideration.<br />The value of innovation<br />Roth’s respondents also talked about the need to “keep innovating”. Across the eight agency types we measured, it was the New Media companies that stood out as being perceived as performing most strongly on ‘effective use of cutting edge technologies/more sophisticated tools’ (although larger, global Market Research companies were not far behind), as well as for being ‘specialists in a specific technique or approach’ – these were their core strengths, based on the attributes that we included in our survey.<br />Of note here, when we look at performance perceptions in Australia compared to UK/US for New Media companies, we see a tendency for UK/US respondents to give higher ratings in most areas with the exception of New Media Companies. Perceptions are aligned when rating New Media Companies in Aust, UK and US. These agencies are rated as being specialists in a specific technique, having ability to leverage resources/earnings from other markets, ability to turn things around quickly, being cost effective/providing great value and their ability to create successful partnerships.<br />Where performance appears to be lower, it is the Australian respondents who have rated the New Media companies down, for example on: business acumen, ability to provide clarity and prioritisation of business issues and providing consistency, and there is also an apparently large discrepancy in perceptions of their ability to challenge you, provide in-depth consumer understanding and offering tailored solutions.<br />We could reach a number of conclusions here; one that New Media businesses locally are not performing as strongly on these aspects (commercial focus, insight, tailored solutions) compared to possibly more established overseas businesses, or that Australian businesses have higher expectations or that they are simply more critical of performance compared to our UK/US counterparts.<br />Nevertheless, the New Media companies are performing more strongly on the ‘effective use of cutting-edge technologies’ than any other agency type and are clearly specialists in their area, which is being valued by research buyers.<br />We believe there is evidence that we need to invest in and drive new techniques or technologies to help, for example, to cope with and harness the surge in available data and to remain competitive against the non-traditional Market Research companies that are springing up to deal with new data sources. We believe that the Market Research industry needs to remain flexible and adaptable, with the end game being use of new technologies to help achieve positive client outcomes (e.g. faster, better, cheaper) – not the tool as an end in itself. <br />“We need to craft methodologies that add direct bottom line impact by engaging consumers in an ongoing relationship with brands”<br />The risk of not doing so is to lose not only relevance and freshness, but to lose out to DIY research or other competition:<br />“We are also trialling new things ourselves in that area of research. Research companies are not up to date with those 2 challenges. I think FMCG companies are more up to date than research agencies” – FMCG IDI<br />What can Market Research learn from Media and Advertising Agencies? <br />Advertising agencies stand out for their use of traditional or proven techniques, but are not seen as cost-effective/providing great value (on par with Media agencies and global Market Research businesses). Overall performance is ‘average’ for Advertising agencies and lowest (top 2 box %) for Media agencies amongst the agency types we surveyed.<br />However, Media agencies perform very strongly (highest) on being ‘specialists in a specific technique or approach’ and also for ‘offering innovative approaches that help provide competitive edge’, but also ‘provide consistency’ with what was done before. Across the eight agency types measured, Media agencies scored lowest performance on ‘having knowledge of/being specialists in your category/business’ though.<br />Overall, perceptions of Media agency performance were similar across markets (and industry types), but there was a tendency for UK/US respondents to rate them higher across the board (all attributes), and this is also true for Advertising agencies (although overall opinion was also slightly higher in UK/US).<br />In our further analysis, we were not able to reliably identify performance drivers based on the attributes captured (R2 0.236 for Advertising, R2 0.169 for Media and not significant) suggesting that there are other reasons which come into play in determining why companies work with Media and Advertising agencies that were again not included in our survey (which was focused on Market Research industry performance). One such reason, as we came to understand in our Qualitative discussions with clients, was that Advertising agencies, for example, are often paid on a retainer basis rather than on an ad hoc basis, and therefore performance may not be measured on a project-by-project basis.<br />While one client did not think that research agencies would be able to work off this model, they did believe that this was something the industry could learn from in terms of the partnership created:<br />“I don’t think the model that we have with our ad agencies and media agencies is going to happen with research agencies... With the retainer there are no confidentiality issues that are something I think the research industry can learn from, because we don’t keep anything from them (the retained advertising agency). They are a true partner in the business, and while I don’t think we can ever get to that level, I think we can learn from it” – FMCG IDI<br />In conclusion, like the Management Consultants and with the added benefit of being experts in their area, Media and Advertising agencies may be building stronger relationships with clients as ‘true business partners’, which is, we believe, the direction the Market Research industry will need to take in order to make possible commercially relevant foresight and strategies, as reiterated in one of our In-depth Qualitative interviews:<br /> “We work with agencies where we have a conversation and we can collaborate together” ... “We work together to make sure we have the most actionable findings for our business ... our agencies need to understand our business” – FMCG IDI<br />“There are very few (Market Research agencies) who can act as consulting supplier. This is more people dependent than agency related, although it is fair to say that some agencies try.” FMCG IDI<br />“20 years ago traditional research was to provide tables without summary, today what does traditional mean? ....it is more important than ever to connect the dots, because there are even more dots with new technologies” FMCG IDI<br />Specialist techniques are desired but it’s what you do with them that matters<br />Sales & Trend specialists, including the likes of Nielsen, Mintel etc., also stood out for being ‘specialists in a specific technique or approach’ (like Media agencies), and were also rated strongly on their ‘ability to leverage overseas resources/learning’ and ‘use of traditional/proven techniques’ and hence rated lower on innovative or cutting edge approaches.<br />We found agreement between Australian and UK/US companies on these core performance areas, but marked differences on most other attributes, with Australian respondents rating Sales & Trend companies down, particularly in their ‘ability to provide clarity and prioritisation of business issues’, as well as on ‘challenging you’.<br />Overall performance was on par with most other agency types (in the middle) and key drivers grouped into three dimensions, all similarly weighted as show in Figure 8. <br />Figure 4: Performance Perception Drivers – Sales & Trend Specialists<br />While performance across all respondents was rated strongest on being specialists in a certain technique, use of proven techniques and ability to leverage resources from other markets, it appears that these are not main driving factors; rather it is their business acumen, pro-activity and ability to provide insight, tailored solutions and be cost-effective that drive opinion of performance in helping businesses make successful commercial decisions (and these were also the areas in which there were discrepancies between Australia and overseas respondents, i.e. driving the overall difference in performance perceptions).<br />These findings reflect those from the GRIT report which concluded that:<br />In an age of high tech solutions, the “high touch” factors...remain as important today as they have ever been. Listening well and having a good relationship with the client – along with familiarity with the client’s needs, rapid response, meeting deadlines, and having a knowledgeable staff – are key discriminators.<br />While new technologies, techniques and innovative approaches are desired and clearly have a role to play, the lesson here is again clear – it is not the tools that help clients make successful commercial decisions, but the knowledge or commercially relevant insight which comes from harnessing such tools, which is the role that the Market Research and other consulting agencies play.<br />Market Research Agencies – what we do well and what we could do better<br />If we include small, senior-led, medium and larger, global Market Research agencies as one business, our research indicates that the industry has tremendous breadth and also variety. While small, senior-led agencies are performing strongly on business acumen, partnering factors, ability to deliver commercial insight and consumer understanding, as well as offering novel approaches or specialist techniques (strengths most similar to the Management Consultancies), the larger, global Market Research businesses are more able to leverage resources/learnings from overseas partners, effectively use cutting-edge tools, or work with proven techniques and provide consistency. Larger Market Research companies also score slightly higher on creating successful partnerships. <br />Figure 5: Performance Drivers – Small senior-led Market Research<br />For smaller, more senior-led research agencies, it is exactly these strengths, their commercial, customised focus, their expertise and ability to partner with clients and be innovative, that drives perceptions of their performance in helping businesses make successful commercial decisions. But for larger, global Market Research agencies, their ability to provide consistency with what was done before and leverage learnings from other markets where they operate that are their key performance drivers. <br />The medium-sized, local Market Research companies stand out for being able to turn things around quickly and efficiently, offering tailored solutions and for providing in-depth understanding and insight; they also scored highest on being cost-effective and providing great value. In comparison to the small and larger Market Research agency types, the analysis indicates that the 4 dimensions driving performance perceptions are almost equally balanced, but their ability to be cost-effective/provide value is a key driver on its own.<br />Figure 6 SEQ Figure * ARABIC : Performance Drivers – Medium, local Market Research<br />Figure 7: Performance Drivers – Larger, global full service Market Research<br />Summary<br />In summary, research agencies are clearly performing well across a wide variety of factors, each with their own particular style and angle (although all performing well on being a source of independent objectivity – which is possibly being taken for granted and not really discriminating between the agency types discussed, either Market Research or otherwise) – but no one agency is everything to every man.<br />Having looked at clients’ business challenges and changing needs, we believe there are a number of performance gaps and therefore learnings that can be taken from this research, which impinge upon how research or information providers do business. Clearly, there are ever-increasing time and budget pressures; budgets are stretched and more needs to be done with less, and in less time than before; decisions need to made and actions taken without delay. Agencies need to be able to react to this environment and be faster, flexible and more efficient than ever before:<br />“We’re looking for newer, quicker ways of working ... but gone are the days when you’re given 2 weeks to write a proposal... it’s more likely that if we give you a brief we need to proposal the next day.... so we need research that is as dynamic as our business, which is where there is a gap at the moment” – FMCG IDI<br />At the same time, agencies need to be more cost-effective or be perceived as giving greater value (at the same price), but not at the expense of quality. One or two agencies types appear to be doing reasonably well on cost-effectiveness in comparison to others, but there is a sense that, at least for some clients, it’s not necessarily about lower cost, but more about finding alternative ways of doing things that are just as good, if not better, but which are also more cost-effective, faster and add value:<br />“Agencies need to think like us, smarter, efficient, and more for less – it’s about value” – FMCG IDI<br />“The key things I look for in an agency are expertise, senior contact person who manages my project from beginning to end and competitively priced” – FMCG IDI<br />Further, we have seen that the world is changing – fast. Data is not only becoming more available more easily and more cheaply, there is a plethora of data from a growing number of new sources, thanks to the internet and social media. However, clients recognise that the data has little intrinsic value in itself, but it’s what you do with the information that counts. Our research and analyses also showed that it is the interpretation of data, the delivering of insights and understanding from the information and the thinking that drive perceptions of strong performance:<br />“For research agencies going forward, they should act more like consultants. By that I mean put more thinking into the reports, not just report slides and slides of data” – FMCG IDI<br />“Anyone can do research, it’s not the depth of data that is going to win anyone starts, it’s what you do with it. ...Agencies need to think more about their depth of understanding not breadth... they don’t want to weaken their expertise in something” – FMCG IDI<br />“It’s not the traditional research that we would have done, but it’s about interpreting the facts. We’ve always had the info but it’s what you do with that that’s important” – FMCG IDI<br />Again, some agencies are already delivering on this, clients feel, so it’s not necessarily perceived to be a performance gap. Rather, the need to deliver insight and understanding is, on the one hand, related to the areas on which there is room for further development (i.e. commercial expertise and relationships), and on the other hand, may become a different type of challenge as data sources and volume explode. Do research agencies have the right skills and technologies for dealing with this commoditisation of data?<br />Clients are facing ongoing, tough challenges with competition intensifying, and with various technological, economic and social changes impacting customer behaviour and needs. Clients are looking for new ways to tackle these challenges and are looking to research and other business partners to help them. To do this well, and in order to help ‘provide clarity and prioritisation of business issues’, agencies need to be able to act more as consultants, to develop stronger, ongoing partnerships, challenge thinking in the business, be proactive and have deep knowledge of the client’s business/category (as well as turning data into insight, foresight or strategy), as highlighted by the following comments:<br />“Agencies just need to work at blurring the line of the researcher and the agency – being seen as a consultant...“– FMCG IDI<br />“We now use agencies that either come to us with thought leadership on some of these areas and we trial them, or we use agencies that we have a good relationship with and are willing to work with us and where we think the thought leadership is. They are flexible and wanting to work with us, and are hungry to try new things for new solutions” – FMCG IDI<br />“The days of factory research agencies are going” – FMCG IDI<br /> “I think research agencies are on the right path. They just need to have a presence in the business, a history with us and be close to the team” – FMCG IDI<br />All in all, these last quotes sum up our thoughts. Research agencies are not doing badly, but, as with much market research, we are looking back in time. Looking forward, we know that our clients’ landscape is changing rapidly, and predict that if we want to continue to be successful, we will need to change too and potentially quite radically if we want to really help our clients tackle the big challenges ahead. <br />6.Conclusions – where to from here?<br />There is a sense that we are on the verge of a significant shift in the Market Research paradigm; given the explosion of new technologies, emergence of new competitors including DIY research, and the move from a long period of (data) scarcity to one of infinite abundance, and commoditisation.<br />We have seen (from our reading) that the world is changing – both in terms of the market conditions (economic, political and social), and in terms of the technologies now available to and assisting in the research business. <br />Looking back at the history of the Market Research industry in Australia, we have evidenced the evolution of the Market Research business model; the Market Research industry has had to change and adapt with the external conditions and challenges, as well as exploit new technologies as they emerge and become viable and accessible for Market Research purposes to meet new needs and remain competitive. 80 years of innovation and growth surely proves that the industry has the ability to adapt and survive.<br />The shift from data scarcity to abundance which, coupled with the new technologies available that continue to expand exponentially, will inevitably change the Market Research business model again, shifting the competitive framework and client expectation.<br />We believe we will need to move our focus from being data and insight providers, bound by our rules and methodological rigor, and project by project based; to being creators of foresight and strategies, being ‘methodologically agnostic’ and working hand in hand with clients, not as suppliers but as an extension to their business. This requires moving from an ad hoc project by project orientation, to a continual process of insight utilising multiple data streams that are invisible for consumers.<br /> In this emerging approach, we will need to be extremely clear about what business/client questions we are looking to find the answers to in the vast ocean of information, and we will need to develop (or borrow and adapt) new tools and techniques to harness this abundance of data (e.g.; data mining, analytics and integration) to ensure what has been our speciality does not become a simple commodity.<br />Our research has revealed that clients are facing challenges which reflect the current market trends and social changes, so client understanding and innovation remain vital, however commercial reality drives tighter budgets and faster insights. Some clients are resolving the pressures by looking to new approaches, including considering how to make the most of social media, ideally at potentially lower cost than ‘traditional’ Market Research approaches. <br />The research buyers that we surveyed; both in Australia and overseas were still relatively supportive of the Market Research industry as a whole versus other consulting options. There were no standout shortfalls in Market Research support versus consultants and other information providers. However, we anticipate that we are unlikely to continue to enjoy such a measure of satisfaction unless we continue to evolve with the changes around us. This research does find evidence of differing consultancies having varied strengths versus