MWD Advisors White paper: Unlocking the potential of Big Data


Published on

Big Data is one of the hottest trends in the IT industry today.
Why should you care? Download this independent report .

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

MWD Advisors White paper: Unlocking the potential of Big Data

  1. 1. mwdadvisorsUnlocking the potential of Big DataHelena SchwenkA special report prepared for ActuateMarch 2012Big Data is one of the hottest trends in IT industry circles. Although overused as a buzzword it isgenerally characterised by the large, varied and rapidly growing volume of information that oftenremains untapped by existing BI and data warehousing systems. It’s data that comes in all shapes andsizes emanating from sources as diverse as mobile phone, sensors, smart energy meters, e-commerceand social media sites. Yet within all this data lies significant value – especially for those businessesthat successfully tap into it, exploit it and put it to work for better business effect. As an emergingtechnology discipline it also brings its own set of challenges in terms of scarcity of skills and IT bestpractices. But for those organisations that can overcome these obstacles there’s huge potential tounlock its value as a way of enhancing productivity, driving efficiencies and growth, and creating asustainable competitive advantage.This is a special report prepared independently for Actuate. For further information about MWDAdvisors’ research and advisory services please visit Advisors is a specialist IT advisory firm which provides practical, independent industryinsights that show how leaders create tangible business improvements from IT investments. We useour significant industry experience, acknowledged expertise, and a flexible approach to advisebusinesses on IT architecture, integration, management, organisation and© MWD Advisors 2012
  2. 2. Unlocking the potential of Big Data 2SummaryDefining Big Data is not Pinning down a definition for Big Data is an ongoing challengestraightforward especially since the industry and marketplace has yet to reach consensus. Until there is some form of agreement it’s best to consider Big Data by its core characteristics. To begin with, Big Data is not just about data volume – it also needs to take into account its shape, speed, complexity and variety. Secondly, Big Data can often be characterised as data that’s either too difficult or not economically viable to store and process using traditional data warehousing systems and BI tools.Analytics brings Big Data to life While it’s easy to get hung up on the complexities of storingand unlocks its potential and crunching Big Data, this activity on its own will not help you unlock its true business value. Leveraging advanced analytic capabilities such data mining and text analytics, on the other hand, can provide the means to enable you to answer new questions, discover hidden insights, or find unknown relationships in data to drive real business advantage. In turn this can enable you to keep ahead of the curve, discover new revenue streams, reduce costs, enhance the customer experience and build sustainable competitive advantage.Harnessing and exploiting Big Data The mining of Big Data has the potential to reveal actionablecan bring significant rewards and valuable insights across multiple industries, organisational sizes and business functions. This is possible not least because at the same time as the quantity and variety of data continues to grow, the technology for capturing, managing and analysing all of this data is steadily improving – and at an increasingly affordable price. So although we’re at an early stage of market maturity, the potential for Big Data applications to create value, enhance competitiveness and improve productivity are widespread – including those for better fraud detection, deeper levels of customer segmentation and more accurate consumer behaviour predictions.Success requires blending business As you plan your Big Data initiative there are a range ofneeds with investments in Big Data considerations that need to be taken into account to ensuretechnology, data integration success. These include understanding the business need orpolicies, and the right analytic challenge; securing the right level of commitment andtalent. investment from senior management; getting to grips with the types and complexity of data sources at your disposal; ensuring you navigate the technology landscape and choose the right tools; and ensuring you invest in the right people with the right skills to exploit your Big Data to its full effect.© MWD Advisors 2012
  3. 3. Unlocking the potential of Big Data 3Big Data makes its Big ImpactWhat’s the Big Deal with Big Data?Unless you’ve been living in a vacuum it’s hard to avoid a conversation in todays business technologycircles without touching on the subject of Big Data. Similarly most press coverage of the topic centreson Big Data as the new do-or-die technology that businesses need to leverage if they want to stay inthe game and remain one step ahead of the competition. It’s not surprising given these headlinestherefore that certain commentators have already written off Big Data as a bubble that is set to burstand leave many IT organisations despondent in its wake.While there is no shortage of hype – and there may very well be casualties along the way – Big Data’sprominence and ascendency is driven by a very real business challenge, namely the unprecedentedgrowth of digital data across nearly every industry, region and size of organisation. It’s a challenge thatisn’t going to go away, as the figures demonstrate. According to a McKinsey Global Institute report1,in 2010 enterprises globally stored more than 7 exabytes of new data on disk drives, while consumersstored more than 6 exabytes of new data on devices such as PCs and notebooks. Likewise otherindustry insiders point to the fact that that 90% of the data in the world today has been created in thelast two years alone. This tsunami of digital data is being generated by businesses and consumers alikethrough social networks, sensors, online videos, e-commerce sites, GPS signals, printer streams andCall Detail Records (CDR), to name but a few.However, storing and managing this data is only one part of the challenge; the exponential growth ininformation is also being matched by a strategic need and desire by businesses to find hidden nuggetsof information within this data. Extracting value and insight can help organisations keep ahead of thecurve in their quest to discover new revenue streams, reduce costs, enhance the customerexperience and build sustainable competitive advantage. Harnessing and extracting value from BigData is seen by many as a route to achieving these aims, where data is no longer purely seen as a ‘byproduct’ of doing business, but is instead seen as an important asset that can be utilised to inform,guide and improve the quality and speed of decision making.In truth, any Big Data effort is likely to bring both opportunities and challenges for organisations. Tobegin with, the management of Big Data is a difficult and complex undertaking. This is not onlybecause of the sheer volume of data that is being created, but also due to the variety of data types itencompasses (such as unstructured and structured data), as well as the speed of its delivery, which insome cases might be in real time. Similarly, once this data has been captured, stored and analysed,organisations need to understand how those insights pertain to their business and how they can acton them in a timely and effective manner. Yet in spite of this, the overriding fact remains that BigData, if used and harnessed successfully, can bring enormous benefit and value to organisations –something that far outweighs the challenges and obstacles present in storing and processing it. In fact,for some the benefits will only be limited by their ability to use data in more imaginative and valuableways.What’s in a name?Given its ubiquity as a term, coming up with a definition for Big Data is not as straightforward as youmight think, especially as the technology industry has yet to reach any kind of consensus. Whilepinning down a definition can be akin to hitting a moving target, it’s helpful to consider Big Data by thecharacteristics and traits it exhibits and in terms of how it differs from other more traditional datamanagement approaches.1 Big data: The next frontier for innovation, competition, and productivity, May 2011, McKinsey Global Institute -© MWD Advisors 2012
  4. 4. Unlocking the potential of Big Data 4Our research suggests that to understand the full scope of Big Data management, it needs to beframed in the following contexts: Big Data is not just about data volume – it also needs to take into account its shape, complexity and scope. In contrast to more traditional data management approaches, it encompasses semi- and unstructured data as well as structured data, and includes data generated not only by humans but machines too. Similarly, the management of Big Data needs to takes into account data that is both ‘at rest’ – where data is captured and analysed at a point in time – as well as ‘in motion’ – where data is analysed as a continuous stream on the move. Big Data can often be characterised as data that’s not economically viable to store and process using traditional data warehousing systems and BI tools. In this sense it often requires a new technology, analysis and architectural approach to data management to harness it effectively. Don’t get distracted by size. Big Data is a subjective measure and can start from anything from hundreds of terabytes to datasets that hit the petabyte range. What’s more important is the context of its use in a traditional enterprise setting: Big Data projects typically apply to scenarios where data has previously been too challenging to store and process or where data simply hasn’t been accessible before. Big Data can be sourced from both inside and outside the organisation, whether it’s in social media data streams, sensor logs or transactional data stored behind the firewall.© MWD Advisors 2012
  5. 5. Unlocking the potential of Big Data 5Business opportunities associated with Big DataTapping into the gold mine of Big DataWhile a lot of buzz has focused on the technicalities of storing and processing Big Data this view oftenoverlooks the most important question: why should you care? The answer lies in uncovering the BigData sources that hold potential treasure troves of information that can be explored, mined andcombined with existing data to unlock secrets, opportunities and potential successes that are alignedwith the needs of your particular business. This means that there’s no simple ‘one size fits all’ answer.The effective management of Big Data promises deeper and richer insights based on the ability towork with individual records, rather than basing insights on an aggregated data slice (typically foundwithin a data warehouse), or a sample of the information to hand. This is especially true inexploratory data analysis where analysts don’t always have a clear understanding of the questions theywant to ask of data. Without the benefit of Big Data technologies and techniques, analysts have nochoice but to work with partial data, which can introduce errors or limit the scope of analysis,whereas analysing a complete set of data allows organisation to get answers to questions that haven’tbeen posed before. In this sense, taking advantage of a Big Data opportunity requires a more creativeand inquisitive approach to data analysis and problem solving – one that combines the ‘science’ ofanalytics and data discovery with the ‘art’ of applying it to real-world scenarios and revenue models.Likewise, since a lot of what commentators call Big Data emanates from embedded sensors found inmobile phones, medical devices, automobiles or smart energy meters, the use cases for its analysis canextend to areas outside of the traditional domain of BI and analytics, within industries such ashealthcare, oil and gas, and transportation. In these scenarios Big Data can enable organisations to useadvanced correlation techniques to identify potentially useful patterns that would otherwise remainhidden in petabytes of data.Analytics brings potential to Big DataGiven all this potential it’s worth underlining the fact that Big Data on its own cannot unlock businessvalue. Instead it’s the application of Big Data to real-world business scenarios that provides scope forcompetitive advantage. As shown in figure 1, it’s about pulling data together, combining the righttechnologies and tools, applying analytics and creating actionable insights that business managers canuse to make better, higher quality and quicker decisions.© MWD Advisors 2012
  6. 6. Unlocking the potential of Big Data 6Figure 1: The Big Data mix Business need Big Data technologies Analytic & skills & architecture techniques Data integration Actionable insightsSource: MWD AdvisorsGetting the right mix enables organisations to sift through, find and exploit new patterns andrelationships in the data in order to, for example, identify risks, anticipate and respond to changes inmarket conditions, and predict customer behaviour, conditions and events in ways that previouslyhaven’t been possible before. Similarly it can be used to add insight to existing analytics such as fine-tuning customer segments for more targeted marketing campaigns, crafting better marketingstrategies, devising more profitable pricing strategies, offering more sophisticated productrecommendations and helping organisations discover new products and services.It’s clear from some early use cases that the power of Big Data can yield some impressive businessresults. However, the challenge for most organisations comes not only from how you process,explore and mine Big Data, but also from understanding how those insights are relevant to yourbusiness and how you can act on them in a timely and effective manner. In the next section we willmove on to talk about some of the most common use cases for leveraging Big Data in this businesscontext.© MWD Advisors 2012
  7. 7. Unlocking the potential of Big Data 7Understanding the use cases for Big DataThe possibilities for tapping Big Data to reveal valuable insights seem almost limitless, particularly asit’s a phenomenon that impacts multiple industry sectors, organisational sizes and business functions.Just as the quantity and variety of data continues to expand, the technology for capturing, managingand analysing all of this data is steadily improving at an increasingly affordable price, allowing morebusinesses to leverage and exploit the potential of Big Data. Although it’s still early days in terms ofreal-world use cases, there are signs that organisations are actively pursuing Big Data opportunities tocreate value, enhance competitiveness and improve productivity. Today organisations are mining thedata they’re currently capturing and storing, although may not necessarily be exploiting to its fullpotential. At present, our research suggests that the usage scenarios for Big Data fall into one of fourbroad business opportunities and drivers, as shown in figure 2.Figure 2: The business opportunity for Big Data Business Driver Opportunity Example Improving Identifying and Telcos can analyse growing volumes of CDRs, together with operational preventing customer interaction usage, network and transactional data to discover and efficiencies churn predict new forms of churn in their network. Fraud detection Insurance companies can identify patterns of fraudulent behaviour much much faster found in terabytes of online, mobile and transactional data for insurance claim fraud and anti-money laundering, Pinpointing areas for cost Retailers can use data captured from loyalty reward programs, and efficiencies in store, mobile and online transactions to optimise and improve margins for product inventory, and markdowns Mitigating risk Financial service companies can monitor and analyse financial data streams in faster timescales to identify and minimise their credit and market risk exposure. Enhancing the Understanding customer Consumer Package Goods companies can acquire and mine customer experience sentiment unstructured data from social networks to get an overall picture of their brand’s perception and conduct real-time market research. Fine tuning customer Financial institutions can segment customers by credit card segmentation behaviour at a finer level of granularity to target and tailor products more effectively to specific risk profiles Gaining a 360-degree Organisations of all sizes can capture and accumulate a wider range view of the customer of customer attributes to gain deeper and more accurate insight into customer behaviour and model it with greater precision. Improving revenue Identifying new sales Web-based companies can get a fuller picture of visitor usage and generation opportunities purchase patterns to help optimise website design, content creation and develop product recommendations that boost traffic and sales. More granular customer A retailer can collect and mine customer purchase data to micro- targeting segment its customer base that is used to optimise its product mix, pricing, and promotions more accurately Driving strategic Better planning and Utilities and energy companies can tap into vast volumes of smart change performance meter data to accurately predict retail demand and control supply management costs in ways that have not been possible before. Understanding new A credit card provider can create value from the wealth of data it is markets storing and analysing by selling consumer insights based on the data streams it generates from processing payments. Discovering and Healthcare providers can aggregate and analyse enormous volumes developing new products of clinical and claims data, to find the next big ‘super’ drug that will or services help .Source: MWD Advisors© MWD Advisors 2012
  8. 8. Unlocking the potential of Big Data 8Bigger, better and fasterWhile some of these application areas and use cases are familiar and well understood by BI and datawarehousing communities, what’s different now is the scale and scope of analysis that Big Data canenable. In other words, if leveraged in meaningful and more accurate ways Big Data can help youexploit information to do things bigger, better and faster. This in turn will place new requirements onBI and analytic toolsets as they are called upon to support the volume, speed, variety and workloaddemands of Big Data. It’s an effort that will require you to look seriously at the technologies used todrive both your current and future data management strategies and information needs.To begin with, your BI environment will need to extend its support for analytic techniques such asdata mining, predictive modelling, natural language processing, machine learning and advanced SQL, aswell as improving support for collaboration, data discovery and visualisation techniques to helpinterpret the results of Big Data analysis.At the same time this needs to be married from a data management and integration perspective withcapabilities for sourcing new forms of data, including unstructured and structured data, the ability tosupport both high and low latency data demands, as well architectural support for scale-out and highspeed data processing.Today these Big Data challenges cannot be solved by a single platform or engine but instead need toemploy a variety of technologies, components and architectures. These may include technologies suchas Hadoop, MapReduce and distributed NoSQL databases, but it could also include technologies suchas in-memory databases, columnar databases and massively parallel processing architectures. Howeverfor some, the real potential value of Big Data will only come when it’s merged and integrated withexisting business processes and data assets, such as a data warehouse, to provide a fuller and morecomplete picture of their business.Finally, any Big Data effort will require you to think carefully about sourcing and investing in the rightpeople, analytic skills and experience to make sure you can take advantage of the huge opportunitiesthat Big Data presents.© MWD Advisors 2012
  9. 9. Unlocking the potential of Big Data 9Where to start on your Big Data journeyAs you plan to embark on a Big Data initiative there are a range of considerations to take intoaccount and challenges to overcome if your initiative is to realise its full potential. You need todevelop a practice that involves assessing business priorities and needs and match these withinvestments in Big Data technology and techniques, data integration policies and the right analytictalent. To assist you on the path to Big Data success the following steps provide guidance about howand where to start your Big Data journey. Get buy-in and commitment. It’s true to say that all IT programmes benefit from having senior-level sponsorship and buy in, but this is especially true in the case of Big Data projects. A sponsor needs not only to invest time and money in any effort but also match this with a compelling vision and understanding of how Big Data can unlock real business potential for your organisation. Choose your data sources. A large part of the Big Data effort involves assessing the type and format of data sources you want to use. In many cases this could mean considering opportunities for analysing new types of data such as log files, sensor data or video streams that were previously not available or possible before. Good data preparation reaps rewards. It doesn’t make sense to always subject Big Data to the same rigorous data cleansing, scrubbing and matching routines required in an enterprise data warehousing environment. However, in certain scenarios you will still need to transform the data and apply hygiene routines to Big Data in order to maximise its potential, for example by ensuring you have prepared the data for analysis and rectified any data quality issues in the source data. Change the way you think about data. The ability to analyse all of your data rather than just a subset or sample will require a subtle but different analytic mindset. Big Data environments are often regarded as exploratory platforms where analysts can dig and play around in the data as they attempt to uncover new and interesting insights. It’s a mindset that requires a more creative and inquisitive approach to data analysis and problem solving, and one that combines traditional analytic disciplines with the ability to apply these to real-world business scenarios. Pick your tools. With such an array of technologies and architectures to choose from, expect a considerable part of any Big Data effort to be spent on understanding and navigating the technology landscape. You need to consider key capabilities such as the performance, scale, and data delivery rates of each tool or platform alongside support and integration with BI and advanced analytic tool and techniques. Invest in skills, skills, skills. Finding the right talent to utilise Big Data technologies and techniques will continue to be a challenge for most. Those of you who are new or have had limited exposure to disciplines such as Hadoop, data mining or statistics will need invest time in sourcing or training staff. However, this is only part of the story: there should also be an equally concerted effort to employ and develop those skills for aligning the data with the business, so insights derived from Big Data can be used to drive better decision-making and business outcomes.© MWD Advisors 2012