• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Six Step approach to identify a Big data problem and choose the right Solution
 

Six Step approach to identify a Big data problem and choose the right Solution

on

  • 936 views

Many have heard the dire predictions about the state of information technology with 10x data growth projections over the coming years. While there is truth to the exploding growth rate of data and ...

Many have heard the dire predictions about the state of information technology with 10x data growth projections over the coming years. While there is truth to the exploding growth rate of data and the accompanying complexity of analysis, we have faced similar exponential growth in data over reach of the most recent decades; and every time, technology has risen to the challenge and delivered needed capacity for business, governments and individual users. For parallels we need only look to distributed computing in the 1990s and websites in the 2000s.

Statistics

Views

Total Views
936
Views on SlideShare
933
Embed Views
3

Actions

Likes
0
Downloads
37
Comments
0

1 Embed 3

http://affordabletermitecontrolnews.com 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Six Step approach to identify a Big data problem and choose the right Solution Six Step approach to identify a Big data problem and choose the right Solution Document Transcript

    • Six-Step Approach to Identify a Big DataProblem and Choose the Right SolutionAuthor : Manju DevadasVP Solutions and Technology, Bodhtreemdevadas@bodhtree.comwww.linkedin.com/in/manjudevadas
    • Many have heard the dire predictionsabout the state of information technology with 10x datagrowth projections over the coming years. While there is truth to the exploding growth rate ofdata and the accompanying complexity of analysis, we have faced similar exponential growth indataovereach of the most recent decades; and every time,technology has risen to the challengeand delivered needed capacity for business, governments and individual users. For parallels weneed only look to distributed computing in the 1990s and websites in the 2000s.In the 2010s, big data is a phenomenon nearly everyone comes into contact with, whether theyrealize it or not. If you carry a smartphone for work or dump thousands ofdigital photos on yourhome computer, youre already swimming in the Big Data ocean. You just may not have thetools yet to capture, store and process that massive data flow for better decision-making.The purpose of this paper is to demystify Big Data and provide a methodology to assesswhether the problems you encounter in your enterprise are Big Data problems. Working withlarge companies and start-ups in the Silicon Valley has allowed me to validate this methodologyin diverse business verticals and company sizes.If nothing else, this paper will help you act froma position of knowledge surrounding Big Data, avoiding the hype and misinformation thatcommonly accompanies the latest technologies.First Our EcosystemEach time you press a key, rate a product or navigate a GPS map, you generate data in someform. Of all this stored data,usually only a small portion is being analyzed to find new answersto challenging questions.If Samsung launches a new phone, the most unbiased and direct feedback today probablycomes from Facebook rather than traditionalcustomer surveys and support lines. Book a flightonline today at Kayak.comor Expedia.com and switch to your inbox to find emails with Pricelinehotel recommendations in a matter of seconds. How the hell did Priceline know that I wanted a3.5 star hotel in Monterey for the holiday weekend?With or without my knowledge, I allowedthem to capture my personal preferences and travel plans, analyze current web offerings usingBig Data, then email merecommendations.The Australian government is using Big Data to analyze seismology patterns and predictearthquakes precious minutes earlier. Big data analysis has found a role in vehiclemaintenance, predicting part failure; space exploration such as the Rover landings on Mars; and
    • fraud prevention, often identifying unauthorized purchases before a customer even realizestheir credit card is stolen.The total computing power provided by an optimized Big Data system is capable of analyzing allthe data on every desktop in your neighborhood in less than 1 second. The digital image of ahundred-year-old document could be retrieved from a city government’s archive databases inless time than it takes to pull a book from the shelf. The evolving technologies in Big Dataworld are not only making this kind of analytical power possible, they are democratizing itthrough approaches affordable even to the smallest businesses.The analysis of large volumes of data at lightning speedsis great, but does it actually create realvalue for people and businesses?Let’s start with your next promotion, which depends on thesuccessful launch of a new product line. Big Data becomes your cheat sheet for understandingcustomers, allowing you to proactively analyze buying trends and marketing strategies over thelast ten years of similar launches. Or consider biometric monitoring that signals you to go tothe ER before you actually feel any symptoms. Or maybe your goal is enterprise efficiency in acompetitive industry, and you need to identify and eliminate bottlenecks in your supply chain.Each of these challenges can be addressed with Big Data solutions.Solving most business problems in large companies involves some form of data analysis. Withthe data now being captured in all forms including human, environment, and machine-generated, it is necessary to identify which problems are Big Data-relatedand which can besolved using traditional data analysis techniques. The last thing management wants is topurchase a new system only to realize existing tools were capable of achieving the same results.(Remember the story of NASA spending millions to invent a pen to write in space when a pencilwould have been an adequate solution). Nothing is more wasteful in business than a greatsolution in search of a nonexistent problem.What exactly is Big Data?Big Data is simply complex data sets in massivevolumes (petabytes) and multiple formats (tablecontents, text, audio, video). With the speed and amount of data being generated today, thecorresponding technology demand is drivingnew ways to analyze the information faster,cheaper, and with better results.Three types of data may be present in your enterprise:
    • a. Large Volumes, e.g. Data stored in Database tables, Excel spreadsheets, Access database b. Unstructured data, e.g. Video, Audio, Facebook, Twitter, Blogs, Customer Reviews, Log files c. ‘Gray’ data, e.g. web traffic where the exact usage is yet to be determined based on business needs that may ariseEnterprises with ever increasing data volumes must take measures to better analyze these datasets to accelerate progress toward company goals and objectives. Even if business seems finenow, you may be ignoring this data at your peril since your competition could be using it to runmore efficiently, respond faster, and make better business decisions. As is so frequently thecase with technology, if you’re holding still, you’re falling behind.Enterprises also need to have people who can think about data in new ways – not justinformation stored in tables, rows and columns, but also data as blogs, videos, Facebook posts,GPS coordinates, and traffic sensors. As of today, these ‘Data Scientists’ are difficult to traininternally; and the natural reaction is to look for outside hires. But hiring from the outside canpresent its own set of challenges as these transplants may not bring the same understanding ofyour business challenges and differentiators. I recommend that you begin with the employeesyou already have and apply the methodology outlined below toward creating an effective BigData strategy and roadmap.How do I know if my problem is a Big Data Problem?Without delving into details about the nature of the business challenge and existing sources ofdata, it is difficult for anyone to determine for sure if the problem is a Big Data problem. AFortune 100 High Tech company in San Jose, California, paid us to fix what they labeled a BigData problem. Following the initial analysis, we concluded the problem was best solved withtraditional data analysis techniques rather than a Big Data implementation. We educated thecustomer about the unique characteristics of a Big Data problem, and saved the teamsubstantial money since their existing tools were adequate to solve the issues. Hence, eventhough no general model can substitute for a thorough hands-on analysis, the simplemethodology we outline below has been highly effective at quickly determining whether achallenge isBig Data-related.
    • Quite a few companies see Big Data as a concern only forweb product companies like Facebookand Google with petabytes of data to organize and process. However, a 2011 McKinsey GlobalInstitute study argues otherwise. The McKinsey report found that investment firms averagingless than 1,000 employees have 3.8 petabytes of data stored, a data growth rate of 40 percentper year and a mix of structured, semi-structured and unstructured data types. Overall,McKinsey found in 15 of 17 USindustry sectors have more data stored per company than theU.S. Library of Congress (which currently has 235 terabytes of data) and companies from allsectors have at least 100 terabytes stored, as shown in Figure 1:
    • Big Data Solution classification:There are five data conditions called the “Vs” that assist in defining a Big Data problem: 1. Volume, e.g. multiple petabytes of data 2. Velocity e.g. results need to be analyzed in seconds or less 3. Variety, e.g. Structured and unstructured data like social media posts and video files 4. Variability, e.g. Constantly changing like a stock marketValue, e.g. You’ve identified the clear business value you plan to derive from the dataWhat does all this mean?The relentless growth of data, new data formats todeal with, and the competitive advantagesachieved from managing large volumes of data all emphasize why Big Data should matter toyou. If you are an IT professional, you already recognize how difficult it can be to find a solutioncapable of handling a task as monumental as big data management. Whether you are lookingfor growth, profitability or productivity in your organization, you are invariably dealing withdata; and when that data shows the 5 V characteristics, you now need to start thinking of it as aBig data problem and approach it differently than traditional solutions.How do you get started?Many of the enterprises fail to implement a Big Data solution because they have not identifiedclear business cases for the tools. The common trigger to initiate Big Data development is adata blast that existing systems can no longer manage. As these datasets continue to grow insize, the enterprises face the problem of managing, storing and processing the data at thespeed required for timely business response.Below is the Bodhtree’s six-step process to take enterprises from Big Data ProblemDefinition to Solution Implementation, a methodology which has been applied with excellentresults at a large Bay Area networking company and several other Bodhtree customer locations:
    • Bodhtree Six Step process for a Big Data problem definition to solution delivery:Step 1: Understand the Use Case Depending on where you reside in the organization,the chances are high that you will first feel a sense of data overload before you can articulate a clear business case to leverage that data. Often this prompts enterprises to reactively implement a Big Data solution without deciding in advance what problems it will be used to resolve.
    • It is critical that you deep dive and understand the business case first before even thinking along the lines of Big Data. Otherwise, there will be a lack of focusthat feels a little like staring through a microscope at unintelligible detail without ever stepping back to see what is specimen sitting on the glass. In terms of IT, one Bodhtree client managing a large warehouse of customer, product and geography information with 100s of terabytes of data said he had a Big Data use case, but everything he spoke about involved only structured data, failing the 5 Vs test. Even before worrying about Big Data, do a litmus test by asking the following questions:  Business Case – Do I understand the value of solving the problem in hand? Can I quantify the potential value of a big data solution or at least articulate the qualitative benefits? • Dependencies – Have I collectedall the relevant information about the customer, install base etc.? • Complexity –Have I inventoried the data sources and characteristics to determine the complexity? • Lead Time – Have I created a reasonable plan with adequate time to acquire relevant hardware and data? • Initiative Alignment – Is the project aligned with corporate objectives and are project sponsors committed to the end-to-end process?Step 2: Understand the Current Landscape • Carefully analyzing the use cases defined in step 1 enables you to identifyall data entry and storage points. Often critical data entry points are discovered during this review process which were not realized initially. • Map the end-to-end process and data flows for the business capabilities, e.g.How does the data flow to you from the customer and among internal teams?
    • • Build a Reference Architecture to highlight the current systems and tools and its readiness for Big Data. Validate you have access to the data you plan to analyze.Step 3: Build a Blueprint • Define your overarching architecturalchallenges in doing theBig Data analysis defined in the use cases, e.g. What architecture will I need to store the customer install base information along with product information? • Identify the right high level Big Data solutions leveraging technology agnostic vendors and advisors. • Document a clear delta between As-Is & To-Be with the introduction of the Big Data solutions while addressing the pain pointsat eachtransition phases • Document the Risks & Dependencies that could impact business results, cost or schedule. Remember rolling out sophisticated tools does not guarantee success. Watch out for hidden landmines, e.g. Data Quality.
    • Step 4: Identify the Big Data Technologies • Deep dive into the Big Data technology dependencies and the impacts they have on the system/tools and organizations. For example, you might consider howHadoop adoption overlaps with your Business Objects installation to analyze the customer and product data. • Determine which users will be consuming the information and analysis. What formats do these reports need to be in? Do they require mobile interfaces? Current BI reports and subscribers often provide relevant insight to these questions.Step 5: Build a Big Data Roadmap • Avoid the traps of either over investing or under investing – have the business cases drive the solution. • Plan the roadmap for your Big Datarollout based on such factors as – • Business priority and management support. Remember, your execs may need to be educated in order to understand the relative business value offered by each phase. • Timeframe of expected results and ROI. • Big Data technology complexities, i.e. Apply the right order to ensure a clean data foundation before conducting analytics.
    • Step 6: Big Data Solution Rollout • Formalize the right team, experienced in conducting multiple implementations. • Divide scope items across multiple phases/releases to track progress and provide important quality checkpoints. • Document Business Requirement, Functional Analysis and the Solution Architecture • Begin user training before the implementation is complete so analysts can immediately realize business value, building momentum for expanded uses.ConclusionBig Data, by its very nature, contains endless possibilities for business insight and improvedoperations. But much like venturing into space without a defined mission, the Big Data worlddemands that businesses clearly define what they intend to achieve in advance. Otherwiseenterprises can spend substantially on fancy tools that may never happen upon real businessbenefit.Once those business goals are defined, and you have captured a clear picture of the currentstate of your data, apply the 5 Vs screening questions to determine if the problem trulywarrants a Big Data solution. An objective vendor that specializes in a broad cross section of BIand Big Data solutions can assist in this process and advise solutions that maximize your ROI.Upon identifying a Big Data problem, carefully proceed through the six steps of the Problem toSolution Methodology. Realize the real value of Big Data solutions do not come simply withimplementation but through applying creative and insightful approaches to harvesting businessvalue from the data. Ensure all dependencies are considered so that your data foundation is
    • clean, comprehensive and current. Finally, proceed with the implementation, highlighting“quick wins” to convey business value to execs and analysts building momentum for the fullimplementation.If the above methodologies are applied right, you will end up with saving time, energy andachieving better results with by applying the right Big Data prescription for a REAL Big Dataproblem.Contributors : Ryan Madsen, Sushanth ReddyReferences :• McKinsey Global Institute study• Bodhtree Customer Case Studies