If companies are not careful, "Big Data" will become "Big Dilbert"


Published on

A Dozen Things to Remember on your Big Data Journey. A presentation by John Bostick of LUCRUM, July 2013 to the JAX Chamber IT Council.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

If companies are not careful, "Big Data" will become "Big Dilbert"

  1. 1. If companies are not careful, “Big Data” will become “Big Dilbert” -- John Bostick, LUCRUM, July 2013 A Dozen Things to Remember on your Big Data Journey
  2. 2. A Dozen Things to Remember on your Big Data Journey 1. Data is Growing 2. Decision-Making is Accelerating 3. Data is Changing 4. Questions are Maturing 5. Methods are Changing 6. Architectures are Expanding 7. Technologies are Evolving 8. Terminology is Expanding 9. Business Cultures Are Transforming 10. Errors occur in milliseconds 11. Resources Demands are Escalating 12. IT Departments are Falling Behind 2
  3. 3. 1. Data is Growing Exponentially • Industry experts estimated that 75% of that growth comes from individuals. • Additionally, they estimate that 80% of data is used by commercial organizations. • The number of mobile- connected devices will exceed the world's population in 2013. • Your digital footprint extends across Facebook, Google, Twitter, Netflix, etc. 3 Anonymity is becoming algorithmically impossible. – MIT Technology Review http://mashable.com/2013/05/07/big-data-anonymity/
  4. 4. 1. Data is Growing Exponentially -- continued-- • Wal-Mart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes of data. • Akamai analyzes 75 million events per day to better target advertisements. • 20B credit card transactions per year in the US. • Kroger processed ~2B transaction logs per week (2004 Stat). • Individuals create 70 percent of all data. Enterprises store 80 percent. • In other words, the amount of data in the world today is equal to: – Every person in the US tweeting three tweets per minute for 26,976 years. – Every person in the world having more than 215M high-resolution MRI scans a day. – More than 200B HD movies – which would take a person 47MM years to watch. 4 Sources: http://wikibon.org/blog/big-data-statistics, www.indexcreditcards.com, http://www.csc.com/
  5. 5. 2. Decision-Making is Accelerating 5 • Is my brand profitable? • Is my customer profitable? Data Velocity --------- > DataVolume---------> • How do I influence my customer? • How do others influence my customer? • Do I need to act now to keep my customer happy? Answers are needed at more granular level and a faster pace!
  6. 6. “We welcome change and openness; for we believe that freedom and security go together, that the advance of human liberty can only strengthen the cause of world peace. There is one sign the Soviets can make that would be unmistakable, that would advance dramatically the cause of freedom and peace. General Secretary Gorbachev, if you seek peace , if you seek prosperity for the Soviet Union and eastern Europe, if you seek liberalization, come here to this gate. Mr. Gorbachev, open this gate. Mr. Gorbachev, Mr. Gorbachev, tear down this wall!” 3. Data is Changing 6 Internal Structured Data External Structured Data Internal Unstructured Data External Unstructured Data Sales Analysis, Financial Reports, Key Performance Indicators, Inventory Analysis, ……….. Sentiment Analysis, Customer Churn, License Plate Tracking, Reputation Analysis, Brand Monitoring, …….. Big Data Market Share, Share of Wallet, Credit Worthiness, Background Check, Competitive Pricing, …….. Call Center Training, Medical Text Mining , Product Research Mining, Legal Records Research, Fraud Detection, ……
  7. 7. Improved Decision Making 4. Questions are Maturing Increased Business Value Query Drilldown Alerts Adhoc Reports Standard Reports Predictive Modeling Optimization Forecasting Statistical Analysis What happened? How many, how often? Where exactly is the problem? What actions are needed? Why is this happening? What if these trends continue? What will happen next? What is the best that can happen?
  8. 8. 5. Methods are Changing Agility to grow and change is key • Facebook – There is no information on the exact count, but estimates from power user put their server counts at approximately: – 180,000 (Aug 2012) from… – 60,000 (June 2010) from… – 30,000 (Oct 2009) 8 Facebook • Google is estimated to be over a million servers. • Facebook now updates its code twice every day • Flickr – 10 releases per day • Instagram – 100M users, 5B images, 3 engineer and Amazon’s Elastic Cloud http://news.cnet.com/8301-1023_3-57486696-93/facebook-now-updates-its-code-twice-every-day Sources: CNET, Instagram Engineering, Flickr, Facebook, Mvdirona
  9. 9. 6. Architectures are Expanding Marketing Management Marketing Operations Customer Experience E-commerce Social Networks Mobile, SMS, .. Surveys Mail, Billing Phone email In Store Media KIOSK PURL, QR POS 3rd Party Website Mobile Website Voice of Customer Email engine Social Monitoring Couponing Local Marketing Lead Gen Call Center RoboCall More.. Print Media CRM Marketing Analytics Segmentation Trends Behavior Event Pattern Context Content Language Sentiment Customer Value Customer Churn Marketing Information People, Places, & Things Activity, Transactions, Etc. Customer Master Content Repository Product Catalog Promotion Catalog Sales Web logs Social Activity Emails, Calls, Texts, … Multi –channel Campaign Mgmt Performance & Financial Mgmt Optimization & Modeling Integrated Marketing Mgmt Big Data in Customer Communications
  10. 10. 7. Technologies are Evolving 10 Technology continues to reinvent itself.
  11. 11. 8. Terminology is Expanding HFT ACID V3 Infomediary Metadata Situational Awareness Quant CDO Authoritative Source Data Lifecycle NLP Latency YottaByte HiPPO 11
  12. 12. 9. Business Cultures are Transforming 12 Becoming “data aware” is a journey. Incubated in a series of projects and ending with a cultural transformation. Gartner BI Maturity Model Is the HiPPO going the way of the dinosaur?
  13. 13. 10. Errors occur in milliseconds • Which is better: faster or slower? – A “Twitter hoax” briefly erased $200 billion of value from the US Stock Market in April. • False reports of explosions in the White House triggered a set of algorithms monitoring news feeds into a two minute selling spree. • DOW drops 145 points. • Why? New technology can ‘read’ social media messages and place bets accordingly 13 What losses were incurred by algorithms reacting to a news feed and potentially other algorithms reacting to those algorithms???
  14. 14. 10. Errors occur in milliseconds -- continued -- • The cost of bad data exceeds $600B dollars for US businesses annually. • Almost, 50% of respondents cite data quality as the greatest barrier to adopting Business Intelligence. • Poor data quality will cost the UK’s 4 largest supermarkets $1B dollars over the next 5 yrs. • Poor data is cited as the number one reason for project overruns. • For a median Fortune 1000 company, a 10% increase in data usability would increase revenue by $2B. 14 http://www-new.insightsquared.com
  15. 15. 10. Errors Re-Occur over Days – continued-- 15 Amazon Sale Price $23.7M + 3.99 for Shipping Two sellers with two different pricing algorithms that automatically set prices based on competing prices Price of book rises to $23.7M over 10 days!
  16. 16. 11. Resource Demands are Escalating Sales, Payments, Orders, Transactions, … Email, SMS, Twitter, ….. YouTube, Instagram, Netflix, flickr, twitpic, Dailymotion, …. Skype, lingo, phonepower, ITP, phone.com,…. 16 A 2011 research report by Mckinsey Global Institute predicted that by 2018 , the US job market would experience a shortage of around 1.5M managers & analysts with the know-how to use analysis on big data. Volume! Velocity! Big Data
  17. 17. 12. IT Departments are Falling Behind Available Resources Run & Maintain Staff Time Questions from the CIO…….. • How do I meet the demands of the business for innovation? • How do I develop business subject matter experts that are adept at applying technology to business problems? • How do I train my employees on new technologies? By the way, which new technology(s)? • How do I reduce my support time? • How do I find, hire and retain top resources? The Time for Innovation is shrinking! 17
  18. 18. Summary • Big Data is Here – and has been for awhile • Big Data is not a “Technology Project” – Although there are many technology choices • Big Data does not solve every Problem – People do! (i.e., Data Quality) • Big Data is a Journey • Big Data is a Cultural Change 18
  19. 19. 19 John Bostick 513-702-3810 jbostick@lucruminc.com www.lucruminc.com
  20. 20. Mobile Usage is Growing • Global mobile data traffic grew 70 percent in 2012 • Mobile video traffic was 51 percent of traffic by the end of 2012 • Globally, 33%of total mobile data traffic was offloaded onto the fixed network in 2012. 20 • Mobile Data Traffic is expected to grow at a 66% CAGR from 2012 to 2017. • The number of mobile-connected devices will exceed the world's population in 2013. Source: Cisco Global Mobile Data Traffic Forecast Update, 2012–2017
  21. 21. Cloud Computing Growth Workloads per traditional server: – 2011 = 1.5 – 2016 = 2.0 Workloads per cloud server: – 2011 = 4.2 – 2016 = 8.5 21 Source: Cisco Global Cloud Index: Forecast and Methodology, 2011–2016 By 2016, nearly two-thirds of all workloads will be processed in the cloud.