Big Data Meetup by Chad Richeson


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Big Data Meetup by Chad Richeson

  1. 1. Big Data MeetupChad RichesonCEO, Society Technical ForumJune 19, 2013
  2. 2. Bio:• 17 Years analyzing Businesses, Products, & Customers• 7+ Years building large analytical & technical teams• 7+ Years above Petabyte scale• 12 Years at MSFTMy approach:• Business decisions drive the technical decisions• Focus on outcomes first, processes second• Blend technical, analytical, & business skillsAbout MeJune 19, 2013 2
  3. 3. Big Data’s Impact on IndustryNearly every industry is trying to figure out how to apply Big Data concepts to theirbusiness, to uncover new opportunities, improve efficiencies, and minimize risk.June 19, 2013 3Digital Media & E-Commerce Real-time ad targeting, Web analytics & trendsEnergy and Utilities Smart meter analytics, Asset managementFinancial Services Risk and fraud management, Portfolio management, Customer analyticsGovernment Threat Management, Law Enforcement (Real-time multimodal surveillance,Cyber security detection), Macro economic analyticsHealthcare and Life Sciences New drug development, Medical record text analytics, Genomic analyticsRetail CRM, Targeted marketing analysis, Vendor delivery & Supply chainoptimizations, Market basket analysis, Click-stream analysisTelecommunications CRM, Call detail record analysis, Least cost routing, Fraud managementTransportation Logistics optimization, Traffic congestion
  4. 4. But can we harness it all?It’s no longer a matter of whether we can collect large amounts of data, it’s whether we canharness its power.June 19, 2013 4• Every day approx 2.5 quintillion (2.5×10^18) bytes ofdata is created.• Mobile devices, web tags, smart energy meters, remotesensing, wireless sensors, software machinelogs, cameras, rfid readers, etc. are creating massiveamounts of data• The economic potential of big data is becoming a “C-level” conversation.
  5. 5. Smartphone DataOne driver of data explosion is the smartphone. Today’s smartphone has 14 sensors andgrowing. And each phone = one person, which has profound implications on the amount ofmeaning that can be derived from the data.June 19, 2013 5AccelerometerGyroscopeMagnetometerBarometerProximityLight SensorTouch ScreenGPSWiFiBluetoothGSM/CDMA CellNFC: Near FieldCamera (front)Camera (back)
  6. 6. Web DataAnother driver of data explosion is the amount of data that can be collected from a webpage. Data growth from the web shows no signs of slowing down.June 19, 2013 6• Consider 10 million page views a day on a popularweb site:• Capture User ID for every page view and store them asinteger• 10 million x 4 bytes = 40 MB of storage/day• 40MB x 30 days = 1.17 GB/month, just for User ID• Data quickly grows and so does challenges aroundstorage, processing and analytics.
  7. 7. eCommerce ExperimentationNew techniques such as Experimentation are creating dramatically more data to analyze.Consider a typical eCommerce site (AT&T’s wireless site was chosen here as an example.)June 19, 2013 7
  8. 8. eCommerce ExperimentationFirst, the number of elements that can be varied and the number of variants per element isnearly limitless.June 19, 2013 81- Varyphotos5-Varyoffer2-Varyphotos7-Vary label4-Vary text9-Vary copy6-Varycopy10-Vary label8-Varytext3-Varyphotos
  9. 9. Experimentation DataThen, to properly analyze the impact of an experiment, an analyst must add contextual andconfounding variables to the analysis. Even the simplest version of this analysis contains 500variants that would need to be analyzed. Derived variables would add to this tally.June 19, 2013 9Contextual Variables:- Customer Profile- Time of Year- Time of Data- Customer Location- Customer Device- Referring URLConfounding Variables:- Page load time- Competitive Offers- Other Experiments- Multiple Tabs- User’s Other DistractionsThe Simplest Example:- one treatment group- one control group10 Experiments * 2 = 20 Experimentvariables.20 Experiment variables * 5contextual variables = 100controllable variables.100 controllable variables * 5confounding variables = 500variants to be analyzed.
  10. 10. Creating Successful Big Data ProjectsSuccessful Big Data projects blend Analytics Skills, Business Skills, and Technical Skills in theright proportions.June 19, 2013 10BusinessStrategyTechnicalSkillAnalyticsSkill
  11. 11. The Key StepsWhen moving a Big Data project from the Lab to the Mainstream, the following steps cangreatly increase the chances of success. Don’t neglect steps 1 and 2!June 19, 2013 111. Pick Focus Areas based on Business Strategy2. Gain Agreement From Target Users3. Build The Solution (Start Simple)4. Analyze & Iterate5. Expand, Repeat
  12. 12. A Single View of the CustomerDon’t bite off too much too soon. Creating a single view of the customer is an admirablegoal, but starting with fewer touchpoints and building out the remainder over time isusually a better choice. Use goals such as these to set overall vision & direction.June 19, 2013 12TouchpointsData “Fabric”SEMWeb SiteEmailDisplayAdsMobileCustomerPartnersContact PointsNot Exhaustive• Customer At theCenter• Data as theCommon LanguageBetween Systems• Speed ofCommunicationBetween SystemsMatters• Online & OfflineAnalytics
  13. 13. Predictive AnalyticsPredictive Analytics is another tantalizing concept that takes significant skill and capabilityto achieve. Build up to an advanced concept like Predictive Analytics, don’t start with it.June 19, 2013 13“Every Customer A Segment”3 - Prediction• Gather & monitor customercontext• Evaluate explicit customersignals in the correct context• Generate predictions of whatthe customer is most likely toneed or do next• Rapidly test and iterate• Apply learning from eachcustomer to next customer innear-realtimeLEAP“Cast Different Nets”2 - SegmentationSTEP“Spray & Pray”1 - Mass Market• Analyze historical customer datato determine segments• Create a strategy for eachsegment, and goals to movecustomers between segments• Generate different messagingfor each segment• Review performance and re-craft messages, or re-segmentcustomer base• Perform market research fromsample of customers todetermine needs• Create marketing messagingthat addresses the mostcommon needs• Review performance and adjustmessaging, in context of newestresearchMany digital marketing organizations are seeking to movebeyond segmentation to develop a more personalized, predictiverelationship with each customer. Getting to this advanced stageof digital marketing represents a leap forward in terms ofcompetitiveness, but also in terms of capabilities needed.“1x ROI”“3x ROI”“10x ROI”
  14. 14. Picking Focus Areas – Key QuestionsPick your early focus areas based on what is achievable. Early wins will generateexcitement, momentum, and more funding to sustain and grow your efforts.June 19, 2013 14Strategy Questions:• Which solutions would have the most impact on the business?• Which solutions are quickest & easiest to implement?• What role(s) does the business need to play?Technical Questions:• From which systems will I need data?• Is the data clean, accessible, and timely?• How much will it cost (HW, SW, people, 3rd party data) to build the solution?• Do we have the skills to build it?• How long will it take to build?Analytics Questions:• What are the criteria to evaluate the success of the solution?• Do we have the tools to manage & analyze the data?• Do we have the skills to analyze the data correctly?
  15. 15. A Focused Business QuestionLet’s take an example of trying to connect two customer touchpoints: an eCommerce website, and a Customer Support center. The business goal is to improve the customerexperience and save costs.June 19, 2013 15Solution Goal: Determine Which Changes to theeCommerce Experience impact Customer Support CostsKey Questions:• Which changes to the purchase funnel increase the proportion of sales madeonline vs. via phone?• Do changes to the purchase funnel reduce the amount of phone time required tomake a purchase?• Do changes to eCommerce help content reduce the number of phone calls?
  16. 16. Capabilities NeededTo create this solution, a number of important capabilities need to be in place.June 19, 2013 16• Ability to identify a customer across both channels• Ability to collect, store, and manage the data.• Ability to connect key data points to the Customer ID (i.e. lotsof processing.)• Ability to analyze the data effectively (tools such as AdobeInsight, SAS, and Comscore Digital Analytix.)• Ability to gauge customer impact over time and in context (thisis hard, often requiring advanced statistical skill.)
  17. 17. Key StepsOnce the capabilities are in place, perform the analysis and operationalize the changes.This is a repeatable process that can be run at any scale.June 19, 2013 17• Bring the data together (pour some Hadoop on it!)• Find the success clusters, then the success factors.• Be your own worst critic. Analyze competing explanations.• Get another analyst to serve as a second set of eyes. Greatanalysis survives all scrutiny.• EXPERIMENT before rolling changes out to all customers.Business must be bought into this, or the analysis will likely siton the shelf.• With the momentum you gain, repeat for other businessquestions.
  18. 18. In Summary• Treat Big Data projects as a combination ofBusiness, Technical, & Analytics skills.• Gain up-front agreement from the users of what youare building.• Keep focused. Build up wins, then expand.• Don’t be afraid to fail early, but set expectationsappropriately.• Network with your peers – keep learning. Big Data isan exercise in learning.June 19, 2013 18