Piloting Big Data: Where To Start? - StampedeCon 2014

1. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience Piloting Big Data Where to Start? 29 May 2014 – StampedeCon – St. Louis John Akred (@BigDataAnalysis), www.svds.com @SVDataScience

2. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience 2 Solving diﬃcult problems with technology, data, and science Cross-‐func<onal teams Agile delivery methods Business-‐driven technology strategy and advisory

3. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience 3 1 Why big data? 2 What is a pilot? 3 Choosing a use case 4 Deﬁning success Doing a Big Data Pilot Fielding a team Delivering

9. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience 9 UP OR OUT? The SaaS Edition Users Revenue scale-out cost good times bummer Different products and features put different demands on the data infrastructure •  Profitable •  Unprofitable Increasing cost per user from scale-up architectures causes a barrier to economic expansion of the product user base.

10. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience 10 UP OR OUT? in the enterprise Diﬀerent use cases put diﬀerent demands on the data infrastructure •  UC1 •  UC2 •  UC3 •  UC4 •  UCn Increasing cost per unit of capability from scale-up architectures causes rationing of resources. Only the most valuable use cases are pursued. Data Resource Usage Value scale-out cost UC 1 UC2 UC3 UC4 UCn

12. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience 12 From idea to production Agile: Iterate to value, answering the most valuable questions as quickly as possible Plan Prototype Pilot Production þ þ þ þ þ

13. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience What is a Pilot? Plan and Initialize • Define architectural approach • Identify resources • Provision training • Choose use case • Define success • Populate initial backlog and sprint plans Prototype and Prove • Identify poorly understood functionality • Isolate and experiment • Determine solution approaches • Evaluate solution(s) • Correctness • Scale • Economics Pilot • Define end-to- end “steel thread” • Partition off pilot population • Build and integrate system components • Modify associated processes • Train pilot user team Production • Expand to entire user/ customer/ partner/ etc population • Industrialize monitoring capabilities • Re-engineer processes • Train user community 13

15. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience 15 STRATEGIC IMPERATIVES BUSINESS OBJECTIVES MAP OBJECTIVES TO TECHNICAL WORKLOADS RATIONALIZE WORKLOADS Strategic Workloads

16. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience 16 BUILDING A DATA PLATFORM External Systems Data Acquisition Internal Data Sources Data Management Security, Operations, Data Quality, Meta Data Management and Data Lineage Analytics Data Ingestion Data Repository External Data Sources Persistence Oﬄine Processing Real Time Processing Batch Processing Data Services

18. © 2014 Silicon Valley Data Science LLC All Rights Reserved. www.svds.com @SVDataScience •  Incremental revenue •  Time to market •  Economic functional implementation •  Cost avoidance •  Brand beneﬁt •  Goodwill ✔ 18 Deﬁning Success