Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

7 Dimensions of Agile Analytics by Ken Collier

5,575 views

Published on

We are in the midst of an exciting time. There is an explosion of very interesting data, and emergence of powerful new technologies for harnessing data, and devices that enable humans to receive tremendous benefits from it. What is required are innovative processes that enable the creation and delivery of value from all of that data. More often than not, it is the predictive (what will happen?) and prescriptive (how to make it happen!) analytics that produces this value, not the raw data itself.  Agile software teams are continuously involved in projects that involve rich, complex, and messy data. Often this data represents innovative analytics opportunities. Being analytics-aware gives these teams the opportunity to collaborate with stakeholders to innovate by creating additional value from the data. This session is aimed at making Agile software teams more analytics-aware so that they will recognize these innovation opportunities. The trouble with conventional analytics (like conventional software development) is that it involves long, phased, sequential steps that take too long and fail to deliver actionable results. This deck will examine the convergence of the following elements of an exciting emerging field called Agile Analytics:
sophisticated analytics techniques, plus
lean learning principles, plus
agile delivery methods, plus
so-called "big data" technologies
Learn:
The analytical modeling process and techniques
How analytical models are deployed using modern technologies
The complexities of data discovery, harvesting, and preparation
How to apply agile techniques to shorten the analytics development cycle
How to apply lean learning principles to develop actionable and valuable analytics.

Published in: Technology

7 Dimensions of Agile Analytics by Ken Collier

  1. 1. 7 DIMENSIONS OF AGILE ANALYTICS Ken Collier, PhD Director, Agile Analytics @theagilist #thoughtworks 1
  2. 2. BusinessValue Analytical Complexity What happened? Descriptive Analytics Why did it happen? Diagnostic Analytics What will happen? Predictive Analytics Can we influence what happens? Prescriptive Analytics
  3. 3. BusinessValue Analytical Complexity What happened? Descriptive Analytics Why did it happen? Diagnostic Analytics What will happen? Predictive Analytics How can we make it happen? Prescriptive Analytics 3 Business Intelligence Data Science
  4. 4. THE DIFFERENCE Data Engineering Lean Learning Streaming data pipelines & adaptive architectures Continuously challenge your assumptions by measuring results. Discovery of patterns and signals hidden in data Agile Delivery Data Science Deliver business value early and often. Build your platform over time, not all up front. Your Business Questions = Fast results & Early Value Data Guided Market Advantage
  5. 5. Agile Analytics Data Engineering Solutions Thinking Ethics Agile DeliveryLean Learning Impact Advanced Analytics
  6. 6. Agile Analytics Data Engineering Solutions Thinking Ethics Agile Delivery Lean Learning Impact Advanced Analytics Adaptive Architecture Streaming Data Polyglot Persistence
  7. 7. Strictly Private and Confidential © 2015 ThoughtWorks, Ltd. All rights reserved. REMOVING THE DATA BOTTLE NECK Data Warehouse Incoming data is cleaned and organised into a single schema up front. Data Lake Incoming data goes into the lake in its raw form. Long development times to create new value from data. Analysis activities are distributed across technologists and business users. Analysis is done directly on the curated warehouse data. Data is selected, structured, and organize as needed, when needed.
  8. 8. DATA LAKE DONE RIGHT 8 Operational systems communicate directly with each other via services Operational systems push data to the lake via topical queues Data scientists explore the lake for potential insights Lakeshore marts and services curate and organize the data for self-service analysis Multi-tiered data lake for processing, distribution, serving
  9. 9. ADAPTIVE ARCHITECTURE PRINCIPLES 9 Enable low latency data streaming Store raw, low-level, historized data Enable NoSQL presentation Enable inexpensive scaling Simplify data ingestion Drive logic closer to the business Enable emergent design Enable easy recreation of data
  10. 10. DATA ENGINEERING CONCERNS 10 Streaming Distributed MPP architecture Data StrategyElastic cloud computing Reactive architecture Master data Data Governance ETL techniques
  11. 11. Advanced Analytics Agile Analytics Solutions Thinking Ethics Agile Delivery Lean Learning Impact Data Engineering Adaptive Architecture Streaming Data Polyglot Persistence Data Science Machine Learning Statistics
  12. 12. Discover & Explore Analyze & Act Data Convergence Analytical Divergence Discover Harvest Filter Integrate Augment Analyze Act Analytical Opportunities HOW DATA SCIENCE WORKS Can we anticipate what the customer will want to do next?
  13. 13. THE “DATA SCIENTIST” Machine Learning Statistical Modeling Artificial Neural Networks Decision Tree Learning Support Vector Machines Unsupervised Learning …and many more… Bayesian Classification Monte Carlo Simulation Logistic Regression K-Nearest Neighbor …and many more… Feature Engineering Feature Extraction Dimension Reduction Domain expertise Programming Skills Functional Programming Data “Wrangling” Map/Reduce, SQL, & NoSQL
  14. 14. Objective Truth Discoverable Truth Uninterpretable Irrelevant Noise Not Actionable Actionable Signals MAKING “BIG DATA” INTO “LITTLE DATA”
  15. 15. Advanced Analytics Data Science Visual Storytelling Machine Learning Statistics Agile Analytics Solutions Thinking Ethics Agile Delivery Lean Learning Impact Data Engineering Volume Velocity Variety Adaptive Architecture Streaming Data Polyglot Persistence
  16. 16. Advanced Analytics Data Science Visual Storytelling Machine Learning Statistics Agile Analytics Solutions Thinking Ethics Agile DeliveryLean Learning Impact Continuous Integration Collaboration Evolve Continuous Delivery Hypothesis Build Learn Measure Data Reduction Data Engineering Volume Velocity Variety Adaptive Architecture Streaming Data Polyglot Persistence
  17. 17. drones.pitchinteractive.com Data Visualization
  18. 18. drones.pitchinteractive.com
  19. 19. Advanced Analytics Agile Analytics Solutions Thinking Ethics Agile Delivery Lean Learning Impact Hypothesis Build Learn Measure Data Engineering Adaptive Architecture Streaming Data Polyglot Persistence Data Science Machine Learning Statistics Visual Storytelling
  20. 20. Typical Timeline 3-6 months 1-2 months 2-4 months 22 Data Convergence Analytical Divergence Discover Harvest Filter Integrate Augment Analyze Act Analytical Opportunities CONVENTIONAL DATA SCIENCE If we knew X, we could do Y
  21. 21. Analytical Divergence Analytical Opportunities If we knew X, we could do Y Data Convergence Discover Harvest Filter Integrate Augment Analyze Act Repeat this cycle solving small problems every few days LEARN MEASURE BUILD LEAN DATA SCIENCE
  22. 22. Advanced Analytics Agile Analytics Solutions Thinking Ethics Agile Delivery Impact Reflect & Improve Collaborate Evolve Deliver Data Engineering Adaptive Architecture Streaming Data Polyglot Persistence Data Science Machine Learning Statistics Visual Storytelling Lean Learning Hypothesis Build Learn Measure
  23. 23. Retain high value customers Problem solved or continue? High value business goal What’s the smallest, simplest thing we can do? Is it useful & actionable? Repeat! What leads to customers leaving? LIKE THIS EXAMPLE… Common features of defectors? Shopping behaviors of defectors? What do defectors say about us? Customers’ sentiment before defecting? What encourages customers to stay? Do incentives reduce defection rates?
  24. 24. Retain high value customers High value business goal Like this example…
  25. 25. What’s the smallest, simplest thing we can do? Retain high value customers Like this example… Common features of defectors?
  26. 26. Is it useful & actionable? Retain high value customers Like this example… Common features of defectors?
  27. 27. Repeat!Retain high value customers Like this example… Common features of defectors? Shopping behaviors of defectors?
  28. 28. Retain high value customers Like this example… Common features of defectors? What leads to customers leaving? Shopping behaviors of defectors? What do defectors say about us? Customers’ sentiment before defecting? What encourages customers to stay? Do incentives reduce defection rates?
  29. 29. Problem solved or continue? What leads to customers leaving? Like this example… Common features of defectors? Shopping behaviors of defectors? What do defectors say about us? Customers’ sentiment before defecting? What encourages customers to stay? Do incentives reduce defection rates?
  30. 30. Advanced Analytics Data Science Visual Storytelling Machine Learning Statistics Agile Analytics Solutions Thinking Ethics Agile DeliveryLean Learning Impact Hypothesis Build Learn Measure Insight Knowledge Action Disruption Data Engineering Volume Velocity Variety Adaptive Architecture Streaming Data Polyglot Persistence Reflect & Improve Collaborate Evolve Deliver
  31. 31. Advanced Analytics Data Science Visual Storytelling Machine Learning Statistics Agile Analytics Solutions Thinking Ethics Agile DeliveryLean Learning Impact Hypothesis Build Learn Measure Insight Knowledge Action Disruption Business First Evolve the Platform Monitor & Measure Data Engineering Adaptive Architecture Streaming Data Polyglot Persistence Reflect & Improve Collaborate Evolve Deliver
  32. 32. QUESTIONS FIRST, DATA SECOND 34 “We built a platform for self service, and now we’re trying to get the business to use it.” From this… …to this
  33. 33. Advanced Analytics Data Science Visual Storytelling Machine Learning Statistics Agile Analytics Solutions Thinking Ethics Agile DeliveryLean Learning Impact Hypothesis Build Learn Measure Insight Knowledge Action Disruption Business First Evolve the Platform Monitor & Measure Privacy Controls Radical Transparency Data Democracy Open Data Data Engineering Volume Velocity Variety Adaptive Architecture Streaming Data Polyglot Persistence Reflect & Improve Collaborate Evolve Deliver
  34. 34. 38 http://bit.ly/DataAndPrivacy Jonny Leroy, NA Head of Technology, ThoughtWorks
  35. 35. 39
  36. 36. 40
  37. 37. 41
  38. 38. 42
  39. 39. 43
  40. 40. 44
  41. 41. 45
  42. 42. 46 PRIVACY IS DEAD DON’T BE EVIL
  43. 43. 47
  44. 44. 48
  45. 45. 49
  46. 46. 50
  47. 47. 51
  48. 48. 52
  49. 49. Advanced Analytics Data Science Visual Storytelling Machine Learning Statistics Agile Analytics Solutions Thinking Ethics Agile DeliveryLean Learning Impact Hypothesis Build Learn Measure Insight Knowledge Action Disruption Business First Evolve the Platform Monitor & Measure Privacy Controls Radical Transparency Data Democracy Open Data Data Engineering Volume Velocity Variety Adaptive Architecture Streaming Data Polyglot Persistence Reflect & Improve Collaborate Evolve Deliver
  50. 50. Ken Collier, Director, Agile Analytics kcollier@thoughtworks.com Value Creation Cool New Technologies + Sophisticated Analytics + Lean Learning Principals + Fast Agile Delivery =

×