Analytics Teams: 5 Things You Need to Know Before You Deploy Your Model

180 views

Published on

Get deployed! Many Analytics Teams have experience with building what seems like a great model–valid, predictive, powerful–only to see disappointing or even no business impact. Some models are not deployed, or take so long to deploy their accuracy is lost. Even deployed models are often not used effectively.
What can you do? Learn the 5 questions to ask before deploying your model.

Published in: Software
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
180
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Analytics Teams: 5 Things You Need to Know Before You Deploy Your Model

  1. 1. @jamet123 #decisionmgt © 2016 Decision Management Solutions James Taylor CEO Analytics Teams: 5 Things You Need to Know Before You Deploy Your Model
  2. 2. @jamet123 #decisionmgt © 2016 Decision Management Solutions 2 James Taylor Decision Management Solutions ▶ We work with clients to improve their business by applying business rules & analytic technology to automate & improve decisions. ▶ Vendor-neutral ▶ Original DMN submitter ▶ Using decision modeling since 2011 ▶ I have spent the last 14 years focused on the development of analytic applications using Decision Management
  3. 3. @jamet123 #decisionmgt © 2016 Decision Management Solutions 3 Agenda ▶ Challenges Deploying Analytics ▶ Five Questions To Ensure Deployment ▶ Decision Modeling for Analytics ▶ Decision Models Answer The Five Questions ▶ Wrap Up
  4. 4. @jamet123 #decisionmgt © 2016 Decision Management Solutions 4 Challenges Deploying Analytics
  5. 5. @jamet123 #decisionmgt © 2016 Decision Management Solutions 5 Data Science Insights Business Value There Is A Dangerous Gap In Data Science
  6. 6. @jamet123 #decisionmgt © 2016 Decision Management Solutions 6 Closing The Gap Delivers Analytic Success Data Science Insights Business Value Frame The Problem Build The Model Plan For Deployment
  7. 7. @jamet123 #decisionmgt © 2016 Decision Management Solutions 7 Challenges Deploying Analytics Analytic results are prioritized, not business resultsIneffective deployment leaves analytics on the shelf Timeliness and sustainability are not considered
  8. 8. @jamet123 #decisionmgt © 2016 Decision Management Solutions 8 Operational Systems Analytic Systems Analytics need to drive operations to add value 40%-60% of analytic models are not deployed and used Models On The Shelf Add No Value
  9. 9. @jamet123 #decisionmgt © 2016 Decision Management Solutions 9 Analytics Must be Deployed ? ? “A presentation ABOUT a model, however interesting, can’t power the website” “A 40 page report, full of math”
  10. 10. @jamet123 #decisionmgt © 2016 Decision Management Solutions 10 Business Results ▶ Too many projects focus on analytic accuracy ▶ Need to focus on business value ▶ Teams must connect their analytic to this value
  11. 11. @jamet123 #decisionmgt © 2016 Decision Management Solutions 11 Timeliness and Sustainability ▶ Time to deploy matters ▶ Deployed models age
  12. 12. @jamet123 #decisionmgt © 2016 Decision Management Solutions 12 Five Questions To Ensure Deployment 1. Does the model evaluate well against business objectives? 2. Who’s using the model, to decide what, where and how? 3. What will it take to make the model actionable? 4. What’s the plan for capturing and analyzing data about model effectiveness? 5. When do we need to revisit the model to make sure it is still effective?
  13. 13. @jamet123 #decisionmgt © 2016 Decision Management Solutions 13 Decision Modeling for Analytics
  14. 14. @jamet123 #decisionmgt © 2016 Decision Management Solutions 14 Decision Modeling Frames Analytics ▶ Provides “structure” - who, what, how, when ▶ Standardizes approach to decision making ▶ Provides “transparency” of decision approach ▶ Fosters innovation in the business model ▶ Improves/changes the business model ▶ Promotes “buy in”
  15. 15. @jamet123 #decisionmgt © 2016 Decision Management Solutions 15 Decision Models Frame The Problem Decision Input Data Knowledge Sources Sub- decisions Analytic Source: DecisionsFirst Modeler @jamet123 #decisionmgt
  16. 16. @jamet123 #decisionmgt © 2016 Decision Management Solutions 16 Decision Models Put Analytics In Context Process Decision Motivation Organization Department Team Role Role Team Department Organization Analytic
  17. 17. @jamet123 #decisionmgt © 2016 Decision Management Solutions 17 Decision Models Answer The Five Questions
  18. 18. @jamet123 #decisionmgt © 2016 Decision Management Solutions 18 Five Questions To Ensure Deployment 1. Does the model evaluate well against business objectives? 2. Who’s using the model, to decide what, where and how? 3. What will it take to make the model actionable? 4. What’s the plan for capturing and analyzing data about model effectiveness? 5. When do we need to revisit the model to make sure it is still effective?
  19. 19. @jamet123 #decisionmgt © 2016 Decision Management Solutions 19 1. Does the model evaluate well against business objectives? ▶ Objectives of the sponsors ▶ Measures of success ▶ For instance ▶ Reduce claims in litigation ▶ Improve the loss ratio ▶ Objectives of the modelers ▶ Measures of accuracy ▶ For instance ▶ Error rate ▶ Lift Business Measures Not Analytic Ones
  20. 20. @jamet123 #decisionmgt © 2016 Decision Management Solutions 20 Consider Impact On All Business Objectives Loan Repayment Insurance Revenue Customer Satisfaction
  21. 21. @jamet123 #decisionmgt © 2016 Decision Management Solutions 21 2. Who’s using the model, to decide what, where and how? ▶ What is the analytic used for? ▶ Who has to use the analytic model? ▶ Where is the analytic used? ▶ How is the analytic used? ▶ The analytic is used in an explicit decision ▶ Those that make the decision ▶ The decision has a business process context ▶ The other elements of the decision are clear
  22. 22. @jamet123 #decisionmgt © 2016 Decision Management Solutions 22 3. What will it take to make the model actionable? ▶ Decisions often involve analytics and business rules Business Rules Analytics
  23. 23. @jamet123 #decisionmgt © 2016 Decision Management Solutions 23 Take Action With Business Rules ▶ Adding business rules to analytic models turns predictions into actions ▶ Managing sets of business rules and analytic models in a single decision flow, a single execution, encapsulate decision-making ▶ Business user find business rules approachable, increasing their comfort with analytic decision- making ▶ Business rules give them some control over what to DO about the predictions you give them
  24. 24. @jamet123 #decisionmgt © 2016 Decision Management Solutions 24 4. What’s the plan for capturing and analyzing data about model effectiveness? Decision Made Supporting Decisions Scores Business Results ▶ Decision Models show what to store
  25. 25. @jamet123 #decisionmgt © 2016 Decision Management Solutions 25 5. When do we need to revisit the model to make sure it is still effective? Data Changes Direct Outcomes Indirect Outcomes Business Results ▶ Decisions models reveal tracking needs
  26. 26. @jamet123 #decisionmgt © 2016 Decision Management Solutions 26 Champion Challengers Champion/Challenger in Decision Models
  27. 27. @jamet123 #decisionmgt © 2016 Decision Management Solutions 27 Five Questions To Manage Deployment 1. Does the model evaluate well against business objectives? 2. Who’s using the model, to decide what, where and how? 3. What will it take to make the model actionable? 4. What’s the plan for capturing and analyzing data about model effectiveness? 5. When do we need to revisit the model to make sure it is still effective? ▶ Decision models link analytics to decisions to business metrics ▶ Decision models show who owns, makes and cares about decisions ▶ Decision models wrap analytics in rules-based decisions ▶ Decision outcomes can be captured with input data, business results and scores ▶ Decision models clarify business context, pace of change and show data usage clearly
  28. 28. @jamet123 #decisionmgt © 2016 Decision Management Solutions 28 Getting Started
  29. 29. @jamet123 #decisionmgt © 2016 Decision Management Solutions 29 Decision Most Analytic Projects Leave Decisions In The Dark Modeling Decisions Brings Transparency 29©2015 Decision Management Solutions
  30. 30. @jamet123 #decisionmgt © 2016 Decision Management Solutions 30 Data Business Understanding Data Understanding Modeling Data Preparation Evaluation Deployment CRISP-DM Phases 1. Business Understanding 2. Data Understanding 3. Data Preparation 4. Modeling 5. Evaluation 6. Deployment
  31. 31. @jamet123 #decisionmgt © 2016 Decision Management Solutions 31 Specific Analytic Decision Analytic Models Overall decision to be improved Business Understanding As a Decision Model Regulations and Policies Data
  32. 32. @jamet123 #decisionmgt © 2016 Decision Management Solutions 32 Decision Models Ensure Analytic Deployment Tie the analytic to decisions and decisions to business results Specify the decision(s) that use the analytic Use the decision context to frame deployment and sustainability needs
  33. 33. @jamet123 #decisionmgt © 2016 Decision Management Solutions 33 Decision Modeling For Analytic Projects ▶ Steps 1. Training in decision modeling 2. Establish business value 3. Model effective decision-making 4. Add process, organization, business context 5. Frame and scope the analytic project ▶ Deliverables ▶ Decision Model showing impact of proposed analytic ▶ Analytic, business and IT teams trained & self-sufficient ▶ Best practices and go-forward recommendations ▶ Software, consulting and training
  34. 34. @jamet123 #decisionmgt © 2016 Decision Management Solutions 34 Decision Modeling for Analytic Projects ▶ Learn more ▶ Download the five questions brief ▶ bit.ly/5QAnalytics ▶ Ready to start? ▶ Kick off a decision modeling project today ▶ bit.ly/DMNAnalytics
  35. 35. @jamet123 #decisionmgt © 2016 Decision Management Solutions 35 Questions?
  36. 36. @jamet123 #decisionmgt © 2016 Decision Management Solutions 36 Decision Modeling for Analytic Projects ▶ Learn more ▶ Download the five questions brief ▶ bit.ly/5QAnalytics ▶ Ready to start? ▶ Kick off a decision modeling project today ▶ bit.ly/DMNAnalytics
  37. 37. @jamet123 #decisionmgt © 2016 Decision Management Solutions 37 DecisionsFirst Modeler Collaborative, cloud-based decision modeling software that puts decisions at the heart of your business architecture, building better requirements and driving successful business rules implementations Key Features Multi-user, Collaborative Decision Model and Notation (DMN) Support Shared Database, Reusable components Automated Consistency and Completeness Checks Living Repository
  38. 38. Thank You @jamet123 #decisionmgt © 2016 Decision Management Solutions For more on Decision Management, go to: decisionmanagementsolutions.com James Taylor james@decisionmanagementsolutions.com

×