You've built a great model — all you need are the approvals. Will the business trust your model? Will they appreciate what it can and can’t do? Will they invest time and resources for development? Expert witness and Pentagon consultant John Miller shares what he’s learned about transparency and reproducibility, helping the audience “get” machine learning, and providing the right decision support. You’ll leave with practical techniques to help you get the go-ahead to move forward.

1. GETTING THE GREEN LIGHT
FOR YOUR MACHINE LEARNING MODEL
John Miller, Benbrook Analytics
@JohnMillerTX

2. 2

3. 3
Consider your audience.
Embrace transparency.
Make tough decisions easy.
To win the race…

4. Consider your Audience
o Focus on ‘Why’
o Relate at the right level

5. – Start-up CEO, to me
You have a great story – a simple story –
and you’re overcomplicating it.“
”

6. Focus on ‘Why’.
6
Alittle bit of “Why”
Lots andlotsof“What”
Passes & Runs Punts Risk Multiple
2016 42 61 1.4
2017 47 56 1.2
UNDERSTANDING THE PROBLEM
My first activity was to explore the problem presented by the NFL. I researched information on football and sports
safety to improve my understanding. Resources included the following items:
- SPORTS ARTICLES ON PUNT PLAYS AND PLAYERS - NFL OFFICIAL RULES
- VIDEOS AND GAME HIGHLIGHTS - NFL PRESS RELEASES, ARTICLES AND GENERAL STATISTICS
- INFORMATION FROM OTHER SPORTS - ARTICLES AND RESEARCH PAPERS ON SPORTS-RELATED CONCUSSIONS
I also needed to get an independent idea of concussion risk for punt plays relative to other plays. I analyzed data
provided for the competition along with data scraped from nfl.com and concussion data compiled by IQVIA. The table
below shows the number of concussions per 1000 passing and running plays vs. concussions per 1000 punt plays.
According to these approximate calculations, punt plays were 1.2 to 1.4 times more dangerous than pass and run plays
in 2016 and 2017. These numbers include all punt plays to include out of bounds punts and down punts.

7. 7
Relate at the right level.
1
1
1
1
1
2
5
5
20
0 5 10 15 20
Punter
Fullback
Protector
Linebackers
Jammers
Defensive Linemen
Gunners
Returner
Offensive Linemen

8. Embrace Transparency
o Design “brain-friendly” models
o Use open source tools

9. I can't comment on that. I don't know how to
compare who is more knowledgeable.“
”– Opposing expert on who knew more about t-SNE:
him, or Hinton and van der Maaten

10. Design “brain-friendly” models.
10
TOTAL SALES
MONTHLY SALES CHANGE
DAILY SALES VARIANCE
AVERAGE SALE AMOUNT
OUT-OF-COUNTY SALES
REPEAT CUSTOMER SALES
Known
Suspects
ModelFeatures

11. 11
Use open source tools.
Comprehension Notebook = Clarity * Readability

12. Make Tough Decisions Easy
o Provide critical information
o Align with existing priorities

13. – US Army Operations Executive, to me.
We’re going to need something more than
PowerPoint-deep [analysis].“
”

14. Sample CCIRs for
Machine Learning
14
Provide critical information.
Source: CCIR Insights and Best Practices Focus Paper, 3rd ed.
Military Commander’s Critical Information Requirements (CCIR)
Business value resulting from
model deployment
Deployment costs in terms
of money, time, opportunity
Recurring costs for support
and monitoring
Anticipated impact on
customer behavior
Data quality now and over
the model’s lifecycle

15. 15
Align with existing priorities.
CEO’s Top 3 Business Priorities (Gartner Survey, Apr 2017)

16. Thank you.
Questions?
@JohnMillerTX