Machine learning has evolved past an esoteric technique worked on by academics and research institutes into a viable technology being deployed at many companies. Machine learning has been significantly changing the competitive landscape of business models worldwide, contributing to the demise of established business, such as Blockbuster, to creating entirely new businesses, such as algorithmic advertising. This presentation strives to address the questions of what assurance professionals need to know about this technology and how to provide assurance around machine learning implementations and its unique risks.

1. Machine Learning: What
Assurance professionals need to
know
Andrew Clark, Principal, Capital One

2. About me
● B.S. in Business Administration with a concentration in Accounting, Summa Cum
Laude, from the University of Tennessee at Chattanooga.
● M.S. in Data Science from Southern Methodist University
● Ph.D. student in Economics with a concentration in International Monetary
Policy at the University of Reading.
● American Statistical Association Graduate Statistician (GStat), INFORMS
Certified Analytics Professional (CAP), and AWS Certified Solutions Architect -
Associate.
● Has designed, built, and deployed numerous machine learning and continuous
auditing solutions using open source technologies.
● Avid conference presenter and writer on the topic of machine learning

3. Overview
● What is machine learning?
● Why is it important?
● What do all of the buzzwords mean?
● What are the two broad types of machine learning?
● Non-technical introduction
● How does it pertain to auditors?
● Security Issues
● Case studies
● What would a machine learning audit entail?
● Where can I learn more about machine learning?
Kong, Qingkai . "Machine Learning 1 - What is
machine learning and real world example."
Qingkai's Blog (web log), October 4, 2016.
Accessed September 3, 2018.
http://qingkaikong.blogspot.com/2016/10/machin
e-learning-1-what-is-machine.html?showComme
nt=1484689212391#c4748865641151946089.

4. Learning Objectives
● Recall the different types of machine learning
● Understand how machine learning works at a high level
● Explain where machine learning can be applied
● Define common machine learning terminology

5. What is Machine
Learning?
A computer recognizing patterns
without having to be explicitly
programed.

6. What many businesses get wrong

7. Why is Machine Learning Important?
● Disrupting business. Example ML powered businesses disrupted Blockbuster,
Taxis, etc.
● Revolutionizing existing business models. Predictive maintenance in
manufacturing, retailing, credit card fraud detection, loan underwriting
● One of the key technologies in driving economic growth

8. What Machine Learning is not
● Magic
● Not going to take your job (for the majority of professionals)
● Always the best tool for the job

9. What do all these buzzwords mean?
“Machine Learning based, artificial intelligent, Big Data spewing, Deep Learning,
Neural Network touting, Cognitive Computing, Virtual Reality Natural Language
Processing,…Chat Bot.”

10. Two broad types of machine learning
● Supervised
● Unsupervised

11. Supervised Learning
● Given a labeled dataset, ‘fraud not fraud’, the algorithm is ‘trained’, to recognize
which items are fraud and which items are not fraud.
● Examples:
○ Transaction fraud detection
○ Classifying images: dog/not dog
● Common techniques include:
○ Logistic Regression
○ Support Vector Machines

14. Unsupervised Learning
● Given some cleaned data, the algorithm, divides the data into like groups.
● Examples:
○ Pattern recognition
○ Anomaly detection
○ Clustering
● Popular models:
○ Kmeans
○ Gaussian mixture models
○ DBSCAN

16. A non-technical introduction
● Process, when strung together, called a
pipeline
○ Business Understanding
○ Data Understanding
○ Data Preparation
○ Modeling
○ Evaluation
○ Deployment
Kearn, Martin . "Machine Learning is for Muggles too!" Microsoft Developer
(web log), March 1, 2016. Accessed February 21, 2017.
https://blogs.msdn.microsoft.com/martinkearn/2016/03/01/machine-learning-is-f
or-muggles-too/.

17. Business Understanding
● The most important step – ‘The Why’
● Why is this needed and what is the desired outcome?

18. Data understanding
● An understanding of where the data is coming from is key to good modeling
● SQL relational database? NoSQL database? Csv, txt, webpage, Tweets?
● What scale is the data on? For example, Celsius or Fahrenheit?

19. Data Preparation
● Currently, close to 90% of what Data Scientists do
● ‘Munging’
● Data scaling
● Select variables
● Divide into test and train sets
● “I’m a data janitor. That’s the sexiest job of the 21st century. It’s very
flattering, but it’s also a little baffling.” – Josh Wills, Head of Data
Engineering @ Slack
Press, Gil. "Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says." Forbes. March 23, 2016. Accessed March 13,
2017.
https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#21e789136f6
3.

20. Modeling

21. Evaluation
● Accuracy
● Precision
● Recall
● Does the model solve the problem?

22. Deployment
● Integrated into existing infrastructure or application?
● Separate web application?
● Scheduled job?
● Run adhoc?

23. How Machine Learning
Works
The animal version

30. As an auditor, what does this mean for you?
● New opportunities and risks
● Machine Learning control frameworks
● Catch-22 of businesses accepting the risk of black boxes or becoming irrelevant
● Use cases in audit analytics
● More complicated environment, new skills required to understand business
implications and audit algorithms

31. Machine Learning Security issues
Mikhailov, Emil, and Roman Trusov. "How Adversarial Attacks Work." Y Combinator. November 02,
2017. Accessed January 17, 2018.
http://blog.ycombinator.com/how-adversarial-attacks-work/?imm_mid=0f81cc&cmp=em-data-na-na-n
ewsltr_20171115.

32. Machine Learning Security issues cont.
Mikhailov, Emil, and Roman Trusov. "How Adversarial Attacks Work." Y Combinator. November 02,
2017. Accessed January 17, 2018.
http://blog.ycombinator.com/how-adversarial-attacks-work/?imm_mid=0f81cc&cmp=em-data-na-na-n
ewsltr_20171115.

33. Use cases in Assurance and Compliance
● Anomaly detection
○ Unsupervised journal entry anomaly detection
○ Clustering on invoice and AP data for outliers
○ Outlier user access
● ‘Auditor sense’ investigation
○ Supervised model for expense report investigation
○ Supervised model for journal entries
○ AP transactions, customer transactions, etc.
● Document review

34. The Machine Learning Algorithm Audit
● With algorithms increasingly dictating our lives, how do we know that they are
operating as intended?
○ Weapons of Math Destruction by Cathy O'Neil
● Unfilled role for assurance professionals.
● Review assumptions, and when available, look at the weighting for features in the
model.
○ Decision tree, logistic regression, etc.
● Can provide a lot of value when using only SDLC audit methodologies
● ISACA Paper

35. Where can I learn more about Machine Learning?
● Visual Intro, highly recommended, short and sweet
○ http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
● Wikipedia
○ https://en.wikipedia.org/wiki/Machine_learning
● Good beginning article with some fantastic books
○ http://machinelearningmastery.com/4-steps-to-get-started-in-machine-learning/
● Weka
○ http://www.cs.waikato.ac.nz/ml/weka/
● Scikit-Learn
○ http://scikit-learn.org/

36. Conclusion
● Definition of Machine Learning
● Buzzword breakdown
● Broad algorithm overview
● Machine Learning process
● Real world use cases
● The Machine Learning Audit
● Where to learn more about Machine Learning

37. Questions?

38. Thank you!