This document provides observations and recommendations for reconciling machine learning with business needs. Some key points made include:
- In many cases, machine learning is not needed to solve a problem and simpler solutions like collecting missing data can work better.
- The data companies already have is sometimes useless for machine learning problems. Domain expertise alone also often means less than expected.
- Not understanding technical constraints can cause machine learning projects to fail. Always create a proof-of-concept first before full development.
- It is important to establish causality through proper testing like A/B testing, as this validates models and addresses financial risks of implementations.
- Framing learning problems is challenging due to issues like lous
An introduction to immediate-reward reinforcement learning. Covers introductions, motivation, challenges with full RL, contextual bandits, policy evaluation, and architectural considerations.
Rinse and Repeat : The Spiral of Applied Machine LearningAnna Chaney
Analyze and Improve Performance of Machine Learning in Four Easy Steps
Step 0. Deploy your machine learning application
Step 1. Assess performance of app using human judgement
Step 2. Analyze and optimize operating thresholds
Step 3. Retrain machine learning with golden examples from humans
Step 4. Go to Step 0 with new changes
Video: http://videos.re-work.co/videos/464-agile-deep-learning
Deep Learning has been called the ‘new electricity’ — transforming every industry. Innovative architectures and applications receive deserved attention. But to turn innovation into value requires integrating deep learning into practical technology products. Such products, including Spotify's, are often developed following the principles of agile. This talk focuses on approaching deep learning in an agile way and on integrating deep learning into the agile cadence of a modern software development organization.
Recently I gave a talk at UC Berkeley regarding the transition from academia to industry in the context of Machine Learning and Data Science related roles. I based most of my slides on my own transition from being an Astrophysicist to a Machine Learning Expert. I hope this will be useful to many. Feedback is welcome!
An introduction to immediate-reward reinforcement learning. Covers introductions, motivation, challenges with full RL, contextual bandits, policy evaluation, and architectural considerations.
Rinse and Repeat : The Spiral of Applied Machine LearningAnna Chaney
Analyze and Improve Performance of Machine Learning in Four Easy Steps
Step 0. Deploy your machine learning application
Step 1. Assess performance of app using human judgement
Step 2. Analyze and optimize operating thresholds
Step 3. Retrain machine learning with golden examples from humans
Step 4. Go to Step 0 with new changes
Video: http://videos.re-work.co/videos/464-agile-deep-learning
Deep Learning has been called the ‘new electricity’ — transforming every industry. Innovative architectures and applications receive deserved attention. But to turn innovation into value requires integrating deep learning into practical technology products. Such products, including Spotify's, are often developed following the principles of agile. This talk focuses on approaching deep learning in an agile way and on integrating deep learning into the agile cadence of a modern software development organization.
Recently I gave a talk at UC Berkeley regarding the transition from academia to industry in the context of Machine Learning and Data Science related roles. I based most of my slides on my own transition from being an Astrophysicist to a Machine Learning Expert. I hope this will be useful to many. Feedback is welcome!
Teaching Your Computer To Play Video Gamesehrenbrav
A rapid tour through some of the most exciting areas of machine learning, presenting the author's own efforts at training a computer to master Super Mario Bros.
Half day session on Machine learning and its applications. It introduces Artificial Intelligence, move on Machine Learning, applications, algorithms, types, using Cloud for ML, Deep Learning and some resources to start with
If you have heard about machine learning and want to try out some of it, please check this out. In this article I am just trying to jot down few basics and must know stuff to kick start in this field. The objective of this compilation; to trigger the interest in this field of data analytics and to demystify the abstract concept. This article is not for the advanced data scientists, this is for the beginners or those who want a quick refresher.
Semi supervised learning machine learning made simpleDevansh16
Video: https://youtu.be/65RV3O4UR3w
Semi-Supervised Learning is a technique that combines the benefits of supervised learning (performance, intuitiveness) with the ability to use cheap unlabeled data (unsupervised learning). With all the cheap data available, Semi Supervised Learning will get bigger in the coming months. This episode of Machine Learning Made Simple will go into SSL, how it works, transduction vs induction, the assumptions SSL algorithms make, and how SSL compares to human learning.
About Machine Learning Made Simple:
Machine Learning Made Simple is a playlist that aims to break down complex Machine Learning and AI topics into digestible videos. With this playlist, you can dive head first into the world of ML implementation and/or research. Feel free to drop any feedback you might have down below.
Crafting Recommenders: the Shallow and the Deep of it! Sudeep Das, Ph.D.
I present a brief review, and an outlook on the rapid changes happening in the field of recommendation engine research on the heels of the deep learning revolution!
Scott Clark, Software Engineer, Yelp at MLconf SFMLconf
Abstract: Introducing the Metric Optimization Engine (MOE); an open source, black box, Bayesian Global Optimization engine for optimal experimental design.
In this talk we will introduce MOE, the Metric Optimization Engine. MOE is an efficient way to optimize a system’s parameters, when evaluating parameters is time-consuming or expensive. It can be used to help tackle a myriad of problems including optimizing a system’s click-through or conversion rate via A/B testing, tuning parameters of a machine learning prediction method or expensive batch job, designing an engineering system or finding the optimal parameters of a real-world experiment.
MOE is ideal for problems in which the optimization problem’s objective function is a black box, not necessarily convex or concave, derivatives are unavailable, and we seek a global optimum, rather than just a local one. This ability to handle black-box objective functions allows us to use MOE to optimize nearly any system, without requiring any internal knowledge or access. To use MOE, we simply need to specify some objective function, some set of parameters, and any historical data we may have from previous evaluations of the objective function. MOE then finds the set of parameters that maximize (or minimize) the objective function, while evaluating the objective function as few times as possible. This is done internally using Bayesian Global Optimization on a Gaussian Process model of the underlying system and finding the points of highest Expected Improvement to sample next. MOE provides easy to use Python, C++, CUDA and REST interfaces to accomplish these goals and is fully open source. We will present the motivation and background, discuss the implementation and give real-world examples.
The term Machine Learning was coined by Arthur Samuel in 1959, an american pioneer in the field of computer gaming and artificial intelligence and stated that “ it gives computers the ability to learn without being explicitly programmed” And in 1997, Tom Mitchell gave a “ well-Posed” mathematical and relational definition that “ A Computer Program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E”.
Machine learning is needed for tasks that are too complex for humans to code directly. So instead, we provide a large amount of data to a machine learning algorithm and let the algorithm work it out by exploring that data and searching for a model that will achieve what the programmers have set it out to achieve.
Automating fetal heart monitor using machine learningTamjid Rayhan
This is a webinar held by the IEEE student branch of University of Chittagong. This talks about how a beginner can gain expert level knowledge in Machine learning and deep learning using online resources. It focuses on how the presentar solved a biomedical engineering problem using Machine learning. Also gives reference to many interesting references to advices given by the leaders of Machine learning field.
Machine Learning: Opening the Pandora's Box - Dhiana Deva @ QCon São Paulo 2019Dhiana Deva
Introducing Machine Learning is like opening the Pandora's Box - it unveils important issues in your data, metrics, and product. In order to deal with such complexity, pragmatic practices are required to obtain reliable results. In this talk, we will go through learnings gained from introducing Machine Learning in different contexts, from academia, start-ups, consulting to tech giants - covering practices for experimentation, infrastructure, planning, performance evaluation and product vision in the context of machine learning products.
Future of data science as a professionJose Quesada
How can you thrive in a future where machine learning has been popular for a few years already?
In this talk, I will give you actionable advice from my experience training serious data scientists at our retreat center in Berlin. You are going to face these pointy, hard questions:
- What is the promise of machine learning? Has it happened yet?
- Is it easy to take advance of machine learning, now that most algorithms are nicely packaged in APIs and libraries?
- How much time should I spend getting good at machine learning? Am I good enough now?
- Are data scientists going to be replaced by algorithms? Are we all?
- Is it easy to hire talent in machine learning after the explosion of MOOCs?
Teaching Your Computer To Play Video Gamesehrenbrav
A rapid tour through some of the most exciting areas of machine learning, presenting the author's own efforts at training a computer to master Super Mario Bros.
Half day session on Machine learning and its applications. It introduces Artificial Intelligence, move on Machine Learning, applications, algorithms, types, using Cloud for ML, Deep Learning and some resources to start with
If you have heard about machine learning and want to try out some of it, please check this out. In this article I am just trying to jot down few basics and must know stuff to kick start in this field. The objective of this compilation; to trigger the interest in this field of data analytics and to demystify the abstract concept. This article is not for the advanced data scientists, this is for the beginners or those who want a quick refresher.
Semi supervised learning machine learning made simpleDevansh16
Video: https://youtu.be/65RV3O4UR3w
Semi-Supervised Learning is a technique that combines the benefits of supervised learning (performance, intuitiveness) with the ability to use cheap unlabeled data (unsupervised learning). With all the cheap data available, Semi Supervised Learning will get bigger in the coming months. This episode of Machine Learning Made Simple will go into SSL, how it works, transduction vs induction, the assumptions SSL algorithms make, and how SSL compares to human learning.
About Machine Learning Made Simple:
Machine Learning Made Simple is a playlist that aims to break down complex Machine Learning and AI topics into digestible videos. With this playlist, you can dive head first into the world of ML implementation and/or research. Feel free to drop any feedback you might have down below.
Crafting Recommenders: the Shallow and the Deep of it! Sudeep Das, Ph.D.
I present a brief review, and an outlook on the rapid changes happening in the field of recommendation engine research on the heels of the deep learning revolution!
Scott Clark, Software Engineer, Yelp at MLconf SFMLconf
Abstract: Introducing the Metric Optimization Engine (MOE); an open source, black box, Bayesian Global Optimization engine for optimal experimental design.
In this talk we will introduce MOE, the Metric Optimization Engine. MOE is an efficient way to optimize a system’s parameters, when evaluating parameters is time-consuming or expensive. It can be used to help tackle a myriad of problems including optimizing a system’s click-through or conversion rate via A/B testing, tuning parameters of a machine learning prediction method or expensive batch job, designing an engineering system or finding the optimal parameters of a real-world experiment.
MOE is ideal for problems in which the optimization problem’s objective function is a black box, not necessarily convex or concave, derivatives are unavailable, and we seek a global optimum, rather than just a local one. This ability to handle black-box objective functions allows us to use MOE to optimize nearly any system, without requiring any internal knowledge or access. To use MOE, we simply need to specify some objective function, some set of parameters, and any historical data we may have from previous evaluations of the objective function. MOE then finds the set of parameters that maximize (or minimize) the objective function, while evaluating the objective function as few times as possible. This is done internally using Bayesian Global Optimization on a Gaussian Process model of the underlying system and finding the points of highest Expected Improvement to sample next. MOE provides easy to use Python, C++, CUDA and REST interfaces to accomplish these goals and is fully open source. We will present the motivation and background, discuss the implementation and give real-world examples.
The term Machine Learning was coined by Arthur Samuel in 1959, an american pioneer in the field of computer gaming and artificial intelligence and stated that “ it gives computers the ability to learn without being explicitly programmed” And in 1997, Tom Mitchell gave a “ well-Posed” mathematical and relational definition that “ A Computer Program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E”.
Machine learning is needed for tasks that are too complex for humans to code directly. So instead, we provide a large amount of data to a machine learning algorithm and let the algorithm work it out by exploring that data and searching for a model that will achieve what the programmers have set it out to achieve.
Automating fetal heart monitor using machine learningTamjid Rayhan
This is a webinar held by the IEEE student branch of University of Chittagong. This talks about how a beginner can gain expert level knowledge in Machine learning and deep learning using online resources. It focuses on how the presentar solved a biomedical engineering problem using Machine learning. Also gives reference to many interesting references to advices given by the leaders of Machine learning field.
Machine Learning: Opening the Pandora's Box - Dhiana Deva @ QCon São Paulo 2019Dhiana Deva
Introducing Machine Learning is like opening the Pandora's Box - it unveils important issues in your data, metrics, and product. In order to deal with such complexity, pragmatic practices are required to obtain reliable results. In this talk, we will go through learnings gained from introducing Machine Learning in different contexts, from academia, start-ups, consulting to tech giants - covering practices for experimentation, infrastructure, planning, performance evaluation and product vision in the context of machine learning products.
Future of data science as a professionJose Quesada
How can you thrive in a future where machine learning has been popular for a few years already?
In this talk, I will give you actionable advice from my experience training serious data scientists at our retreat center in Berlin. You are going to face these pointy, hard questions:
- What is the promise of machine learning? Has it happened yet?
- Is it easy to take advance of machine learning, now that most algorithms are nicely packaged in APIs and libraries?
- How much time should I spend getting good at machine learning? Am I good enough now?
- Are data scientists going to be replaced by algorithms? Are we all?
- Is it easy to hire talent in machine learning after the explosion of MOOCs?
Machine Learning has become a must to improve insight, quality and time to market. But it's also been called the 'high interest credit card of technical debt' with challenges in managing both how it's applied and how its results are consumed.
Synergy of Human and Artificial Intelligence in Software EngineeringTao Xie
Keynote Talk by Tao Xie at International NSF sponsored Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE 2013) http://promisedata.org/raise/2013/
Where have all the data entry candidates gone?Infrrd
If you are struggling to hire data entry roles to help extract data from documents, please take comfort in the fact that you are not alone. Businesses and institutions of all sizes, even the IRS, are challenged by an acute labor shortage.
Complete Article: https://hubs.ly/Q01b-7Cg0
Data Science for Business Managers - An intro to ROI for predictive analyticsAkin Osman Kazakci
This module addresses critical business aspects related to launching a predictive analytics project. How to establish the relationship with business KPIs is discussed. A notion of data hunt, for planning & acquiring external data for better predictions is introduced. Model quality and it's role for ROI of data and prediction tasks are explained. The module is concluded with a glimpse on how collaborative data challenges can improve predictive model quality in no time.
Operationalizing Machine Learning in the Enterprisemark madsen
TDWI Munich 2019
What does it take to operationalize machine learning and AI in an enterprise setting?
Machine learning in an enterprise setting is difficult, but it seems easy. All you need is some smart people, some tools, and some data. It’s a long way from the environment needed to build ML applications to the environment to run them in an enterprise.
Most of what we know about production ML and AI come from the world of web and digital startups and consumer services, where ML is a core part of the services they provide. These companies have fewer constraints than most enterprises do.
This session describes the nature of ML and AI applications and the overall environment they operate in, explains some important concepts about production operations, and offers some observations and advice for anyone trying to build and deploy such systems.
[DSC Europe 22] Avoid mistakes building AI products - Karol PrzystalskiDataScienceConferenc1
Based on Gartner's research, 85% of AI projects fail. In this talk, we show the most common mistakes made by the managers, developers, and data scientists while building AI products. We go through ten case studies of products that failed and analyze the reasons for each failure. We also present how to avoid such mistakes and deliver a successful AI product by introducing a few lifecycle changes.
Top 3 ways to use your UX team - producttank DFW MeetupJeremy Johnson
As a product owner or manager how should you be using your User Experience team? In this quick talk I go over the top three ways to use your UX team to support you in building better products.
Similar to (In)convenient truths about applied machine learning (20)
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
4. Machine learning is
sometimes at odds with
business in general
Machine learning is perhaps the best example of applying
the scientific method:
“It involves formulating hypotheses, via induction, based on
such observations; experimental and measurement-based
testing of deductions drawn from the hypotheses; and
refinement (or elimination) of the hypotheses based on the
experimental findings”
All machine learning projects are, in effect, a series of
experiments, where outcomes are uncertain.
5. Machine learning is
sometimes at odds with
business in general
A key tenant of the scientific method is that “failed”
experiments don’t equal failure.
Failed experiments add to the body of knowledge, and allow
us to do better in the future.
7. Machine learning is
sometimes at odds with
business in general
Unfortunately, business rarely looks at failed projects in the
same way scientists do. This can be hard to reconcile.
8. Machine learning is
sometimes at odds with
business in general
Project A: “let’s build a new webshop for our product”
Project B: “We lose 2 million each year because of wasted
inventory. Let’s solve that using ML”
9. How do we reconcile the scientific method with
the business world?
There’s no silver bullet. But by studying the experiences of others, and bringing ML
closer to what businesses care about, we can avoid some mistakes.
10. What follows are some observations. Some seem very obvious,
some not, but all still pose a challenge in practice.
Disclaimer: all of the following examples are based on personal experience, personal
failures, or personal opinion. Please consume with a healthy grain of salt.
11. In many cases, you don’t need machine learning in
order to solve a problem
12. In many cases, you don’t
need machine learning in
order to solve a problem
Data Scientist: “we built a model for predicting the channel
customers contact us in”
PO: “awesome, let’s take this to production!”
Data Scientist: “great, I’ll work with our engineers to make it
happen”
(development continues)
Engineer: “why don’t we just collect the correct channel
information when someone calls or emails us?”
PO: “...”
Data Scientist: “...”
13. In many cases, you don’t
need machine learning in
order to solve a problem
“Rule #1: Don’t be afraid to launch a product without
machine learning.
Machine learning is cool, but it requires data. Theoretically,
you can take data from a different problem and then tweak
the model for a new product, but this will likely
underperform basic heuristics. If you think that machine
learning will give you a 100% boost, then a heuristic will get
you 50% of the way there.”
Rules of Machine Learning: Best Practices for ML Engineering (Martin Zinkevich et al),
http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf
15. Sometimes, the data you
already have is useless
Client: “we want to be able to predict who is most likely to be our
customer in the future”
Data Scientist: “OK, for whom would you like to be able to predict
that?”
Client: “for all people that aren’t already our customers”
16. Sometimes, the data you
already have is useless
Client: “we want to be able to predict who is most likely to be our
customer in the future”
Data Scientist: “OK, for whom would you like to be able to predict
that?”
Client: “for all people that aren’t already our customers”
18. Not understanding technical
constraints can make a
machine learning project
fail
Business: “let’s use machine learning to automatically assign
tickets to the proper technician”
Data Scientist: “sounds plausible, I’ll get to work”
(development continues)
Data Scientist: “here’s the best model I could make. In
simulation, it’s only wrong 0.1% of the time”
Business: “that’s unacceptable – it can’t assign work to the
wrong technician”
Data Scientist: “but it’s function approximation...by
definition, it can’t–”
Business: “no exceptions”
Data Scientist: “...”
19. Not understanding technical
constraints can make a
machine learning project
fail
Data Scientist: “I’ve made a non-parametric model for a
recommendation engine and now we need to deploy it to
production”
Engineer: “OK, where’s the data you need at prediction
time?”
Data Scientist: “Oh, some of it is in two data warehouses and
the rest is in S3”
Engineer: “We have to make that data accessible in an
operational DB. How much data are we talking about?”
Data Scientist: “Around 2 billion rows”
Engineer: “...”
Data Scientist: “Oh, and since the model is non-parametric
and in-memory, it needs 50GB of RAM to run and doesn’t
scale horizontally”
25. In machine learning,
domain expertise means
less than you might think
Predicting customer churn in an eCommerce business
Data Scientist: “OK, i’ll start with these features, gridsearch a good
XGBClassifier and iterate from there”
Predicting if heavy machinery is likely to break down within
the next day
Data Scientist: “OK, i’ll start with these features, gridsearch a good
XGBClassifier and iterate from there”
26. In machine learning,
domain expertise means
less than you might think
Just about any image recognition task, regardless of industry:
Data Scientist: I’ll use a convnet
27. In machine learning,
domain expertise means
less than you might think
During planning
Business owner(s): “the model should take a,b,c,d,e,f & g into
account when making a decision”
Data Scientist: “OK”
During modelling
Data Scientist:
“let me use a,b,c,d,e,f & g and make a baseline model”
“hmm, these results aren’t great. I’ll add h,i,j & k”
“hmm. a, b, c & d have no predictive power – I’ll drop those”
Data Scientist: “all done!”
Business owner(s): “this model takes into account the stuff we
talked about, right?”
Data Scientist: “sure.”
28. In machine learning,
domain expertise means
less than you might think
Consult with stakeholders to understand the
problem and get an idea of what types of
data might be useful (include everyone’s
ideas)
Figure out what data is viable to get/use as
features
Through modelling, learn what type of data
is actually useful
Business heuristics? (Possibly) add them as features, never as
hard-coded logic. Let learning algorithms figure out if they are useful.
31. Always make a
proof-of-concept
Hidden Technical Debt in Machine Learning (D. Sculley, Gary Holt, Daniel Golovin et al),
http://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
Machine Learning: The High-Interest Credit Card of Technical Debt (D. Sculley, Gary Holt, Daniel
Golovin, Eugene Davydov et al),
https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43146.pdf
32. Always make a
proof-of-concept
Machine learning projects can, and will, fail from time to time. To
start, make the simplest model possible, and test its effectiveness
using the simplest possible process. Adding surrounding
infrastructure without validating the approach first is asking for
trouble.
33. Always do a proper test to establish causality - or why
you need to take a financial risk
34. Always do a proper test to
establish causality - or why
you need to take a financial
risk
The gold standard for establishing causality is a randomised
controlled experiment (A/B-test), though other useful causal
inference methods also exist for situations where A/B-testing isn’t
possible.
35. Always do a proper test to
establish causality - or why
you need to take a financial
risk
During a controlled experiment, you are invariably taking a
financial risk to determine the effectiveness of a machine learning
model.
Sometimes, it is surprisingly difficult to convince everyone that you
have to take a risk.
36. Always do a proper test to
establish causality - or why
you need to take a financial
risk
Example: predicting customer churn
“Can’t we just log churn risks without actually acting upon them,
and then follow up on how many people churned?”
37. Always do a proper test to
establish causality - or why
you need to take a financial
risk
Example: predicting customer churn
“Can’t we just log churn risks without actually acting upon them,
and then follow up on how many people churned?”
Problem 1: data scientists already do these counterfactual tests as
part of modelling (testing accuracy on new data)
38. Always do a proper test to
establish causality - or why
you need to take a financial
risk
Example: predicting customer churn
“Can’t we just log churn risks without actually acting upon them,
and then follow up on how many people churned?”
Problem 2: the treatment action may itself influence future
behaviour
39. Always do a proper test to
establish causality - or why
you need to take a financial
risk
Example: predicting customer churn
“Can’t we just log churn risks without actually acting upon them,
and then follow up on how many people churned?”
Problem 3: if we did a “risk-free” run, and the model worked well,
we’d still need a real A/B test, effectively doubling time spent
testing
40. Always do a proper test to
establish causality - or why
you need to take a financial
risk
Moral of the story: don’t shy away from real experimentation.
Mitigate risks during modelling and/or by varying treatment group
sizes (Bayesian methods handle the latter naturally)
42. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Let’s say we are tasked with building a recommender system for a
news site.
Do we build a model that:
● Predicts clicks/non-clicks?
● Predicts read time?
● Predicts conversion rates?
● Predicts explicit ratings?
● Predicts implicit ratings?
● Predicts something else?
Side note: all of the above have be used for recommendations in the
past.
43. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Let’s say we are tasked with building a recommender system for a
news site.
Do we use a:
● Regression algorithm?
● Binary classification algorithm?
● A pairwise classification algorithm?
● A ranking algorithm?
● A multiclass classification algorithm?
● A multilabel classification algorithm?
● A matrix factorization algorithm?
● A non-parametric similarity algorithm?
● A reinforcement learning algorithm?
● ...
● A hybrid approach?
Side note: all of the above can be used for recommendations.
44. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Thumb rule: first choose a good metric, then experiment with
different learning algorithms.
Problem: most metrics used in business range from bad to
terrible.
45. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
On The Theory of Scales of Measurement (S.S. Stevens),
https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf
47. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Avg. rating, “The website has a friendly user interface”: 3.5/5
48. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Avg. rating, “The website has a friendly user interface”: 3.5/5
Not strictly allowed, but yet we do this all
the time. Why?
49. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Good metrics are:
+ Measurable
+ Objective and unhackable*
+ Derived from strategy
+ Describe what you want and need to know
+ Are usable in every-day work
+ Understanded and accessible by everyone
+ Validated regularly
Bad metrics are:
- Unmeasurable
- Subjective and/or hackable
- Derived from coffee table conversation
- Chosen because they were easily available
- Too big to have an impact on or too narrow to describe different
cases
- Unknown to other stakeholders and in worst case even to you
- Not trusted or fully understood
Credit: Jan Hiekkaranta
50. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Good metrics are:
+ Measurable
+ Objective and unhackable*
+ Derived from strategy
+ Describe what you want and need to know
+ Are usable in every-day work
+ Understanded and accessible by everyone
+ Validated regularly
Bad metrics are:
- Unmeasurable
- Subjective and/or hackable
- Derived from coffee table conversation
- Chosen because they were easily available
- Too big to have an impact on or too narrow to describe different
cases
- Unknown to other stakeholders and in worst case even to you
- Not trusted or fully understood
Credit: Jan Hiekkaranta
51. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Theoretically, the best way to apply ML in business is to optimise
directly against critical business KPIs, such as profit.
In practice, this is extremely difficult, because so many other things
can influence highest-level KPIs.
The solution? Derive a good proxy metric.
52. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics< closer to your problem farther from your problem >
read time
engagement
customer value
EBIT
53. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Q: How do you know your proxy metric is good?
A: Validate that it tracks well with higher-level metrics. This can
even be done statistically, e.g. using IEEE’s standards for software
measurement (IEEE Standard for a Software Quality Metrics
Methodology. Technical report, December 1998, ISBN
1-55937-529-9). The standards aren’t made for this purpose, but
work quite well!
54. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
Statistical validation aside, thoughtful reasoning is still valuable.
Consider recommender systems that predict click-through-rates
(CTR):
● Does a click really mean I’m interested?
● Who would really care about CTRs if I can improve total
minutes spent with our system?
○ Conversely, who would care if CTRs were high but
read times lousy?
● What biases are at play here?
● ...
● Where does money change hands?
55. Framing learning problems
isn’t as easy as it seems, and
it’s mostly because of lousy
metrics
To business developers: set out well-designed, validated KPIs &
proxy metrics and require that ML projects target those. Data
Scientists can help with metric designs.
57. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
True positives
11,854
False positives
582
False negatives
134
True negatives
300,297
F1-Score: 0.9707, Recall: 0.9888, Precision: 0.9532
58. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
True positives
11,854
False positives
582
False negatives
134
True negatives
300,297
F1-Score: 0.9707, Recall: 0.9888, Precision: 0.9532
Use case: predicting fraud
59. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
True positives
11,854
False positives
23
False negatives
1333
True negatives
300,297
F1-Score: 0.9451, Recall: 0.8989, Precision: 0.9981
60. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
True positives
11,854
False positives
23
False negatives
1333
True negatives
300,297
F1-Score: 0.9451, Recall: 0.8989, Precision: 0.9981
Use case: detecting malignant tumours
61. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
All classification problems are cost-sensitive
classification problems.
62. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
All classification problems are cost-sensitive
classification problems.
Expected cost in €
63. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
Strategies for cost-sensitive classification:
● Upsampling
● Downsampling
● Rejection sampling
● Importance weighting
● Using a native cost-sensitive classification algorithm
64. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
Data Scientist: “on validation data, the accuracy is 98% with
an F1-score of 94%. This is a 19% improvement over our
baseline”
Data Scientist: “we estimate 3,4 euros more per month per
user if we put this model into production”
65. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
Data Scientist: “on validation data, the accuracy is 98% with
an F1-score of 94%. This is a 19% improvement over our
baseline”
Data Scientist: “we estimate 3,4 euros more per month per
user if we put this model into production”
66. Data Scientists should
optimise a model against
real costs & returns, but
often can’t
Predicting customer churn
Data Scientist: “What’s the expected cost to the company if we fail
to keep a customer from leaving?”
PO: “Well, the expected lifetime value of a customer is around 350
euros”
Business Manager A: “100 euros”
Software Engineer B: “1210 euros”
Accountant C: “420 euros”
Another Data Scientist: “It depends”
68. Existing business processes
can severely restrict the
potential of machine
learning
“We already have a logic-based system for flagging critical alarms,
but some still slip through. We’d like to replace the entire system
with ML”
Data Scientist: “OK, where’s the control group data?”
69. Existing business processes
can severely restrict the
potential of machine
learning
“We want to forecast the number of customer service chats each
day, for resource allocation purposes. We’ve got data on all the calls
our reps take”
Data Scientist: “Are the incoming chat attempts recorded
somewhere? Is the customer service number closed during
evenings/weekends?”
71. Existing opinions can
severely restrict the
potential of machine
learning
“The prices the algorithm suggest is sometimes to low, so we
disregard those”
“On Fridays, we don’t use the recommendation engine because our
content creators want to promote something else”
“We can’t release this to production; the recommendations I got
were pretty bad”
All of the above are strawman examples. Edge cases that are truly
suboptimal should be addressed on the algorithm level, not by
slapping opinions on top.
If you aren’t ready to let machine learning do its thing, don’t
use it. The less you override it, the better it works.
73. When the machine learning part of a machine learning
project fails, it’s because of bad features/feature
engineering
74. When the machine learning
part of a machine learning
project fails, it’s because of
bad features/feature
engineering
Garbage In, Garbage Out.
75. When the machine learning
part of a machine learning
project fails, it’s because of
bad features/feature
engineering
When a model fails to predict something, it’s because the
information used to train it lacks predictive power.
This, in turn, is because either the information used is wrong, or
not engineered into useful features.
There are no exceptions to this rule. Applied machine learning
is basically an exercise in feature engineering (note: feature
engineering is hard).
76. When the machine learning
part of a machine learning
project fails, it’s because of
bad features/feature
engineering
Good feature engineering + a naïve learning algorithm trumps bad
engineering + a sophisticated learning algorithm 99% of the time.
77. When the machine learning
part of a machine learning
project fails, it’s because of
bad features/feature
engineering
At the end of the day, some machine learning projects succeed and
some fail. What makes the difference? Easily the most important
factor is the features used. If you have many independent features
that each correlate well with the class, learning is easy. On the other
hand, if the class is a very complex function of the features, you
may not be able to learn it. Often, the raw data is not in a form that
is amenable to learning, but you can construct features from it that
are. This is typically where most of the effort in a machine learning
project goes. It is often also one of the most interesting parts, where
intuition, creativity and “black art” are as important as the
technical stuff.
A Few Useful Things to Know about Machine Learning (Pedro Domingos),
https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf
78. When the machine learning
part of a machine learning
project fails, it’s because of
bad features/feature
engineering
82. Machine learning can’t
really be called “intelligent”
unless you allow for
exploration
Direct Feedback Loops. A model may directly influence the
selection of its own future training data. It is common practice to
use standard supervised algorithms, although the theoretically
correct solution would be to use bandit algorithms. The problem
here is that bandit algorithms (such as contextual bandits [9]) do
not necessarily scale well to the size of action spaces typically
required for real-world problems. It is possible to mitigate these
effects by using some amount of randomization [3], or by isolating
certain parts of data from being influenced by a given model.
Hidden Technical Debt in Machine Learning (D. Sculley, Gary Holt, Daniel Golovin et al),
http://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
83. Machine learning can’t
really be called “intelligent”
unless you allow for
exploration
Learn
Log
Deploy
Almost all production machine learning
systems
84. Machine learning can’t
really be called “intelligent”
unless you allow for
exploration
A fundamentally correct machine learning
system
Learn
Log
Explore
Deploy
86. Having company-wide
control groups is a
non-negotiable part of
data-driven decision
making & modelling
Some things I’ve seen happen:
- Random uniform choices working better than
human opinions (including my own)
- Machine learning models tested only against other
machine learning models
- “Controlled” experiments run without control groups
- A/B tests failing due to other treatments happening at the
same time
88. Programming languages for
Data Science aren’t all that
great
R: made for data science, with other stuff added later
Python: built for general purpose computing, with data science
stuff added later
Both: too slow in many cases
Others: not always viable because of meagre ecosystems
89. The tool & service ecosystem for machine learning is
fragmented, non-standardised, and fragile
90. The tool & service
ecosystem for machine
learning is fragmented,
non-standardised, and
fragile
91. The tool & service
ecosystem for machine
learning is fragmented,
non-standardised, and
fragile
Current status of model exchange formats
93. Sometimes, Data Scientists
make good models using
learning algorithms they
don’t fully understand
Me: “neural networks learn through backpropagation, which
adjusts weights based on the chain rule and the partial derivative of
the loss function with respect to the weights in each layer.
Initialisation must however be symmetry-breaking...”
Me: “gradient boosted trees learn using a set of weak learners”
Me “Random Forests are made up of trees”
Me: “what’s an SVM?”
94. Sunk costs are almost always taken into account when
productionising machine learning projects, but they
shouldn’t be
95. Sunk costs are almost
always taken into account
when productionising
machine learning projects,
but they shouldn’t be
“The license for this platform cost us 1.2 M€, so it should be our
primary platform going forward.”
96. Sunk costs are almost
always taken into account
when productionising
machine learning projects,
but they shouldn’t be
Internal thinking: “developing this model & A/B test took 4
months, so we’re definitely taking it to production”
97. Sunk costs are almost
always taken into account
when productionising
machine learning projects,
but they shouldn’t be
In 1968 Knox and Inkster,[2] in what is perhaps the classic sunk
cost experiment, approached 141 horse bettors: 72 of the people
had just finished placing a $2.00 bet within the past 30 seconds, and
69 people were about to place a $2.00 bet in the next 30 seconds.
Their hypothesis was that people who had just committed
themselves to a course of action (betting $2.00) would reduce
post-decision dissonance by believing more strongly than ever that
they had picked a winner. Knox and Inkster asked the bettors to
rate their horse's chances of winning on a 7-point scale. What they
found was that people who were about to place a bet rated the
chance that their horse would win at an average of 3.48 which
corresponded to a "fair chance of winning" whereas people who
had just finished betting gave an average rating of 4.81 which
corresponded to a "good chance of winning".
98. Sunk costs are almost
always taken into account
when productionising
machine learning projects,
but they shouldn’t be
99. Sunk costs are almost
always taken into account
when productionising
machine learning projects,
but they shouldn’t be
Though from a different domain, adapting the Markov property
is a good rule of thumb.
“The future should be independent of the past given the present”
101. Why can’t we use machine
learning to optimise an
airport?
Optimal aircraft parking & people transportation using linear
programming. Live at Kittilä Airport.
102. Why can’t we use machine
learning to generate a single
malt whisky?
Machine-generated single malt whisky recipes, curated by
Mackmyra’s Master Blender. A mix of old & new learning
algorithms, including generator/discriminators. Full reveal at
The Next Web 2019.
104. Thank you! Questions?
A special thanks to Jarno Kartela & Jan Hiekkaranta for their
contributions.
max.pagels@fourkind.com
www.fourkind.com
@fourkindnow