Pixels.camp - Machine Learning: Building Successful Products at Scale

© 2016 Feedzai Confidential 1
@antonioalegria
Product Lead for Cloud
Machine Learning:
Building Successful Products at Scale

© Feedzai Inc. Confidential.
This talk is about
Tips on building a successful Machine Learning product
One that works on multiple use cases within the same ecosystem
One that works on structured data and in classification use cases
Challenges with doing this for a generic SaaS Fraud Prevention
product
Data API design choices
Taking advantage of the data to power Machine Learning
5

This talk is NOT about
Unstructured data problems
Natural Language Processing
Image Recognition
Speech Recognition
Virtual Assistants
Self-driving cars
Specific technologies
Dataviz or UI
6

Machine Learning Products

Examples of ML Products
Gmail Spam detection
Recommendations:
Movies
Books
Music
Dating
Automating Access Control @ Amazon
Predicting Heart Attacks / Diseases on a Fitness Tracking Service
Fraud Detection ;-)

© Feedzai Inc. Confidential. 9
What makes them good?

They do their job very well

They blend in the environment

They self-evolve

They seem magical

Fraud Detection

Feedzai in a Nutshell
What? Detect fraudulent payments and their customers, in real-time
How?
We receive transaction and behavior data
We continuously update 1:1 profiles for every entity (e.g. cards, IPs, merchants, etc)
Machine Learning model analyses each payment and its history in real-time
User receives scores immediately, with human-readable explanations
User can give feedback by labeling transactions as “ok” or “fraud” – our models will learn automatically
Where? Deployed on-site or used in the cloud through REST API
When? our AI never sleeps and it responds in a few milliseconds

Huge Data
Securing over $2B per day (over $700B/year, 3.3x Portugal’s GDP)
Growing soon to reach trillion scale
Fighting crime across the globe
US – our clients use Feedzai to process $4 of every $10 in all US
Canada
Brazil
India
Nigeria
Europe
We have to make decisions in 25ms

DATA-DRIVEN FRAUD DETECTION
1:1 Profiling & Analytics
✖
✖ ✖
✖
Payments & Actions
$ € ¥ £
Machine Learning
Data Enrichment
★★★★★
Risk Analysis
Decision:
Approve, Decline, Review
User Feedback
Human-built Rules
Request
Response

White-box Scoring
Human explanations from AI reasoning

Challenges
SaaS for Online Commerce
Fraud Prevention in Online Commerce is a very broad scenario
Different geographies
Widely different use cases
Fraud and abuse are a case of extremely unbalanced classes
It can involve many abuse scenarios:
Payment fraud (e.g. stolen credit cards)
Account Takeovers
Money Laundering
Abusing employee benefits
Solution needs to SCALE

Scale…
I don’t think it means what you think it
means

Lets look at some of the key components of
a good ML product

The Data API

Data API Specificity Spectrum
• Very specific to a particular use case
• Strict validations
• Clients need to fully adapt to the API
• Defines bare minimum generic terms
• Clients have full flexibility to integrate
• Custom events with custom data fields
• API defines common “language”
• Comprehensive set of well-defined optional
fields
• Supports custom fields and events
• Clients adapt to the Native fields but can
use custom data
+ Shared Models
+ Shared Model Features
+ Very easy to fully automate and scale
– Low adaptation to clients
+ Potential for total adaptation to clients
– Costly adaptation to clients
– Hard to do feature engineering
– Fully Separate Models
– Fully Separate Model Features
API Flexibility
Model Shareability
+ Potential for high adaptation to clients
+ Tiered model possible (shared + specific)
+ Shared and specific model features
– Automation and scaling is not trivial
Generic ML Platform
(e.g. BigML)
Very use-case specific
(e.g. Email spam detection)
Platform for classes of use cases
(e.g. Feedzai for Online Commerce)

Responses should include the following
Score(s) (e.g. probability of being fraud)
Decision:
Accept
Review
Decline
Human-Readable Explanations
Machine-Readable Reason Codes
24

Feedzai API Example
POST /v1.1/payments
{
"id": "1477020120",
"user_id": "af00-bc14-1245",
"amount": 280000,
"currency": "USD",
"ip": "212.10.114.18",
"items": [
{
"item_id": "cell_400200",
"name": "Cellphone 1450",
"price": 25000
}
],
"payment_methods": [
{
"type": "card”,
"card_fullname": "HUGH Howey",
"card_pan": “4539488752989912",
"card_exp": "06/17”
}
],
"user_defined": {
"is_po_box": true,
"expedited_delivery": true
}
}
HTTP 200 (OK)
{
"id": "1477020120”,
"score": 740,
"decision": ”review”,
"reason_codes": [
{ "name": "Fraud" },
{ "name": "MoneyLaundering" },
{ "name": "AccountTakeover" }
],
"explanation": [
{
"description": ”Customer used over 3 cards in past
week.",
"risk": 0.4,
"confidence": 5
},
{
"description": "Customer has used a single internet
address in the last 24 hours.",
"risk": 0.003,
"confidence": 5
}
]
}
Request Response

The Machine Learning Algorithm
(in 30 seconds)

Machine Learning Algorithm
Encapsulate the actual ML into an isolated component
Start with algorithms that are fast to train and evaluate, adapt to
different use cases, support classification and regression and are
whitebox
Random Forest
Gradient Boosting Machines (GBMs)
Deep Learning shows potential for more unstructured problems
Though it’s heavyweight to train, requires a lot of pre-processing
Still unclear how much it can “replace” feature engineering
27

Machine Learning is 90% Data Processing

Machine LearningIn Production
Live Input Data Instance Vector
Enrich
Filter
Transform
Aggregate
Project
Historical Input Data
Naïve ML Pipeline
Instance Vector + Class Annotation
Training
Classify
Historical
Data
Enrich
Filter
Transform
Aggregate
Project

Example Input Data
Transaction:
Amount
Currency
User ID
User Name
Credit Card Number
Cardholder Name
IP Address
30

Example Naïve Features
Amount in USD
Currency
Time of Day
Day of Week
Is IP a Proxy
IP Country == Store Country
IP Country
User ID
Card
Device ID
Some of these features are aweful (don’t do this):
Having such high-cardinality categoricals is bad and leads to overfitting
Also, the model isn’t learning patterns just which users/devices/cards are bad
31

The model won’t be very good

It can’t distinguish between two equal
transactions from two different people.
We need to go further

Goal: the model must see
The current event
+
All* past events
+
All* related events (e.g. same card)

A good approximation to this are 1:1 Profiles

What’s a profile?
An aggregation or summarization of events over a certain time window
and for a group of entities
Examples:
Number of transactions in last 24h for this card
Number of transactions in past month for this customer with this card
Number distinct cards for this customer
Last 5 used card countries
1:1 refers to the fact that profiles are tracked by specific entities
36

Characteristics of a Profile
It’s applied over a (usually time) data window
Sliding
Tumbling
Delayed
It has a set of dimensions or entities to group by
It has an aggregation function
37

Challenges
How do you calculate these profiles continuously and in real-time?
How do you calculate profiles for both short term and long-term
windows?
How do you reproduce exactly the same processing in training, testing
and in production?
How do you make it so that Data Scientists can ship something to
production without having Developers’ intervention?
How do you easily “code-review” it?
38

Reproducibility between Training and
Production is essential
This is the most important thing

Reproducibility
Without a training pipeline that mirrors real-time the model will
learn something different than what it will see in reality
This kind of concept drift can kill your model’s performance
You can fix this in two ways:
Have a very strict (and slow) process of testing and QA
Or you use the same code during training, testing and in production
40

Data Scientists must be able to “code”, test
and ship Feature Engineering logic to
Production
(without having Sw. Eng. having to
implement it based on a spec)

Complex Event Processing
+
Large Scale Data Processing Platforms

Complex Event Processing
Data Processing methodology and family of stream-based
technologies
Relies on DSL, sometimes similar to SQL
Instead of applying queries/logic to data, the data goes
through in-memory queries that update state immediately
43

Example
SELECT user_id,
card,
avg(amount) AS avg_amount,
count() AS num_trx,
count() / last().timestamp - first().timestamp AS velocity
FROM transactions[24 hours]
GROUP BY user_id, card;

Common CEP Operations
Filtering
Correlation
Windowing
Transformation
Aggregation/Grouping
Merging/Union
Sorting
Pattern Detection
45

Complex Event Processing at Scale
CEP technology is usually reliant on in-memory processing
To handle long-term profiles you need to pair this with
distributed data processing platforms
The ability to replay historical data like in production should
be a core requirement for the whole system
46

Other Tips

Support 0-downtime deployment of new
models in staging mode

Good (consistent) Data > Lots of Data

Do things that don’t scale
• Look at specific data rows
• Open the CSVs
• Use SQL to try to find new insights

Throw away good data
(wait, what?)

>99.5% Good Transactions
< 0.5% Fraud
Fraud is extremely unbalanced
Use undersampling to drop good transactions

KeyTakeaways
Design APIs with comprehensive native fields but allow custom
data
Data Processing is 90% of Machine Learning
Must Have: full reproducibility of production behavior offline
and for training
Combine CEP and streaming with distributed batch processing
Combine Machine Learning with Human Intelligence
© 2016 Feedzai Confidential53

54
MACHINE LEARNING
Keep commerce safe
and create a better customer
experience through
machine learning.
INVESTORS
QUICKFACTS
MISSION
WHAT OTHERS SAY
The U.S. market
fraud prevention just
got a new player.
Feedzai’s
machine learning
is the next wave.
Ranked as a cool
technology to
watch.
Startups that are
owning the data
game.
Payment Card
Management: Essential
tools for U.S. card issuers
• Top 50 High Growth startups in Europe,
FASTEST GROWING startup in Portugal
• Founded by data scientists and aerospace
engineers in 2009
• 120+ employees and doubling
• Offices in Portugal, Silicon Valley, New York
City, London
• Series B funded by Oak HC/FT and Sapphire
Ventures (SAP)

Want to be a data Samurai?
We’re hiring! feedzai.com/about-us/careers

REFERENCES
• Automation like Iron Man, not Ultron and the Leftover Principle:
• http://queue.acm.org/detail.cfm?id=2841313
• Six novel ML applications:
• http://www.forbes.com/sites/85broads/2014/01/06/six-novel-machine-learning-
applications/#360c101567bf
• Complex Event Processing with Esper:
• http://www.slideshare.net/antonio_alegria/complex-event-processing-with-
esper-10122384
• Approaching almost any ML problem
• http://blog.kaggle.com/2016/07/21/approaching-almost-any-machine-learning-
problem-abhishek-thakur/
• https://www.datarobot.com/
• XGBoost tutorial – http://xgboost.readthedocs.io/en/latest/model.html

Pixels.camp - Machine Learning: Building Successful Products at Scale

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (17)

Similar to Pixels.camp - Machine Learning: Building Successful Products at Scale

Similar to Pixels.camp - Machine Learning: Building Successful Products at Scale (20)

Recently uploaded

Recently uploaded (20)

Pixels.camp - Machine Learning: Building Successful Products at Scale

Editor's Notes