SlideShare a Scribd company logo
1 of 22
Download to read offline
Solving a business problem in just two weeks
using machine learning
Lucas F Navarro and Pedro Suguimoto
lucas.navarro@getninjas.com.br
pedro.suguimoto@getninjas.com.br
PAPIs Connect - São Paulo 2017
GetNinjas’ Data Science Squad
Product Owner
Bs’ in Computer Science at UFSCar
Ms’ in Artificial Intelligence at
UFSCar
Data Scientist
Mechatronics Engineer from
USP/CAASO
Data Scientist
Bs’ in Applied Maths at UNICAMP
PhD’ in Artificial Intelligence at
USP/POLI
Python Developer
Bs’ in Information System at
USP/EACH
BackEnd Developer
Computer Engineer from
USP/POLI
Agenda
1. How does GetNinjas Work
2. The current Business Model and why we chose it
3. The big problem that has emerged
4. How we solved this problem in just two weeks
5. Results we achieved
6. Touchpoints
How does GetNinjas work?
(Client)
1. Choose what kind
of service that you
need.
2. We indicate up to 4
pros at your region
3. Hire a Pro to solve
your problem
How does GetNinjas work?
(Pro)
1. Choose a service
request from the
store
2. Use credits to unlock
the contact of the client
3. Get in touch with
the client to be hired
OVERVIEW
2,000,000
services hired in the last year
200,000
professionals registered
R$ 210M+
in transactions concluded (GMV) over the
last 12 months
100+
categories ranging from electrician to
english teacher to lawyer and so on...
We were performing a staged rollout of a new pay-per-lead model, with
significant improvements to the pros' experience
Old Model
Subscription
- Pro Pays Fixed
Value Per Month
- Receives up to
30 contacts
New Model
Bundle
- Pro Purchases a
Bundle of credits
- Use credits to
purchase contacts
Vs
First Launch:
South (Sep/18)
Second Launch:
North + Center
(Oct/23)
At last:
SP & RJ (2017)
Main improvements:
● Pros can purchase how many contacts they
want, when they want
● Clients will always receive interested Pros
● More feasible to balance Supply & Demand
Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
two weeks!!!
29%
of the requests were not
being sold
We would have 43.500
clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure
out what causes this rate
Some of the possible reasons:
○ Price
○ Location
○ Clients only estimating price
○ Requests lacking a good description
○ Miscategorized services
○ Not enough pros with credits
How to solve all of this in two weeks?
Our next launch was much bigger and we could not leave our
clients unanswered.
Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
two weeks!!!
29%
of the requests were not
being sold
We would have 43.500
clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure
out what causes this rate
Some of the possible reasons:
○ Price
○ Location
○ Clients only estimating price
○ Requests lacking a good description
○ Miscategorized services
○ Not enough pros with credits
How to solve all of this in two weeks?
Our next launch was much bigger and we could not leave our
clients unanswered.
Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
two weeks!!!
29%
of the requests were not
being sold
We would have 43.500
clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure
out what causes this rate
Some of the possible reasons:
○ Price
○ Location
○ Clients only estimating price
○ Requests lacking a good description
○ Miscategorized services
○ Not enough pros with credits
How to solve all of this in two weeks?
Our next launch was much bigger and we could not leave our
clients unanswered.
The problem gets bigger!
Each point is a “Mini
GetNinjas”, with
100 categories
300 cities
100 categories
x
30k pairs
which we have to deal with that
problem
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have
Pros available to solve your
problem at this moment”
2nd option, let’s give this services for free to signed
up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician
service request near your location! Get the
contact in “My Requests” tab at our app: %url”
11%
of the Free Pros were
purchasing a bundle after
seeing the services store at our App
Only 4% of our subscribed pros were
payees at that time
We had the potential to more than
double our payees base if we sustained
that conversion rate
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have
Pros available to solve your
problem at this moment”
2nd option, let’s give this services for free to signed
up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician
service request near your location! Get the
contact in “My Requests” tab at our app: %url”
11%
of the Free Pros were
purchasing a bundle after
seeing the services store at our App
Only 4% of our subscribed pros were
payees at that time
We had the potential to more than
double our payees base if we sustained
that conversion rate
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have
Pros available to solve your
problem at this moment”
2nd option, let’s give this services for free to signed
up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician
service request near your location! Get the
contact in “My Requests” tab at our app: %url”
11%
of the Free Pros were
purchasing a bundle after
seeing the services store at our App
Only 4% of our subscribed pros were
payees at that time
We had the potential to more than
double our payees base if we sustained
that conversion rate
How could we predict that a client request wouldn’t be purchased?
We had already figured out the reasons that a
service was not sold, so we used it as features and
started to test some machine learning models using R.
Main R packages used:
For data prep:
○ tidyr
○ dplyr
ML algorithms:
○ monmlp (neural net)
○ e1071 (naivebayes)
○ party (decision trees)
Utilities:
○ ROCR (ROC curve)
○ SDMTools (confusion matrix)
○ Boruta (feature selection)
○ Pool + RPostgreSQL (integration
for our data warehouse - RedShift)
Some of the features:
● Number of Pros with credits*
● Avg Distance from pros*
● Number of highly reviewed pros*
● Price
● Does service has a text description?
Class: Was the request sold? (boolean)
We used Rstudio Server remotely from a m4.large EC2 machine at Amazon’s Cloud
*Just pros inside the request area (inside a radius)
From R to Python, Fujin was Born!
We came up with a Neural Network with great performance:
○ With this specificity rate, just 30% of the unsold requests clients
would remain without an answer
○ With this sensitivity rate, we would distribute only 10% of the
good requests for free
After we validated the model, we implemented it in python to
go live:
○ We used the basic DS set (scikit-learn, pandas, numpy)
○ Alse we used Neurolab for the MLP algorithm
○ We used Flask to create the prediction microservice
70%
of specificity
First Model
79%
of accuracy
*Python is the main Data Science language at GetNinjas
90%
of sensitivity
We called the project Fujin!
How fast this entire project was delivered?
To have a functional ML model to predict unpurchased requests we spent just around 2 weeks!
*The first distribution algorithm was a random distribution to Free Pros
In Its first month live, Fujin were able to predict 68% of the unpurchased requests
Unpurchased
Predicted/
Distributed
Contacted
Hired
41%
28%
9%
5%
Service
Requests
With north+center regions, unpurchased request
rate grown from 29% to 41% due sparsity of
S&D in this new regions
With Fujin online, the number of unanswered
clients were just 13% (instead of 41%)
Fujin also helps to increase our Gross
Merchandise Volume(GMV) through hiring rate
*Numbers from our requests of November/2016
Fujin was also helping to activate Free Pros into Payees
*Numbers from our requests of November/2016
≈R$25k
in purchases from Pros after they
received a free client contact from Fujin
65%
of this pros didn’t even had our App
installed
⚠ Only 14% of the distributed leads
were read by the Pros
Already had our App installed Downloaded App post Fujin lead Didn`t Download
Towards to the Future
● The distribution algorithm
○ Pick the best pros to send free requests
○ Determine the amount of free pros to send the contact from a request given the Fujin’s
feedback
● Enhance the model with more features
○ We have more service categories with distinct characteristics nowadays
○ We have SP and RJ state (higher density regions than before)
● Features to increase the visualization rate of free distributed requests
○ Such as adding a deeplink to our the contact screen on our App at the e-mail/sms sent to Pros
Touchpoints
● The non-bureaucratic/non-hierarchical organization of a StartUp company was
crucial for us to develop Fujin in such short time
● ML is amazing, in just two weeks we had a running model to solve the biggest
part of our problem
● ML is super cool, but most of the times we solve our problems with much simpler
approaches
⚠ Do not create a super complicated solution and later adjust your problem to it, start from
the problem, and climb to the complex solution as you need it.
Thanks for your attention!!!
Questions?

More Related Content

More from PAPIs.io

A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...
A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...
A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...PAPIs.io
 
Scaling machine learning as a service at Uber — Li Erran Li at #papis2016
Scaling machine learning as a service at Uber — Li Erran Li at #papis2016Scaling machine learning as a service at Uber — Li Erran Li at #papis2016
Scaling machine learning as a service at Uber — Li Erran Li at #papis2016PAPIs.io
 
Real-world applications of AI - Daniel Hulme @ PAPIs Connect
Real-world applications of AI - Daniel Hulme @ PAPIs ConnectReal-world applications of AI - Daniel Hulme @ PAPIs Connect
Real-world applications of AI - Daniel Hulme @ PAPIs ConnectPAPIs.io
 
Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...
Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...
Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...PAPIs.io
 
Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...
Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...
Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...PAPIs.io
 
Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect
Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs ConnectDemystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect
Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs ConnectPAPIs.io
 
Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
Predictive APIs: What about Banking? - Natalino Busa @ PAPIs ConnectPredictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
Predictive APIs: What about Banking? - Natalino Busa @ PAPIs ConnectPAPIs.io
 
Microdecision making in financial services - Greg Lamp @ PAPIs Connect
Microdecision making in financial services - Greg Lamp @ PAPIs ConnectMicrodecision making in financial services - Greg Lamp @ PAPIs Connect
Microdecision making in financial services - Greg Lamp @ PAPIs ConnectPAPIs.io
 
Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...
Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...
Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...PAPIs.io
 
Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...
Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...
Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...PAPIs.io
 
How to predict the future of shopping - Ulrich Kerzel @ PAPIs Connect
How to predict the future of shopping - Ulrich Kerzel @ PAPIs ConnectHow to predict the future of shopping - Ulrich Kerzel @ PAPIs Connect
How to predict the future of shopping - Ulrich Kerzel @ PAPIs ConnectPAPIs.io
 
The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...
The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...
The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...PAPIs.io
 
Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...
Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...
Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...PAPIs.io
 
Machine Learning Services Benchmark - Inês Almeida @ PAPIs Connect
Machine Learning Services Benchmark - Inês Almeida @ PAPIs ConnectMachine Learning Services Benchmark - Inês Almeida @ PAPIs Connect
Machine Learning Services Benchmark - Inês Almeida @ PAPIs ConnectPAPIs.io
 
Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...
Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...
Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...PAPIs.io
 
How to Build a Successful Data Team - Florian Douetteau @ PAPIs Connect
How to Build a Successful Data Team - Florian Douetteau @ PAPIs ConnectHow to Build a Successful Data Team - Florian Douetteau @ PAPIs Connect
How to Build a Successful Data Team - Florian Douetteau @ PAPIs ConnectPAPIs.io
 
Microdecision Making in Financial Services - Greg Lamp @ PAPIs Connect
Microdecision Making in Financial Services - Greg Lamp @ PAPIs ConnectMicrodecision Making in Financial Services - Greg Lamp @ PAPIs Connect
Microdecision Making in Financial Services - Greg Lamp @ PAPIs ConnectPAPIs.io
 
[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs ConnectPAPIs.io
 
Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...
Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...
Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...PAPIs.io
 
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...PAPIs.io
 

More from PAPIs.io (20)

A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...
A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...
A tensorflow recommending system for news — Fabrício Vargas Matos (Hearst tv)...
 
Scaling machine learning as a service at Uber — Li Erran Li at #papis2016
Scaling machine learning as a service at Uber — Li Erran Li at #papis2016Scaling machine learning as a service at Uber — Li Erran Li at #papis2016
Scaling machine learning as a service at Uber — Li Erran Li at #papis2016
 
Real-world applications of AI - Daniel Hulme @ PAPIs Connect
Real-world applications of AI - Daniel Hulme @ PAPIs ConnectReal-world applications of AI - Daniel Hulme @ PAPIs Connect
Real-world applications of AI - Daniel Hulme @ PAPIs Connect
 
Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...
Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...
Past, Present and Future of AI: a Fascinating Journey - Ramon Lopez de Mantar...
 
Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...
Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...
Revolutionizing Offline Retail Pricing & Promotions with ML - Daniel Guhl @ P...
 
Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect
Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs ConnectDemystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect
Demystifying Deep Learning - Roberto Paredes Palacios @ PAPIs Connect
 
Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
Predictive APIs: What about Banking? - Natalino Busa @ PAPIs ConnectPredictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
 
Microdecision making in financial services - Greg Lamp @ PAPIs Connect
Microdecision making in financial services - Greg Lamp @ PAPIs ConnectMicrodecision making in financial services - Greg Lamp @ PAPIs Connect
Microdecision making in financial services - Greg Lamp @ PAPIs Connect
 
Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...
Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...
Engineering the Future of Our Choice with General AI - JoEllen Lukavec Koeste...
 
Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...
Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...
Distributed deep learning with spark on AWS - Vincent Van Steenbergen @ PAPIs...
 
How to predict the future of shopping - Ulrich Kerzel @ PAPIs Connect
How to predict the future of shopping - Ulrich Kerzel @ PAPIs ConnectHow to predict the future of shopping - Ulrich Kerzel @ PAPIs Connect
How to predict the future of shopping - Ulrich Kerzel @ PAPIs Connect
 
The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...
The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...
The emergent opportunity of Big Data for Social Good - Nuria Oliver @ PAPIs C...
 
Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...
Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...
Automating Machine Learning Workflows: A Report from the Trenches - Jose A. O...
 
Machine Learning Services Benchmark - Inês Almeida @ PAPIs Connect
Machine Learning Services Benchmark - Inês Almeida @ PAPIs ConnectMachine Learning Services Benchmark - Inês Almeida @ PAPIs Connect
Machine Learning Services Benchmark - Inês Almeida @ PAPIs Connect
 
Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...
Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...
Machine Learning Performance Evaluation: Tips and Pitfalls - Jose Hernandez O...
 
How to Build a Successful Data Team - Florian Douetteau @ PAPIs Connect
How to Build a Successful Data Team - Florian Douetteau @ PAPIs ConnectHow to Build a Successful Data Team - Florian Douetteau @ PAPIs Connect
How to Build a Successful Data Team - Florian Douetteau @ PAPIs Connect
 
Microdecision Making in Financial Services - Greg Lamp @ PAPIs Connect
Microdecision Making in Financial Services - Greg Lamp @ PAPIs ConnectMicrodecision Making in Financial Services - Greg Lamp @ PAPIs Connect
Microdecision Making in Financial Services - Greg Lamp @ PAPIs Connect
 
[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
[Keynote] Predictive APIs: What about Banking? - Natalino Busa @ PAPIs Connect
 
Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...
Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...
Real-world Applications of Artificial Intelligence - Daniel Hulme @ PAPIs Con...
 
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
 

Recently uploaded

Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_planJamie (Taka) Wang
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 

Recently uploaded (20)

Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_plan
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 

Solving a business problem in 2 weeks using machine learning — Lucas Fonseca Navarro and Pedro Suguimoto (Getninjas) @PAPIs Connect — São Paulo 2017

  • 1. Solving a business problem in just two weeks using machine learning Lucas F Navarro and Pedro Suguimoto lucas.navarro@getninjas.com.br pedro.suguimoto@getninjas.com.br PAPIs Connect - São Paulo 2017
  • 2. GetNinjas’ Data Science Squad Product Owner Bs’ in Computer Science at UFSCar Ms’ in Artificial Intelligence at UFSCar Data Scientist Mechatronics Engineer from USP/CAASO Data Scientist Bs’ in Applied Maths at UNICAMP PhD’ in Artificial Intelligence at USP/POLI Python Developer Bs’ in Information System at USP/EACH BackEnd Developer Computer Engineer from USP/POLI
  • 3. Agenda 1. How does GetNinjas Work 2. The current Business Model and why we chose it 3. The big problem that has emerged 4. How we solved this problem in just two weeks 5. Results we achieved 6. Touchpoints
  • 4. How does GetNinjas work? (Client) 1. Choose what kind of service that you need. 2. We indicate up to 4 pros at your region 3. Hire a Pro to solve your problem
  • 5. How does GetNinjas work? (Pro) 1. Choose a service request from the store 2. Use credits to unlock the contact of the client 3. Get in touch with the client to be hired
  • 6. OVERVIEW 2,000,000 services hired in the last year 200,000 professionals registered R$ 210M+ in transactions concluded (GMV) over the last 12 months 100+ categories ranging from electrician to english teacher to lawyer and so on...
  • 7. We were performing a staged rollout of a new pay-per-lead model, with significant improvements to the pros' experience Old Model Subscription - Pro Pays Fixed Value Per Month - Receives up to 30 contacts New Model Bundle - Pro Purchases a Bundle of credits - Use credits to purchase contacts Vs First Launch: South (Sep/18) Second Launch: North + Center (Oct/23) At last: SP & RJ (2017) Main improvements: ● Pros can purchase how many contacts they want, when they want ● Clients will always receive interested Pros ● More feasible to balance Supply & Demand
  • 8. Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered Next launch was in two weeks!!! 29% of the requests were not being sold We would have 43.500 clients/month without response (Around 0.5M per year) We made quantitative and qualitative analyses to figure out what causes this rate Some of the possible reasons: ○ Price ○ Location ○ Clients only estimating price ○ Requests lacking a good description ○ Miscategorized services ○ Not enough pros with credits How to solve all of this in two weeks? Our next launch was much bigger and we could not leave our clients unanswered.
  • 9. Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered Next launch was in two weeks!!! 29% of the requests were not being sold We would have 43.500 clients/month without response (Around 0.5M per year) We made quantitative and qualitative analyses to figure out what causes this rate Some of the possible reasons: ○ Price ○ Location ○ Clients only estimating price ○ Requests lacking a good description ○ Miscategorized services ○ Not enough pros with credits How to solve all of this in two weeks? Our next launch was much bigger and we could not leave our clients unanswered.
  • 10. Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered Next launch was in two weeks!!! 29% of the requests were not being sold We would have 43.500 clients/month without response (Around 0.5M per year) We made quantitative and qualitative analyses to figure out what causes this rate Some of the possible reasons: ○ Price ○ Location ○ Clients only estimating price ○ Requests lacking a good description ○ Miscategorized services ○ Not enough pros with credits How to solve all of this in two weeks? Our next launch was much bigger and we could not leave our clients unanswered.
  • 11. The problem gets bigger! Each point is a “Mini GetNinjas”, with 100 categories 300 cities 100 categories x 30k pairs which we have to deal with that problem
  • 12. What if we could predict that a client request wouldn’t be purchased? 1st option, tell the client the truth “Unfortunately we don’t have Pros available to solve your problem at this moment” 2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔ “GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url” 11% of the Free Pros were purchasing a bundle after seeing the services store at our App Only 4% of our subscribed pros were payees at that time We had the potential to more than double our payees base if we sustained that conversion rate
  • 13. What if we could predict that a client request wouldn’t be purchased? 1st option, tell the client the truth “Unfortunately we don’t have Pros available to solve your problem at this moment” 2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔ “GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url” 11% of the Free Pros were purchasing a bundle after seeing the services store at our App Only 4% of our subscribed pros were payees at that time We had the potential to more than double our payees base if we sustained that conversion rate
  • 14. What if we could predict that a client request wouldn’t be purchased? 1st option, tell the client the truth “Unfortunately we don’t have Pros available to solve your problem at this moment” 2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔ “GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url” 11% of the Free Pros were purchasing a bundle after seeing the services store at our App Only 4% of our subscribed pros were payees at that time We had the potential to more than double our payees base if we sustained that conversion rate
  • 15. How could we predict that a client request wouldn’t be purchased? We had already figured out the reasons that a service was not sold, so we used it as features and started to test some machine learning models using R. Main R packages used: For data prep: ○ tidyr ○ dplyr ML algorithms: ○ monmlp (neural net) ○ e1071 (naivebayes) ○ party (decision trees) Utilities: ○ ROCR (ROC curve) ○ SDMTools (confusion matrix) ○ Boruta (feature selection) ○ Pool + RPostgreSQL (integration for our data warehouse - RedShift) Some of the features: ● Number of Pros with credits* ● Avg Distance from pros* ● Number of highly reviewed pros* ● Price ● Does service has a text description? Class: Was the request sold? (boolean) We used Rstudio Server remotely from a m4.large EC2 machine at Amazon’s Cloud *Just pros inside the request area (inside a radius)
  • 16. From R to Python, Fujin was Born! We came up with a Neural Network with great performance: ○ With this specificity rate, just 30% of the unsold requests clients would remain without an answer ○ With this sensitivity rate, we would distribute only 10% of the good requests for free After we validated the model, we implemented it in python to go live: ○ We used the basic DS set (scikit-learn, pandas, numpy) ○ Alse we used Neurolab for the MLP algorithm ○ We used Flask to create the prediction microservice 70% of specificity First Model 79% of accuracy *Python is the main Data Science language at GetNinjas 90% of sensitivity We called the project Fujin!
  • 17. How fast this entire project was delivered? To have a functional ML model to predict unpurchased requests we spent just around 2 weeks! *The first distribution algorithm was a random distribution to Free Pros
  • 18. In Its first month live, Fujin were able to predict 68% of the unpurchased requests Unpurchased Predicted/ Distributed Contacted Hired 41% 28% 9% 5% Service Requests With north+center regions, unpurchased request rate grown from 29% to 41% due sparsity of S&D in this new regions With Fujin online, the number of unanswered clients were just 13% (instead of 41%) Fujin also helps to increase our Gross Merchandise Volume(GMV) through hiring rate *Numbers from our requests of November/2016
  • 19. Fujin was also helping to activate Free Pros into Payees *Numbers from our requests of November/2016 ≈R$25k in purchases from Pros after they received a free client contact from Fujin 65% of this pros didn’t even had our App installed ⚠ Only 14% of the distributed leads were read by the Pros Already had our App installed Downloaded App post Fujin lead Didn`t Download
  • 20. Towards to the Future ● The distribution algorithm ○ Pick the best pros to send free requests ○ Determine the amount of free pros to send the contact from a request given the Fujin’s feedback ● Enhance the model with more features ○ We have more service categories with distinct characteristics nowadays ○ We have SP and RJ state (higher density regions than before) ● Features to increase the visualization rate of free distributed requests ○ Such as adding a deeplink to our the contact screen on our App at the e-mail/sms sent to Pros
  • 21. Touchpoints ● The non-bureaucratic/non-hierarchical organization of a StartUp company was crucial for us to develop Fujin in such short time ● ML is amazing, in just two weeks we had a running model to solve the biggest part of our problem ● ML is super cool, but most of the times we solve our problems with much simpler approaches ⚠ Do not create a super complicated solution and later adjust your problem to it, start from the problem, and climb to the complex solution as you need it.
  • 22. Thanks for your attention!!! Questions?