Since Data Science became a very popular field, machine learning (ML) algorithms are being used more than ever in the industry. ML techniques are amazing to solve a bunch of complex business problems efficiently and also in a very fast manner with all of the available tools that we have nowadays. In this presentation, we will show you how we designed and implemented an ML application to solve an emergent problem in just two weeks. We want to show how a dynamic and pro-creativity environment of a StartUp company combined with machine learning can be powerful to create efficient solutions.

1. Solving a business problem in just two weeks
using machine learning
Lucas F Navarro and Pedro Suguimoto
lucas.navarro@getninjas.com.br
pedro.suguimoto@getninjas.com.br
PAPIs Connect - São Paulo 2017

2. GetNinjas’ Data Science Squad
Product Owner
Bs’ in Computer Science at UFSCar
Ms’ in Artificial Intelligence at
UFSCar
Data Scientist
Mechatronics Engineer from
USP/CAASO
Data Scientist
Bs’ in Applied Maths at UNICAMP
PhD’ in Artificial Intelligence at
USP/POLI
Python Developer
Bs’ in Information System at
USP/EACH
BackEnd Developer
Computer Engineer from
USP/POLI

3. Agenda
1. How does GetNinjas Work
2. The current Business Model and why we chose it
3. The big problem that has emerged
4. How we solved this problem in just two weeks
5. Results we achieved
6. Touchpoints

4. How does GetNinjas work?
(Client)
1. Choose what kind
of service that you
need.
2. We indicate up to 4
pros at your region
3. Hire a Pro to solve
your problem

5. How does GetNinjas work?
(Pro)
1. Choose a service
request from the
store
2. Use credits to unlock
the contact of the client
3. Get in touch with
the client to be hired

6. OVERVIEW
2,000,000
services hired in the last year
200,000
professionals registered
R$ 210M+
in transactions concluded (GMV) over the
last 12 months
100+
categories ranging from electrician to
english teacher to lawyer and so on...

7. We were performing a staged rollout of a new pay-per-lead model, with
significant improvements to the pros' experience
Old Model
Subscription
- Pro Pays Fixed
Value Per Month
- Receives up to
30 contacts
New Model
Bundle
- Pro Purchases a
Bundle of credits
- Use credits to
purchase contacts
Vs
First Launch:
South (Sep/18)
Second Launch:
North + Center
(Oct/23)
At last:
SP & RJ (2017)
Main improvements:
● Pros can purchase how many contacts they
want, when they want
● Clients will always receive interested Pros
● More feasible to balance Supply & Demand

8. Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
two weeks!!!
29%
of the requests were not
being sold
We would have 43.500
clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure
out what causes this rate
Some of the possible reasons:
○ Price
○ Location
○ Clients only estimating price
○ Requests lacking a good description
○ Miscategorized services
○ Not enough pros with credits
How to solve all of this in two weeks?
Our next launch was much bigger and we could not leave our
clients unanswered.

9. Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
two weeks!!!
29%
of the requests were not
being sold
We would have 43.500
clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure
out what causes this rate
Some of the possible reasons:
○ Price
○ Location
○ Clients only estimating price
○ Requests lacking a good description
○ Miscategorized services
○ Not enough pros with credits
How to solve all of this in two weeks?
Our next launch was much bigger and we could not leave our
clients unanswered.

10. Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
two weeks!!!
29%
of the requests were not
being sold
We would have 43.500
clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure
out what causes this rate
Some of the possible reasons:
○ Price
○ Location
○ Clients only estimating price
○ Requests lacking a good description
○ Miscategorized services
○ Not enough pros with credits
How to solve all of this in two weeks?
Our next launch was much bigger and we could not leave our
clients unanswered.

11. The problem gets bigger!
Each point is a “Mini
GetNinjas”, with
100 categories
300 cities
100 categories
x
30k pairs
which we have to deal with that
problem

12. What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have
Pros available to solve your
problem at this moment”
2nd option, let’s give this services for free to signed
up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician
service request near your location! Get the
contact in “My Requests” tab at our app: %url”
11%
of the Free Pros were
purchasing a bundle after
seeing the services store at our App
Only 4% of our subscribed pros were
payees at that time
We had the potential to more than
double our payees base if we sustained
that conversion rate

13. What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have
Pros available to solve your
problem at this moment”
2nd option, let’s give this services for free to signed
up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician
service request near your location! Get the
contact in “My Requests” tab at our app: %url”
11%
of the Free Pros were
purchasing a bundle after
seeing the services store at our App
Only 4% of our subscribed pros were
payees at that time
We had the potential to more than
double our payees base if we sustained
that conversion rate

14. What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have
Pros available to solve your
problem at this moment”
2nd option, let’s give this services for free to signed
up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician
service request near your location! Get the
contact in “My Requests” tab at our app: %url”
11%
of the Free Pros were
purchasing a bundle after
seeing the services store at our App
Only 4% of our subscribed pros were
payees at that time
We had the potential to more than
double our payees base if we sustained
that conversion rate

15. How could we predict that a client request wouldn’t be purchased?
We had already figured out the reasons that a
service was not sold, so we used it as features and
started to test some machine learning models using R.
Main R packages used:
For data prep:
○ tidyr
○ dplyr
ML algorithms:
○ monmlp (neural net)
○ e1071 (naivebayes)
○ party (decision trees)
Utilities:
○ ROCR (ROC curve)
○ SDMTools (confusion matrix)
○ Boruta (feature selection)
○ Pool + RPostgreSQL (integration
for our data warehouse - RedShift)
Some of the features:
● Number of Pros with credits*
● Avg Distance from pros*
● Number of highly reviewed pros*
● Price
● Does service has a text description?
Class: Was the request sold? (boolean)
We used Rstudio Server remotely from a m4.large EC2 machine at Amazon’s Cloud
*Just pros inside the request area (inside a radius)

16. From R to Python, Fujin was Born!
We came up with a Neural Network with great performance:
○ With this specificity rate, just 30% of the unsold requests clients
would remain without an answer
○ With this sensitivity rate, we would distribute only 10% of the
good requests for free
After we validated the model, we implemented it in python to
go live:
○ We used the basic DS set (scikit-learn, pandas, numpy)
○ Alse we used Neurolab for the MLP algorithm
○ We used Flask to create the prediction microservice
70%
of specificity
First Model
79%
of accuracy
*Python is the main Data Science language at GetNinjas
90%
of sensitivity
We called the project Fujin!

17. How fast this entire project was delivered?
To have a functional ML model to predict unpurchased requests we spent just around 2 weeks!
*The first distribution algorithm was a random distribution to Free Pros

18. In Its first month live, Fujin were able to predict 68% of the unpurchased requests
Unpurchased
Predicted/
Distributed
Contacted
Hired
41%
28%
9%
5%
Service
Requests
With north+center regions, unpurchased request
rate grown from 29% to 41% due sparsity of
S&D in this new regions
With Fujin online, the number of unanswered
clients were just 13% (instead of 41%)
Fujin also helps to increase our Gross
Merchandise Volume(GMV) through hiring rate
*Numbers from our requests of November/2016

19. Fujin was also helping to activate Free Pros into Payees
*Numbers from our requests of November/2016
≈R$25k
in purchases from Pros after they
received a free client contact from Fujin
65%
of this pros didn’t even had our App
installed
⚠ Only 14% of the distributed leads
were read by the Pros
Already had our App installed Downloaded App post Fujin lead Didn`t Download

20. Towards to the Future
● The distribution algorithm
○ Pick the best pros to send free requests
○ Determine the amount of free pros to send the contact from a request given the Fujin’s
feedback
● Enhance the model with more features
○ We have more service categories with distinct characteristics nowadays
○ We have SP and RJ state (higher density regions than before)
● Features to increase the visualization rate of free distributed requests
○ Such as adding a deeplink to our the contact screen on our App at the e-mail/sms sent to Pros

21. Touchpoints
● The non-bureaucratic/non-hierarchical organization of a StartUp company was
crucial for us to develop Fujin in such short time
● ML is amazing, in just two weeks we had a running model to solve the biggest
part of our problem
● ML is super cool, but most of the times we solve our problems with much simpler
approaches
⚠ Do not create a super complicated solution and later adjust your problem to it, start from
the problem, and climb to the complex solution as you need it.

22. Thanks for your attention!!!
Questions?