GetNinjas slides for the first Snowplow Meetup in Brazil :)

1. Snowplow Meetup @ GetNinjas
Bernardo Srulzon
bernardo@getninjas.com.br

2. How does it work?
1. Tell us what service
you need
2. We’ll match you with
up to 4 pros
3. See the reviews and
hire the best :)

3. OVERVIEW
1,500,000
services requested in the last 12 months
200,000
registered pros
R$ 190M+
in transactions (GMV), last 12 months
100+
categories, from electricians to wedding
photographers

4. OVERVIEW
Each dot is a
“mini GetNinjas”,
with 100 categories
300 main cities
100 categories
x
30k combinations
for which we have to maintain a
healthly supply vs. demand balance

5. #1
In this context, what should be the
role of Business Intelligence?

6. We’re focused on creating a great experience for clients and pros; to achieve
that, we measure each step of the customer lifecycle
Pirate Metrics
Pirate Metrics by Dave McLure @ 500 Startups

7. We’re focused on creating a great experience for clients and pros; to achieve
that, we measure each step of the customer lifecycle
Pirate Metrics
Are we balancing supply and demand correctly?
What acquisition channels should are more cost-effective?
Pirate Metrics by Dave McLure @ 500 Startups

8. We’re focused on creating a great experience for clients and pros; to achieve
that, we measure each step of the customer lifecycle
Pirate Metrics
Are we matching clients with great pros?
Can pros close their first service a week after signup?
Pirate Metrics by Dave McLure @ 500 Startups

9. We’re focused on creating a great experience for clients and pros; to achieve
that, we measure each step of the customer lifecycle
Pirate Metrics
Will clients come back to request a service in a different category?
Are pros consistently interacting with clients & closing services?
Pirate Metrics by Dave McLure @ 500 Startups

10. We’re focused on creating a great experience for clients and pros; to achieve
that, we measure each step of the customer lifecycle
Pirate Metrics
Do clients share the experience and bring new clients?
Are successful pros referring us to their colleagues?
Pirate Metrics by Dave McLure @ 500 Startups

11. We’re focused on creating a great experience for clients and pros; to achieve
that, we measure each step of the customer lifecycle
Pirate Metrics
Is the lead price fair, and bring significant ROI to pros?
Are we extracting the maximum value from our leads?
Pirate Metrics by Dave McLure @ 500 Startups

12. If the Business Intelligence role is to answer complex questions, the traditional
mindset doesn’t really work...
A more traditional BI team would...
Collect data
Build reports
Send report to decision-maker
Sense of ownership Business impact Autonomy Out-of-the-box thinking

13. Our role is to transform data into actionable insights; in practice, we start from a
hypothesis and participate in the entire decision process
Data science only matters if data drives action
Jeremy Stanley, VP Data Science @ Instacart
Hypothesis Exploration & Validation Decision
Team start from a hypothesis
We should adjust our pricing so
that pros can close at least one
job on their first month in the
platform. This should increase our
retention.
Come up with a plan to validate it
Analyzing historical data for
closed jobs and applying a
statistical model, we should be
able to determine the pricing
options
Participate in decision-making
We concluded that the least
expensive plan should allow pros
to purchase 30 leads. We’ll move
forward with the implementation
and monitor results

14. #2
From 20k to 170k service requests
How did we scale?

15. In the past 3 years, we evolved our focus and structure to reflect the company
priorities
2014 2
Focus
Product-market-fit: making sure
our product actually solves a
market pain point
• Google Analytics setup
• First version of the matching
algorithm
• Analysis focused on
transactional data

16. In the past 3 years, we evolved our focus and structure to reflect the company
priorities
2014 20152 4
Focus
Product-market-fit: making sure
our product actually solves a
market pain point
• Google Analytics setup
• First version of the matching
algorithm
• Analysis focused on
transactional data
Focus
Have a deeper understanding of
how clients and pros interact with
the platform. Question the details.
• Got stuck with GA limitations,
went forward with Snowplow
• Invested a lot of energy in
training the team
• Split the team workload into
acquisition & closing

17. In the past 3 years, we evolved our focus and structure to reflect the company
priorities
2014 2015 2016-2017
Focus
Product-market-fit: making sure
our product actually solves a
market pain point
Focus
Have a deeper understanding of
how clients and pros interact with
the platform. Question the details.
Focus
Build a comprehensive view of
the user, and democratize access
to data
2 4 10
• Google Analytics setup
• First version of the matching
algorithm
• Analysis focused on
transactional data
• Got stuck with GA limitations,
went forward with Snowplow
• Invested a lot of energy in
training the team
• Split the team workload into
acquisition & closing
• Integration with customer
support & sales systems
• Team structured in multi-
functional “squads”
• More complex matching and
marketing algorithms

18. #3
Web analytics +
Data warehousing

19. We deep dive into web analytics data to understand the behavior of clients and
pros
What’s the
conversion rate of
this page?

20. We deep dive into web analytics data to understand the behavior of clients and
pros
What’s the
conversion rate of
this page?
Desktop converts
more than mobile?

21. We deep dive into web analytics data to understand the behavior of clients and
pros
What’s the
conversion rate of
this page?
Desktop converts
more than mobile?
State capitals
convert more vs.
countryside?

22. We deep dive into web analytics data to understand the behavior of clients and
pros
What’s the
conversion rate of
this page?
Desktop converts
more than mobile?
State capitals
convert more than
countryside?
What % of users
load in < 5 sec?

23. We deep dive into web analytics data to understand the behavior of clients and
pros
What % interact
with the form?

24. We deep dive into web analytics data to understand the behavior of clients and
pros
What % interact
with the form?
How much time do
they take?

25. We deep dive into web analytics data to understand the behavior of clients and
pros
What % interact
with the form?
How much time do
they take?
How many have
validation issues?

26. We deep dive into web analytics data to understand the behavior of clients and
pros
What % interact
with the form?
How much time do
they take?
How many have
validation issues?
Are recurrent users
any different?

27. To answer these questions, we need a web analytics structure that allows a very
granular segmentation of data
Channel
Landing
page type
Device A/B Test
Category Funnel
Load TimeCity
Segmentation is key!
Tech Events Classes Consulting
What’s the site conversion rate?
WiFi/3GRecurring
All data in aggregate is crap
Avinash Kaushik

28. Trackings generate insights into the behavior of the group, and the behavior of
each user
Where are the frictions
and what are the
hypotheses?
Visit
Form interaction
2nd step
100%
Interaction on 2nd step
Conversion!
50%
40%
20%
30%
How’s the funnel evolving
over time?
What’s the group behavior?

29. Trackings generate insights into the behavior of the group, and the behavior of
each user
Sign-up (SEO)
Purchased a lead
Received a review
Downloaded app
Purchased credits
Received
onboarding call
E-mail marketing
D+0
D+1
D+5
D+7
D+20
D+25
D+30
Plan (re)marketing
campaigns
Event-based
communication
What do users do before
downloading app?
More info to sales team
What’s the behavior of a specific user?
Debugging
Where are the frictions
and what are the
hypotheses?
Visit
Form interaction
2nd step
100%
Interaction on 2nd step
Conversion!
50%
40%
20%
30%
How’s the funnel evolving
over time?
What’s the group behavior?

30. These complex questions eventually led us to hit the wall with Google Analytics,
and we went out exploring alternatives
Hard to segment data correctly No form tracking Hard to identify users
Limited cross-device tracking
Hard to integrate with other
sources
Can’t apply our business logic
Some of the limitations...

31. These complex questions eventually led us to hit the wall with Google Analytics,
and we went out exploring alternatives
SEM SEO Direct Direct Direct Direct Direct
SEM SEO SEO SEO SEO SEO SEO
Session #1 #2 #3 #4 #5 #6 #7
Channel attribution
Reality
GA was reporting 50% less traffic
compared to what we expected
Hard to segment data correctly No form tracking Hard to identify users
Limited cross-device tracking
Hard to integrate with other
sources
Can’t apply our business logic
Some of the limitations...

32. We decided to implement Snowplow, an open source platform for product
analytics
Snowplow is an enterprise-strength marketing and product analytics platform
https://github.com/snowplow/snowplow
Identifies users, and tracks the way
they engage with the site & app
Stores your users' behavioral data in
a scalable "event data warehouse"
Leverage BI & big data tools to
analyze data
We want to own our data We want to track users on the web and on the app
We want to have a comprehensive view of users We want to answer EVERY question!
Why?

33. We decided to implement Snowplow, an open source platform for product
analytics
Snowplow is an enterprise-strength marketing and product analytics platform
https://github.com/snowplow/snowplow
Beanstalk EMR Redshift Redshift
Identifies users, and tracks the way
they engage with the site & app
Stores your users' behavioral data in
a scalable "event data warehouse"
Leverage BI & big data tools to
analyze data

34. ...at a much lower cost than any other SaaS alternatives

35. We decided to implement Snowplow, an open source platform for product
analytics
Apps
SMS
CRM/Sales
Payments + Transactional data
Sync MySQL -> Redshift
via Amazon DMS
Web
Web/App analytics
via Snowplow
Push
Support
Server-side events
via Snowplow
External integrations
via Stitch
DATA MODEL
300M events
per month
Transforms atomic, unopinionated data into models
that have business logic applied
Email

36. BEFORE | Google Analytics
aggregate conversion data

37. AFTER | Snowplow
highly segmented data
Landing page #1 Landing page #2 Landing page #3
Tech
Support
Events
Home
Services
Home
Improve
ment
Others

38. Redshift e Tableau work great together – connections are super fast and allow
fine segmentation
Billions of Redshift data available
for drag&drop segmentation
Integration with other data
sources (CSV, Excel, MySQL, etc)
Different ways to visualize your
data
Flexibility for customized
calculations & formulas

39. Metabase | Online dashboards
https://github.com/metabase/metabase

40. #4
Team structure & profile

41. We use data to make business decisions, but also to build models & algorithms
Decision Science
2014-2015 2016
Centralized team
Product-market-fit: making sure our product actually solves
a market pain point
Analysts within cross-functional teams
Build a complete view of the user, and democratize access
to data
Focused on making smarter business decision based on
data. Overlaps with the product manager role
Skills
Business + SQL + Excel + Tableau + Basic Programming
Profile
Intern from top Engineering schools
5 Data Science5
Focused on developing models & algorithms to provide a
better customer experience
Skills
Business + Modelling + Programming
Profile
Masters/PhD + few years work experience

42. We’re structured in cross-functional teams, with the specialists meeting every
week to exchange good practices
Tech
Product
Design
BI
Content
“Specialist
alignment” every
week
Cross-functional
teams
Focused on
specific OKRs

43. #5
What’s in the future?

44. We should increasingly empower other areas to create & validate hypotheses,
and make smarter business decisions
Business Intelligence
Build & maintain the data pipelines/infrastructure Model data, applying business logic
Offer training & coaching Continue to create and explore hypotheses
Product Marketing Tech Support & Sales
Decisions focused on the
clients’ and pros’
experience with the app
Budget allocation,
attribution model, return-
on-investment
Page speed, debugging,
cache hits
Team performance,
commission model, lead
scoring

45. Snowplow Meetup @ GetNinjas
Bernardo Srulzon
bernardo@getninjas.com.br