Scout24 aims to become a fully data-driven company, providing a comprehensive data platform while establishing clear guidelines and responsibilities for data producers and consumers. The organization emphasizes data autonomy, transparency, and the importance of quality data management, with a structured approach to data governance. The document outlines the technological and cultural evolution of their data landscape, highlighting the need for robust communication and collaborative tools for effective participation in data-driven initiatives.

1. www.scout24.com
The Scout24 Data Landscape Manifesto:
Building an Opinionated Data Platform
Predictive Analytics World Berlin | 13.11.2018 | Sean Gustafson

2. 5
Core Geographies
and an overall presence
in 18 countries
80m
Household Reach
2
Major Household Brand Names
Scout24 AG
• SDAX
• € 489 million revenue (2017)

3. Our technical evolution

4. How we started in 2007
BI Tool
Middle
Tier
DWH
Staging
Core
DB
CRM
2007
Web
Tier
Analyst
BI Dev

5. How things got complicated in 2011
BI Tool
Middle
Tier
DWH
Staging
Core DB
CRM
Web
2011
API
APP
$$$
APPMySQL
Analyst
BI Dev

6. APPMySQL
APPMySQL
APPMySQL
How we sliced the monolith in 2013
BI Tool
DWH
StagingCRM
Web
2013
API
APPMySQL
Core
DB
EXP
Mongo
SEA
Elastic
Sync APP
APIAPI
API
HADOOP
REST API
Analyst
BI Dev
DE

7. AWS
APP
APP
APP
APPMySQL
APPMySQL
APPMySQL
How a central data team doesn’t scale
BI Tool
DWH
StagingCRM
Web
2015
API
APPMySQL
Core
DB
EXP
Mongo
SEA
Elastic
Sync APP
APIAPIAPI
HADOOP
REST API
APPAPP
Analyst
BI Dev
DE

8. Core
DB
APPAPPAPPAPPAPPAPPAPPAPPAPP
AWS
How we re-architected our Data Landscape
BI Tool
Presto
Central Data Lake on S3
CRM
2017
Core
DB APP
REST API
APPAPPAPP
Analyst
BI Dev
DE
DS
Spark, R, etc.

9. Our organizational evolution

10. Core
DB
APPAPPAPPAPPAPPAPPAPPAPPAPP
AWS
BI Tool
Presto
Central Data Lake on S3
CRM
2017
Core
DB APP
REST API
APPAPPAPP
Analyst
BI Dev
DE
DS
Spark, R, etc.

11. Analyst
BI Dev
DE
DS
Data Platform Engineering
Analysts (in residence)
Data Scientists (in residence)
Analyst
Central Analysts Team

12. Scout24 wants to become a truly data-driven company
Fast & easy data-driven
product development…
…supported by
Data & Analytics

13. Scout24 wants to become a truly data-driven company
Everywhere in the company... ...without bloating up DnA

14. Our cultural revolution

15. SCOUT24
DATA LANDSCAPE
MANIFESTO
ROLES, RESPONSIBILITIES, AND VALUES
FOR A DATA-DRIVEN COMPANY AT SCALE

16. Data is a key asset of our
company.
#1 Preamble

17. #2 Our Responsibility
We, Data & Analytics, are
responsible for providing a
solid Data Platform as well
as clear guidelines and
training how to participate
in the Data Landscape. Data Platform
DnA
Data Landscape

18. #3 Data Autonomy, Not Anarchy
Data autonomy puts data producers
& data consumers in control of
their data & of their metrics and
thereby allows us to be data-driven
at scale, but this comes with
responsibility. Data Platform
Data
Producer Consumer
DnA
Data Landscape

19. #4 Producer’s Responsibility
Data producers are responsible for
publishing data to the central Data
Lake, for the data's quality, and for
publishing metadata that makes it
easy to find and consume the data.
Data Platform
Metadata
Data
Producer
DnA
Data Landscape

20. #5 Consumer’s Responsibility
Data consumers are responsible for
the definition & visualization of
metrics and for driving the
implementation and maintenance of
these metrics.
Data Platform
Producer Consumer
DnA
Data Landscape

21. #6 Exception: Core KPIs
We, Data & Analytics, take the
full ownership and responsibility
of the few top company-wide
core KPIs.
Data Platform
Producer Consumer
DnA
Data Landscape
Core
metric

22. #7 Transparency Over Continuity
We value data transparency over
data continuity, which means
we may break metric
comparability if it is for the
cause of enabling better insights.
Data Platform
Producer Consumer
DnA
Data Landscape
Core
metric

23. The Ultimate Goal
Data Platform
Metadata
Data
Producer Consumer
DnA
Data Landscape
Core
metric
A federal landscape of data
producers and consumers with just
enough rules to ensure seamless
co-operation without severely
impeding autonomy.

24. Centralized Federated
Control Autonomy
Perfection Scale
Pull Push
Product is Data Product is Platform
Reporting Reporting, Ad hoc Analytics,
Machine Learning
Data Warehouse vs. Data Platform

25. How to convince them to go along?
à ‘Nudge’ them to participate
à Promote the platform
à Refuse new use cases in Data Warehouse
Result:
Product teams have much higher responsibility

26. Design ‘nudges’ into the Platform
Make Data Lake easier than something else:
- automatic table publishing, partition detection
- backup and disaster recovery
- access control for restricted data
- optimize file formats (e.g. parquet) for efficiency

27. Learnings and lessons
à Change needs to be technological, organizational
and cultural
à Build features to give benefits that counteract resistance
à Communication is the key

28. Have a strong opinion about
how your company should use data and
build a platform that pushes toward that vision.
Most importantly: