Nowadays, every data-driven company needs a solid data platform to exploit the hidden value of business data. However, to build a centralized self-service data platform that consolidates information from all business units can be challenging due to real-time constraints, data ownership, privacy, usability, maintainability, scalability and cost efficiency. At Letgo we built a cloud data platform embracing all these principles and became a company strategic pillar.

1. How to build a self-service data platform and what it can do for your business
Roger Rafanell & Vito Ressa

2. ● About us
● letgo tech platform overview
● Why a Data Platform?
● Data in-motion
● Data at rest
● Data exploitation
● Conclusions
● Q&A
Topics

3. About us

4. 4
3
Founded in 2015
Phenomenal growth
Focused on Turkey
Offices in Barcelona
and Istanbul

5. letgo tech
platform

6. 6
From Monolith to Microservices
2015 - 2016

7. 7
From Monolith to Microservices
2015 - 2016
2017 - now

8. 8
Event-driven architecture
Event bus
Data
Analytics
Microservice
Microservice
Data Lake
Microservice

9. 9
Event-driven architecture
Search
Microservice
Chat
Microservice
...
Entities
persistence
Entities
persistence
Entities
persistence
Event
bus
search-results
message-sent
domain event
Other
Microservice
Entities
persistence

10. {
"data":{
"attributes":{
"product_ids":[
"b708b091-af61-48d6-9530-b9a2fd529d75",
"..."
],
"search_term":"Honda Civic 2005",
"limit":50,
"filters":{
"country_code":"TR",
"max_price":5360,
"min_price":null
},
"search_radius_km":200,
"latitude":39.925533,
"longitude":32.866287
},
"type":"search-results"
},
"meta":{
"created_at":1559812586250,
"service":search
}
}
10
Events example
search-results
{
"data":{
"id":"294d14e2-3876-4038-aa31-01ca3fd6a255",
"type":"message-sent",
"attributes":{
"conversation_id":"5fe4f018-...-664664da976c",
"product_id":"X75gJ1",
"from_talker_id":"SFNPGFOnYN",
"to_talker_ids":[
"19107167-e470-47c5-8df6-6f1e07d12b53",
],
"message_id":"d2fd0c80-...-80ae58a772d3",
"type":"interested",
"message":"Hi Anna, how are you doing?",
"belongs_to_spam_conversation":false,
"sent_at":1562245794667
}
},
"meta":{
"created_at":1562222991053,
"service":"chat"
}
}
message-sent

11. Why a Data
Platform?

12. 12
12
Why a Data Platform?
● Ensure core principles for data in-motion and data at rest:
○ Data quality through structural validation with schemas
○ Privacy as a Service by anonymization and data access controls
○ Security with infrastructure, data isolation and auditability
○ Compliance in Data Protection Regulations services
○ Cost efficiency through applying data management best practices
○ Reusability with low/easy maintenance by design

13. 13
Data platform key pillars
● Self-service data platform will act as custodian of the data not as owners.
● Any modification on data asset should only happen on the source.
● Employees should only access sensitive data required for their business needs.
● Sensitive data usage should be regulated.

14. 14
Data platform layers
Data at rest
A centralized isolated
storage to collect the
data assets. Only the
processes needed for
data privacy
regulations have direct
access to the data.
Data in-motion
Data
exploitation
Environments dedicated to teams
data consumption. Built around
the self-service platform concept
with only the data each team is
entitled to use to fulfill their
needs.
The backbone for the
microservices to exchange
events. Real time data validation
and governance.

15. Data in-motion

16. 16
Data in-motion design
Search
Microservice
Chat
Microservice
...
Entities
persistence
Entities
persistence
Entities
persistence
Event
bus
search-results
message-sent
domain event
Other
Microservice
Entities
persistence
data
in-motion
layer
Validation
&
Governance

17. 17
17
● User privacy should be ensured and their data kept secure
Data Governance

18. 18
Personal Identifiable information

19. 19
Identifying personal information challenge
"data":{
"id":"00000-00000",
"type":"message_sent" ,
"conversation_id":"11111-11111",
"from_user":"luke",
"to_user":"han",
"user_ip":"1.1.1.1",
"content":"Hello!",
"sent_at": 1566901277000
}
"data":{
"id":"00000-00000",
"type":"account_updated" ,
"account_id":"11111-11111",
"email":"luke@example.com" ,
"reason":"whatever",
"updated_at":1566901277000
}
Two user ids? No user id at all?
● In events the same field can have different names in different events types
● Not all events contain a user_id, need to join with other events

20. 20
Data schemas
● We adopted Apache Avro as data
serialization protocol
● Schemas enforce the structure of
events
● Evolvable schemas data structure
● Polyglot: Java, Scala, JS, Python,
C#...
{
"type": "record",
"namespace": "com.tidx",
"name": "ProductVisit",
"fields": [
{
"name": "product_id",
"type": "string",
"doc": "Identifier of
the visited product"
},
...

21. 21
Personal Information Tagging
● We extended Avro with custom
tags in the schemas to identify PII
in data sources.
● Identifying PII in the schema is
responsibility of the team
producing the event.
{
"name": "latitude",
"type": [
"null",
{
"name": "latitude",
"type": "float",
"arg.properties": {
"range": { "min": -90.0, "max": 90.0 }
}
}
],
"doc": "Latitude",
"default": null,
"letgo.properties": {
"entity": "user/location"
}
}

22. 22
Validation and governance in real-time
Event Avro schema Data Governance rules
Example:
user_registered
Validation & Governance
user_registered

23. 23
Data in-motion
Search
Microservice
Chat
Microservice
...
Entities
persistence
Entities
persistence
Entities
persistence
Event
bus
search-results
message-sent
domain event
data in-motion layer
Validation
&
Governance
dead-letter-*
events-valid-json-*
events-valid-*
CI/CD
Schema
Registry
DataGov
DB
Teams
DPO
Other
Microservice
Entities
persistence
Data at rest

24. Data at rest

25. 25
Data at rest
Data at rest layer acts as:
● Non-anonymized
events repository
● Privacy compliance
services source
● Isolated, secured data
hub for data
exploitation platform

26. Data exploitation

27. 27
Data exploitation
The data exploitation layer enables the data consumption for all teams and is build
around the following principles:
● Self-service platform to empower teams to autonomously self-operate their services.
● Standards to promote the “freedom in a sandbox” concept to add common guardrails.
● Built-in security & compliance by providing access-roles, data retention & security policies.
● Observability by providing monitoring, alerting and costs tracking.
We developed those principles around a concept called Reservoir.

28. 28
Reservoir
● Traversal common infrastructure
○ Integration with infra common services layer
■ Cloud automation, containers platform
■ Networking, secrets management services, etc.
● Core module
○ Basic reservoir roles & SSO integration (Okta)
○ Enterprise resource tagging standards
○ Built-in data retention & security policies
○ Monitoring & alerting integration
○ Optional capabilities (storage, etc…)
● Services modules
○ Use case specific standard infrastructure components (modules)
■ Community modules
■ Custom modules / ensembles
Reservoir

29. 29
Reservoirs overview (services)
Traversal Infrastructure
Team environments (Reservoirs)
Data at rest
Data Science team
Business Intelligence team
Compliant
data flows

30. 30
BI Reservoir deployment example
Traversal Infrastructure
Team environments
(Reservoirs)
Business Intelligence team
Compliant
data flows
Steps
● Transversal infrastructure updated with few lines of
config in Terraform
● Reservoir core deployment and data flows config via
Terraform variables and Data Governance rules
● OKTA roles mapping
● Custom services provisioned through plug-and-play
Terraform modules
1
1
3
4
2
3
4
2
2

31. 31
Why is this good for my business?
For the team For the company
● Self-service low effort adoption
● Clean, high quality data ready to use
● Infrastructure and services ownership
● Transparent built in company standards
● Data governance applied by design
● Teams are accountable for their costs
● Scalability: teams autonomy due to
minimal services sharing in Reservoirs
● Best practices and standards propagated
across all teams

32. Conclusions

33. 33
Summary
Data
Governance
Data
in-motion
Data
at rest
COMPLIANT
Microservices
data exploitation
DS
BI
Main advantages
● Costs
○ Teams are accountable for their costs
○ Teams only keeping the data they need
● Maintainability
○ Mostly AWS managed services
○ Reusable infra templates (build once use many)
● Compliance
○ Governance rules applied to all data layers
○ Fine-grained data anonymization/filtering
○ Centralized access based policies SSO (Okta)
● Usability
○ Easier and faster to capture, process and
exploit new data assets
reservoirs

34. 34
letgo data at scale
30TB
Data Monthly
500M
Events Processed Daily
13K
Events Processed per Second
900+
Event Types
550TB
Storage (S3)
5+
Data Sources

35. 35
Team:
4 x Data Engineers
1 x Site Reliability Engineer
Open Positions:
1x Data Engineer
We are hiring!
Data Engineering Data Science Business Intelligence
https://www.linkedin.com/company/letgo/jobs/
https://careers.olxgroup.com
Team:
3 x Data Scientists
1 x ML Engineer
Team:
1 x Reporting Analysts
2 x TnS Analysts
2 x Data Analysts
Open Positions:
1x BI Analyst

36. Q&A

37. Thank you
roger.rafanell@letgo.com
vito.ressa@letgo.com