This document discusses strategies for evolving a data platform. It begins by describing some common approaches to data storage and scheduling, including using shared storage, weak scheduling, and monolithic data platforms. It then discusses challenges like availability, schema evolution, and the need for data engineering. Key recommendations include introducing granular reusable pipeline steps, separating pipeline concerns, using schema managers, and separating code and schemas. The goal is to build a reliable, scalable and fault-tolerant data platform that can efficiently handle data and schema changes.

1. COWBOY DATING
WITH BIG DATA
DATA PLATFORM EVOLUTION IN ACTION
BORIS TROFIMOV

2. INTRO – PARTNER 1

3. INTRO – PARTNER 2

4. НАГРАДЫ PARTNER 2
• Медаль за Kotlin
• Полный кавалер Spring & Spring Boot
• Орден за взятие Kubernetes

6. EXPECTATIONS
PRODUCT

7. CHAPTER 1 – SHARED STORAGE

8. UI
MVP
API FACADE
DB
SHARED STORAGE

9. UI
MVP
API FACADE
DB
ГДЕ
репортинг?
SHARED STORAGE

10. UI
MVP
API FACADE
DB
3rd PROVIDERS
SHARED STORAGE

11. PROS
• Fast TTM
• Relatively cheap from infra and cost perspective
CONS
• Tight data and code cohesion
• Different Scaling scenarios
• Performance and Availability issues
SHARED STORAGE

12. SHARED STORAGE
UI
MVP
API FACADE
OLTP
DATA
PLATFORM
OLAP
3rd PROVIDERS

13. DATA PLATFORM
DATA PLATFORM
3rd PROVIDERS
SYSTEM
EVENTS
API FACADEDATA PLATFORM

14. CHAPTER 2 – WEAK SCHEDULING

15. SCHEDULING
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS

16. SCHEDULING
DATA PLATFORM
DATA PLATFORM
3rd
PROVIDERS
SYSTEM
EVENTS
DATA PROCESSING SCRIPT
OLAPDATA PROCESSING SCRIPT
DATA PROCESSING SCRIPT
API FACADE

17. DATA PROCESSING SCRIPT
SCHEDULING
DATA PLATFORM
3rd
PROVIDERS
SYSTEM
EVENTS
CROND
QUARTZ
…
OLAP
DATA PROCESSING SCRIPT
API FACADE
DATA PLATFORM
DATA PROCESSING SCRIPT

18. SCHEDULING
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
КАКОЙ
КРОНТАБ?

19. SCHEDULING
Use prod-ready schedulers
§ Airflow
§ Azkaban
§ Oozie
§ Jenkins?
What we gain
§ Identity control and Audit
§ Job Lineage, Logging and Troubleshooting
§ Tools to design Workflows/DAGs
§ Fault Tolerance features (rerun etc.)

20. CHAPTER 3 –MONOLITHIC DATA PLATFORM

21. DATA PLATFORM
3rd PROVIDERS
SYSTEM
EVENTS
Scheduler
OLAP
API FACADE
DATA PLATFORM
DATA PROCESSING SCRIPT
DATA PLATFORM

22. DATA PROCESSING SCRIPT
3rd PROVIDERS
SYSTEM
EVENTS
Scheduler
OLAP
DATA PLATFORM
ПОЧЕМУ
ОПЯТЬ
МОНОЛИТ?
API FACADE
DATA PLATFORM

23. DATA PLATFORM
DWH ANALYTICSREPORTING
DECOUPLING DATA PLATFORM

24. LOADINGEST
DATA PLATFORM
PROCESS
AirFlow
DATA PLATFORM
3rd PROVIDERS
SYSTEM
EVENTS
DWH
DATA LAKE DWH
DECOUPLING DATA PLATFORM

25. LOADINGEST
DATA PLATFORM
PROCESS
AirFlow
3rd PROVIDERS
SYSTEM
EVENTS
DWH
DATA LAKE DWH
LOADINGEST PROCESS
DECOUPLING DATA PLATFORM

26. ML TRAINING
DATA PLATFORM
ANALYTICS
OLAP
ML RUNNING
AirFlow
3rd PROVIDERS
SYSTEM
EVENTS
DWH
DATA LAKE DWH
LOADINGEST PROCESS
LOADINGEST PROCESS
DECOUPLING DATA PLATFORM

27. DATA PLATFORM
ML TRAINING
DATA PLATFORM
REPORTING
REPORTING ENGINE
ML RUNNING
AirFlow
REPORTING CACHE
REPORTING METADATA
SCHEDULED REPORTS
ANALYTICS
API
FACADE
3rd PROVIDERS
SYSTEM
EVENTS
OLAP
DWH
DATA LAKE DWH
LOADINGEST PROCESS
LOADINGEST PROCESS
DECOUPLING DATA PLATFORM

28. DECOUPLING DATA PLATFORM
ARCHITECTURE DECISIONS
§ Introduce granular reusable steps inside pipelines [ingest, validate, enrich, aggregate
etc.]
§ Separate pipeline per vendor/feed
§ Raw data should be stored inside Data Lake
§ Results of any transformations should be persisted
§ Data Linage, easy troubleshooting
§ Data Replay
§ Separate concerns (Scalability, Fault Tolerance)

29. TECHNOLOGY STACK
§ Apache NiFi for ingestion
§ Apache Spark/Flink/Presto for processing
§ Hive for Data Lake Storage
§ Hive/Memsql for DWH
§ Vertica/Redshift/Memsql for OLAP
DECOUPLING DATA PLATFORM

30. CHAPTER 4
WEAK
DATA ORGANIZATION

31. INGEST
DWH
ПОЧЕМУ ТАК
ДОЛГО
РАНЯТСЯ
РЕПОРТЫ ???
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
AGGREGATE IT

32. INSIDE THE PROCESS
DATA PLATFORM
ML TRAININGINGEST
DATA PLATFORM
REPORTING
PROCESS
REPORTING ENGINE
ML RUNNING
AirFlow
REPORTING CACHE
REPORTING METADATA
SCHEDULED REPORTS
ANALYTICS
API
FACADE
3rd PROVIDERS
SYSTEM
EVENTS
OLAP
LOAD
DWH
DATA LAKE DWH

33. COMMON PITFALLS
§ Direct access to Data Lake or cold DWH
§ Reports query RAW data
DATA PLATFORM
ML TRAININGINGEST
DATA PLATFORM
REPORTING
PROCESS
REPORTING ENGINE
ML RUNNING
AirFlow
REPORTING CACHE
REPORTING METADATA
SCHEDULED REPORTS
ANALYTICS
API
FACADE
3rd
PROVIDERS
SYSTEM
EVENTS
OLAP
LOAD
DWH
DATA LAKE DWH

34. INGESTION
VALIDATION
ENRICHMENT
BATCH 1
HDFS/HIVE/...
RAW FACT TABLE
BATCH 2
AGGREGATOR
HDFS/HIVE/…
AGGREGATION TABLE
BATCH 1 BATCH 2
INTRODUCE AGGREGATIONS

35. INGESTION
VALIDATION
ENRICHMENT
BATCH 1
HDFS/HIVE/...
RAW FACT TABLE
BATCH 2
AGGREGATOR
HDFS/HIVE/…
AGGREGATION TABLE
BATCH 1 BATCH 2
BATCH 3
INTRODUCE AGGREGATIONS

36. INGESTION
VALIDATION
ENRICHMENT
BATCH 1
HDFS/HIVE/...
RAW FACT TABLE
BATCH 2
AGGREGATOR
HDFS/HIVE/…
AGGREGATION TABLE
BATCH 1 BATCH 2
BATCH 3
BATCH 3
INTRODUCE AGGREGATIONS

37. INTRODUCE AGGREGATIONS
date hour user_cookie creative_id clicked
05/21/19 03 4444444 123 1
05/21/19 03 5555555 321 1
05/21/19 03 6666666 321 1
05/21/19 04 7777777 567 1
impressions
creative_id campaign_id
123 1
321 1
567 2
campaigns
date hour campaign_id creative_id clicked
05/21/19 03 1 123 1
05/21/19 03 1 321 2
05/21/19 04 2 567 1
performance_ad

38. CHAPTER 5
AVAILABILITY

39. AVAILIABILITY
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS

40. AVAILIABILITY
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
ГДЕ МОИ
РЕПОРТЫ?

41. THINGS EASY TO MISS
AVAILABILITY
§ Be aware of Data Availability vs System Availability
§ Use multiple storages
§ Chose wise between Kappa and Lambda architectures
§ Design solution to be scale-enabled
§ Introduce effective monitoring
§ Know your data latency and design solution based on it

42. THINGS EASY TO MISS
FAULT TOLERANCE
§ Every job should be fail-ready and retry-able by design
§ Enable multiple attempts on scheduler side
§ Use idempotent sinks
§ Implement backpressure: Use Pull over Push, address Blob/S3/HDFS or Kafka
§ Develop Spark* applications cluster-agnostic, easy to switch clusters
§ Separate applications between multiple clusters (EMR)

43. THINGS EASY TO MISS
EFFECTIVE MONITORING
§ Collect system and app-specific metrics
§ Use local monitoring agents
§ Make local daemons as part of terraform/chef scripts
§ Measure data availability [ in-rate, out-rate, lag]
Bandar-Log https://github.com/VerizonAdPlatforms/bandar-log/
§ Think about Datadog [local agents, dashboards, monitors, notes]

44. DASHBOARD EXAMPLE [INGESTION]

45. DASHBOARD EXAMPLE [AGGREGATIONS]

46. CHAPTER 6 – SCHEMA EVOLUTION

47. SCHEMA EVOLUTION
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS

48. UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
ПОЧЕМУ ТАК
ДОЛГО
ДЕЛАЮТСЯ
ИЗМЕНЕНИЯ?
SCHEMA EVOLUTION

49. SCHEMA EVOLUTION PROBLEMS
§ Meet new feeds & data providers
§ Change apps to support updated data schemas
§ Update DB schemas and data
§ Lack of single source of metadata

50. DATA PLATFORM
INGEST
REPORTING
PROCESS LOAD
3rd PROVIDERS
SYSTEM
EVENTS
RAW
AirFlow
DWH ANALYTICS
API
FACADE
DATA PLATFORM
INTRODUCING SCHEMA MANAGER
PROCESSED AGGREGATED AGGREGATIONS

51. DATA PLATFORM
INGEST
REPORTING
PROCESS LOAD
3rd PROVIDERS
SYSTEM
EVENTS
RAW
AirFlow
DWH ANALYTICS
API
FACADE
DATA PLATFORM
INTRODUCING SCHEMA MANAGER
PROCESSED AGGREGATED AGGREGATIONS
SCHEMA
MANAGER

52. DESIGN DECISIONS
§ SEPARATE CODE AND SCHEMA
Keep code schema-agnostic
Let Schema Manager be Single Source of Truth
Think about introducing dynamic validation and enrichment rules
§ SCHEMA FORMAT
Avro as a Serde engine
Avro as a Contract
§ TECHNOLOGY STACK
Confluent Schema Registry
Apache Atlas

53. CHAPTER 7
INTRODUCE
DATA ENGINEERING

54. DATA ENGINEERING DATA ANALYSIS
INGESTION & ETL
§ Flexible data pipelines
INTEGRATION
§ Multiple data vendors
WAREHOUSING
§ Efficient data organization
REPORTING
§ Organizing data into
informational summaries
DATA ANALYTICS
§ Find meaningful
correlations between data
DATA MININIG
§ Extract new knowledge
DATA SCIENCE
§ Insights
§ Modes & Predictions
§ Machine Learning
VISUALISATION
§ Get insights
DATA INFRASTRUCTURE
RELIABLE SERVICES
§ Private vs public clouds
§ Quick scale out/down
§ Instant deployments
§ Efficient maintenance
§ Infrastructure through code
MONITORING
§ Big Picture and total control
on every service
§ Metrics and Alerts
BIG DATA WORLD

55. SHARING RESPONCIBILITY TO DATA
Separate expertise
Involve Data Engineers to make Data Platform
better and faster

56. THANK YOU