This document discusses the evolution of a data platform architecture at a company. It begins by introducing the initial architecture with a shared storage model and outlines pros and cons. It then discusses moving to a weak scheduling model using tools like Airflow. Next, it covers decoupling the data platform into ingestion, processing, and analytics layers. Key aspects covered include introducing schema management and data governance practices. Overall, the document provides a high-level overview of the journey of a monolithic data platform toward a more scalable and modular architecture.

1. COWBOY DATING
WITH BIG DATA
DATA PLATFORM EVOLUTION IN ACTION
BORIS TROFIMOV

2. INTRO – PARTNER 1

3. INTRO – PARTNER 2

4. НАГРАДЫ PARTNER 2
• Медаль за Kotlin
• Полный кавалер Spring & Spring Boot
• Орден за взятие Kubernetes

6. EXPECTATIONS
PRODUCT

7. CHAPTER 1 – SHARED STORAGE

8. UI
MVP
API FACADE
DB
SHARED STORAGE

9. UI
MVP
API FACADE
DB
ГДЕ
репортинг?
SHARED STORAGE

10. UI
MVP
API FACADE
DB
3rd PROVIDERS
SHARED STORAGE

11. PROS
• Fast TTM
• Relatively cheap from infra and cost perspective
CONS
• Tight data and code cohesion
• Different Scaling scenarios
• Performance and Availability issues
SHARED STORAGE

12. SHARED STORAGE
UI
MVP
API FACADE
OLTP
DATA
PLATFORM
OLAP
3rd PROVIDERS

13. DATA PLATFORM
3rd PROVIDERS
SYSTEM
EVENTS
API FACADEDATA PLATFORM

14. CHAPTER 2 – WEAK SCHEDULING

15. SCHEDULING
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS

16. SCHEDULING
DATA PLATFORM
DATA PLATFORM
3rd
PROVIDERS
SYSTEM
EVENTS
DATA PROCESSING SCRIPT
OLAPDATA PROCESSING SCRIPT
DATA PROCESSING SCRIPT
API FACADE

17. DATA PROCESSING SCRIPT
SCHEDULING
DATA PLATFORM
3rd
PROVIDERS
SYSTEM
EVENTS
CROND
QUARTZ
…
OLAP
DATA PROCESSING SCRIPT
API FACADE
DATA PLATFORM
DATA PROCESSING SCRIPT

18. SCHEDULING
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
КАКОЙ
КРОНТАБ?

19. SCHEDULING
Use prod-ready schedulers
§ Airflow
§ Azkaban
§ Oozie
§ Jenkins?
What we gain
§ Identity control and Audit
§ Job Lineage, Logging and Troubleshooting
§ Tools to design Workflows/DAGs
§ Fault Tolerance features (rerun etc.)

20. CHAPTER 3 –MONOLITHIC DATA PLATFORM

21. DATA PLATFORM
3rd PROVIDERS
SYSTEM
EVENTS
Scheduler
OLAP
API FACADE
DATA PLATFORM
DATA PROCESSING SCRIPT
DATA PLATFORM

22. DATA PROCESSING SCRIPT
3rd PROVIDERS
SYSTEM
EVENTS
Scheduler
OLAP
DATA PLATFORM
ПОЧЕМУ
ОПЯТЬ
МОНОЛИТ?
API FACADE
DATA PLATFORM

23. DATA PLATFORM
DWH ANALYTICSREPORTING
DECOUPLING DATA PLATFORM

24. LOADINGEST
DATA PLATFORM
PROCESS
AirFlow
DATA PLATFORM
3rd PROVIDERS
SYSTEM
EVENTS
DWH
DATA LAKE DWH
DECOUPLING DATA PLATFORM

25. LOADINGEST
DATA PLATFORM
PROCESS
AirFlow
3rd PROVIDERS
SYSTEM
EVENTS
DWH
DATA LAKE DWH
LOADINGEST PROCESS
DECOUPLING DATA PLATFORM

26. ML TRAINING
DATA PLATFORM
ANALYTICS
OLAP
ML RUNNING
AirFlow
3rd PROVIDERS
SYSTEM
EVENTS
DWH
DATA LAKE DWH
LOADINGEST PROCESS
LOADINGEST PROCESS
DECOUPLING DATA PLATFORM

27. DATA PLATFORM
ML TRAINING
DATA PLATFORM
REPORTING
REPORTING ENGINE
ML RUNNING
AirFlow
REPORTING CACHE
REPORTING METADATA
SCHEDULED REPORTS
ANALYTICS
API
FACADE
3rd PROVIDERS
SYSTEM
EVENTS
OLAP
DWH
DATA LAKE DWH
LOADINGEST PROCESS
LOADINGEST PROCESS
DECOUPLING DATA PLATFORM

28. DECOUPLING DATA PLATFORM
ARCHITECTURE DECISIONS
§ Introduce granular reusable and testable steps inside pipelines [ingest, validate,
enrich, aggregate etc.]
§ Separate pipeline per vendor/feed
§ Raw data should be stored inside Data Lake
§ Results of any transformations should be persisted
§ Data Linage, easy troubleshooting
§ Data Replay
§ Separate concerns (Scalability, Fault Tolerance)

29. TECHNOLOGY STACK
§ Apache NiFi for ingestion
§ Apache Spark/Flink/Presto for processing
§ Hive for Data Lake Storage
§ Hive/Memsql for DWH
§ Vertica/Redshift/Memsql for OLAP
DECOUPLING DATA PLATFORM

30. CHAPTER 4
WEAK
DATA ORGANIZATION

31. INGEST
DWH
ПОЧЕМУ ТАК
ДОЛГО
РАНЯТСЯ
РЕПОРТЫ ???
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
AGGREGATE IT

32. INSIDE THE PROCESS
DATA PLATFORM
ML TRAININGINGEST
DATA PLATFORM
REPORTING
PROCESS
REPORTING ENGINE
ML RUNNING
AirFlow
REPORTING CACHE
REPORTING METADATA
SCHEDULED REPORTS
ANALYTICS
API
FACADE
3rd PROVIDERS
SYSTEM
EVENTS
OLAP
LOAD
DWH
DATA LAKE DWH

33. COMMON PITFALLS
§ Direct access to Data Lake or cold DWH
§ Reports query RAW data
DATA PLATFORM
ML TRAININGINGEST
DATA PLATFORM
REPORTING
PROCESS
REPORTING ENGINE
ML RUNNING
AirFlow
REPORTING CACHE
REPORTING METADATA
SCHEDULED REPORTS
ANALYTICS
API
FACADE
3rd
PROVIDERS
SYSTEM
EVENTS
OLAP
LOAD
DWH
DATA LAKE DWH

34. INGESTION
VALIDATION
ENRICHMENT
BATCH 1
HDFS/HIVE/...
RAW FACT TABLE
BATCH 2
AGGREGATOR
HDFS/HIVE/…
AGGREGATION TABLE
BATCH 1 BATCH 2
INTRODUCE AGGREGATIONS

35. INGESTION
VALIDATION
ENRICHMENT
BATCH 1
HDFS/HIVE/...
RAW FACT TABLE
BATCH 2
AGGREGATOR
HDFS/HIVE/…
AGGREGATION TABLE
BATCH 1 BATCH 2
BATCH 3
INTRODUCE AGGREGATIONS

36. INGESTION
VALIDATION
ENRICHMENT
BATCH 1
HDFS/HIVE/...
RAW FACT TABLE
BATCH 2
AGGREGATOR
HDFS/HIVE/…
AGGREGATION TABLE
BATCH 1 BATCH 2
BATCH 3
BATCH 3
INTRODUCE AGGREGATIONS

37. INTRODUCE AGGREGATIONS
date hour user_cookie creative_id clicked
05/21/19 03 4444444 123 1
05/21/19 03 5555555 321 1
05/21/19 03 6666666 321 1
05/21/19 04 7777777 567 1
impressions
creative_id campaign_id
123 1
321 1
567 2
campaigns
date hour campaign_id creative_id clicked
05/21/19 03 1 123 1
05/21/19 03 1 321 2
05/21/19 04 2 567 1
performance_ad
JOIN

38. BREAK DOWN DOMAIN
DOMAIN DOMAIN
• Break down domain into business-
concerned areas
• Cover area with dedicated aggregation
• Example For Video Platform
• Ad performance
• Player performance
• Video performance
• Revenue performance
• Build once, reuse between multiple reports

39. CHAPTER 5
AVAILABILITY

40. AVAILIABILITY
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS

41. AVAILIABILITY
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
ГДЕ МОИ
РЕПОРТЫ?

42. THINGS EASY TO MISS
AVAILABILITY
§ Distinguish Data Availability vs System Availability
§ If possible do not share infrastructure between DP with Core services
§ Address using multiple storages
§ Chose wise between Kappa and Lambda architectures
§ Design solution to be scale-enabled
§ Introduce effective monitoring
§ Know your data latency and design solution based on it

43. THINGS EASY TO MISS
FAULT TOLERANCE
§ Every job should be fail-ready and retry-able by design
§ Enable multiple attempts on scheduler side
§ Use idempotent sinks
§ Implement backpressure: Use Pull over Push, leverage Blob/S3/HDFS or Kafka
§ Develop Spark* applications cluster-agnostic, easy to switch clusters
§ Separate applications between multiple clusters (EMR)

44. THINGS EASY TO MISS
EFFECTIVE MONITORING
§ Collect system and app-specific metrics
§ Use local monitoring agents
§ Make local daemons as part of terraform/chef scripts
§ Measure data availability [ in-rate, out-rate, lag]
Bandar-Log https://github.com/VerizonAdPlatforms/bandar-log/
§ Think about Datadog [local agents, dashboards, monitors, notes]

45. DASHBOARD EXAMPLE [INGESTION]

46. DASHBOARD EXAMPLE [AGGREGATIONS]

47. CHAPTER 6 – SCHEMA EVOLUTION

48. SCHEMA EVOLUTION
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS

49. UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
ПОЧЕМУ ТАК
ДОЛГО
ДЕЛАЮТСЯ
ИЗМЕНЕНИЯ?
SCHEMA EVOLUTION

50. SCHEMA EVOLUTION HELL
§ Meet new feeds & data providers
§ Change apps to support updated data schemas
§ Update DB schemas and data
§ Lack of single source of metadata

51. DATA PLATFORM
INGEST
REPORTING
PROCESS LOAD
3rd PROVIDERS
SYSTEM
EVENTS
RAW
AirFlow
DWH ANALYTICS
API
FACADE
DATA PLATFORM
INTRODUCING SCHEMA MANAGER
PROCESSED AGGREGATED AGGREGATIONS

52. DATA PLATFORM
INGEST
REPORTING
PROCESS LOAD
3rd PROVIDERS
SYSTEM
EVENTS
RAW
AirFlow
DWH ANALYTICS
API
FACADE
DATA PLATFORM
INTRODUCING SCHEMA MANAGER
PROCESSED AGGREGATED AGGREGATIONS
SCHEMA
MANAGER

53. DESIGN DECISIONS
§ SEPARATE CODE AND SCHEMA
Keep code schema-agnostic
Let Schema Manager be Single Source of Truth
Think about introducing dynamic validation and enrichment rules
§ SCHEMA FORMAT
Avro as a Serde engine
Avro as a Contract
§ TECHNOLOGY STACK
Confluent Schema Registry
Hortonworks Schema Registry
Apache Atlas

54. CHAPTER 6 – DATA GOVERNANCE

55. DATA GOVERNANCE
UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS

56. UI
MVP
API FACADE
OLTP
Data Platform
OLAP
3rd PROVIDERS
Я НАЙДУ
ТЕБЯ НА ТОМ
СВЕТЕ!!!
DATA GOVERNANCE

57. DATA GOVERNANCE CHECKLIST
Did I think about Personal Data Protection?
Did I think about Data Access Control?
Did I think about Data Eviction?
Did I think about Data Lineage?
Did I think about Data Quality?
Did I think about Data Inventory?

58. PERSONAL DATA PROTECTION
§Learn what Personally Identifiable Data (PID) is
§Think twice before storing any PID
§Anonymize data as soon as possible in ETL and prefer to use anonymized data
over PID where never possible
§Introduce Anonymized Unique ID (AUID) and store relationship PID <-> AUID
separately

59. DATA ACCESS CONTROL
§ Introduce IAM for components and developers inside Data Lake and DWH
Control access to PID and anonymized data
§ Introduce ACL for end users inside OLAP
Leverage OLAP features to support ACL -- per row, table, schema, database

60. DATA EVICTION
§ Design data and applications with evection enabled
§ Introduce data retention policy and schedule cleanup jobs
§ Separate data retention policy per raw and aggregation tables
§ Document retention policy

61. DATA LINEAGE
§ Shit happens
§ Shit will happen, think about it in advance
RECOMMENDATIONS
§ Each ETL step should persist its output with reasonable retention policy
§ Persist any application logs (Spark/Yarn, CMD apps, ETL, …)
§ Log any significant application decisions
§ Persist any provenance logs (NiFi, …)

62. DATA QUALITY
§ Introduce data validation [even if it is undefined] and track validation issues
§Schema errors (wrong type, missed mandatory field)
§Semantic errors (unknown or poorly formatted IDs)
§Business errors (certain business constraints per-event or cross-event)
§ Track any errors and expose metrics
§ Track discrepancies and expose metrics
§raw and aggregation data
§Discrepancy between real-time and batch
§Discrepancy between vendor data

63. DATA INVENTORY
§ Document how data organized
§ Document where data stored
§ Document what and where data exported
§ Document what and where data ingested
§ Document as granular as possible -- per vendor, data source, ETL component
etc.

64. CHAPTER 7
INTRODUCE
DATA ENGINEERING

65. DATA ENGINEERING DATA ANALYSIS
INGESTION & ETL
§ Flexible data pipelines
INTEGRATION
§ Multiple data vendors
WAREHOUSING
§ Efficient data organization
REPORTING
§ Organizing data into
informational summaries
DATA ANALYTICS
§ Find meaningful
correlations between data
DATA MININIG
§ Extract new knowledge
DATA SCIENCE
§ Insights
§ Modes & Predictions
§ Machine Learning
VISUALISATION
§ Get insights
DATA INFRASTRUCTURE
RELIABLE SERVICES
§ Private vs public clouds
§ Quick scale out/down
§ Instant deployments
§ Efficient maintenance
§ Infrastructure through code
MONITORING
§ Big Picture and total control
on every service
§ Metrics and Alerts
BIG DATA WORLD

66. SHARING RESPONCIBILITY TO DATA
Separate expertise
Involve Data Engineers to make Data Platform
better and faster

67. СВАДЬБА

68. THANK YOU