A story of MediaMath's journey from a data warehouse to a data platform and the technical and organizational challenges faced.

1. ©2016 MediaMath Inc. 1
02.16.2016
Rory Sawyer – Software Engineer, Data Platform
Moving Past Infrastructure
Limitations

2. ©2016 MediaMath Inc. 2

3. ©2016 MediaMath Inc. 3
Massive Volume of Data
 180 billion impression opportunities a day
 3+ million peak qps
 3+ TB of data per day (compressed)
 Logs represent financial transactions
Every record counts!

4. ©2016 MediaMath Inc. 4
MediaMath’s Data Platform
 Centralized location for data at MM
Collect data from across the company
Standardize access for internal and external clients
 End-result of data warehouse transformation

5. ©2016 MediaMath Inc. 5
Once Upon A Time
The Old Days
Etc..

6. ©2016 MediaMath Inc. 6
Architecture – 2013

7. ©2016 MediaMath Inc. 7
Data Warehousing at MM circa 2013
 No proper QA/testing environment
 Production workflows and ad-hoc analytics ran side-by-side
 Scaling becomes an issue
Developing/testing/deploying changes to workflows frustrating
Copying data to more monolithic systems
More shell, more problems

8. ©2016 MediaMath Inc. 8
Data Access circa 2013 – Users and Consumers
 Tools: SQL, shell
 Consumers: Data analysts, data engineers

9. ©2016 MediaMath Inc. 9
Data Access circa 2013
 Logs: Custom FTP transfers
Merely extracting data could cause production problems
FTP could run out of space
 Heavy reliance on canned reports
Served via reporting API
Updated at most three times a day, usually just once a day
 Hard to keep pace with growing demands
Internal Clients
External Clients

10. ©2016 MediaMath Inc. 10
Data Liberation

11. ©2016 MediaMath Inc. 11
Moving Past Infrastructure
 Resource flexibility
 Fully own our conceptual problems
Can’t just get a bigger box or a higher support license
 Lower barrier to entry
Decouple storage and computation

12. ©2016 MediaMath Inc. 12
Move to the Cloud
 Simple Storage Service (S3):
Primary data store; source of truth
Append-only. Update = delete + append
 Elastic Map Reduce (EMR):
Transient hadoop clusters
Spot instances – save money
 Redshift:
Columnar storage for efficient querying

13. ©2016 MediaMath Inc. 13
Data Platform – Today

14. ©2016 MediaMath Inc. 14
Data Access – Today

15. ©2016 MediaMath Inc. 15
Developer Experience
 Get to say “yes” more
Rapid development/testing/deployment removes inertia
 Clearly distinct, perfectly synced QA environment
Run multiple versions of workflows simultaneously on same source data
 More control over components
 Localized impact of processing
Each team uses their own compute environment

16. ©2016 MediaMath Inc. 16
We don’t worry about this like we used to

17. ©2016 MediaMath Inc. 17
Improved User Experience
 Augmented standard reporting with easily-accessible data
warehouse
AWS + Qubole provides value to all skill levels
 Transparently handle different data sources
Bridge storage types and AWS accounts
 Choose your preferred query method
Spark, MapReduce, Flink, or BI tool
 All barriers removed

18. ©2016 MediaMath Inc. 18
Productize it, cap’n
 Log level data API
Direct log access on S3
 Interactive Query
Scalable, user-friendly data processing with Qubole

19. ©2016 MediaMath Inc. 19
Hive

20. ©2016 MediaMath Inc. 20
SmartQuery

21. ©2016 MediaMath Inc. 21
Clusters

22. ©2016 MediaMath Inc. 22
Qubole’s Greatest Hits

23. ©2016 MediaMath Inc. 23
Hybrid Life

24. ©2016 MediaMath Inc. 24
New and Old

25. ©2016 MediaMath Inc. 25
Managing a Hybrid Warehouse
 Upfront effort to keep old and new consistent
After that, could migrate in pieces
 Keeping datasets in sync
Store metadata about datasets and processes
Keep record of what data was processed by which batches

26. ©2016 MediaMath Inc. 26
Ch-ch-ch-challenges
 Spot instances: bid too low, jobs never start
Build processes around selecting best/cheapest zones
 Maintaining two systems at once
Consistency, monitoring, updates…
 Migrating mindset
New set of questions to answer

27. ©2016 MediaMath Inc. 27
What we’ve learned

28. ©2016 MediaMath Inc. 28
Life after Liberation
 Decentralize all the things
Single-machine -> distributed computing
Single data team -> data engineers on all the teams
 Engineers on every team
Data Science – Spark (Scala)
Analytics – Spark/Hive (with Redshift connector)
Product – Hive
Engineering – Spark/Hive/MapReduce
Business analysts – SmartQuery

29. ©2016 MediaMath Inc. 29
Data Access circa 2013 – Users and Consumers
 Tools: SQL, shell
 Consumers: Data analysts, data engineers

30. ©2016 MediaMath Inc. 30
Data Access Today – Users and Consumers
 Tools: Hadoop (Scalding, Hive), Spark, RDBMS
 Consumers: Engineers, product managers, business
analysts, etc.

31. ©2016 MediaMath Inc. 31
The Cost of Decentralization
 Different producers and consumers have different priorities
File format, end-to-end latency, correctness, etc…
 Adding a platform layer could add friction

32. ©2016 MediaMath Inc. 32
Not Abandoning Managed Infrastructure
or: There and Back Again
 Managed hardware is still important
On-premises Hadoop cluster
Clients ETL into managed hardware
 Experience with Data Liberation broke down “walled garden” feel
of AWS

33. ©2016 MediaMath Inc. 33
Some sort of “last slide” title
 Moving DW to cloud has proven itself
Quick development allows us to keep pace
Ease of use helps teams and clients fine tune their own reporting
 Re-thinking the tools and skills needed for data warehousing
 Avoid tech debt by evolving our software and ideas before
committing to hardware
 Move away from trickle-down data

34. ©2016 MediaMath Inc. 34
THANK YOU!
Rory Sawyer
Software Engineer
Data Platform
Rsawyer@mediamath.com