Challenges of running a data infrastructure team at a larger company.

1. Data Infrastructure
as Code
Building core data services in a small team
Michal Gancarski, Zalando SE

2. Introduction
Team Nucleo - Data Lake Core
❖ eight engineers, lead, producer and the product specialist

3. Introduction
The Infrastructure
❖ thousands of datasets, hundreds of users, tens of clusters

4. Introduction
Main Challenges
❖ project overload
❖ GDPR
❖ compliance
❖ job market

5. Introduction
Value Multipliers and Cost Savers
RoW - Return on Whatever
V - value
C - cost

6. (+V) Reach Out
Be Friends with Stakeholders
❖ compliance
❖ security & governance
❖ IT architecture
❖ business intelligence
❖ product analytics

7. (+V) Reach Out
Offer Multiple Support Channels
❖ team email
❖ #datalake-users
❖ #databricks-users
❖ office hours
❖ guest developers

8. (-C) Outsource & Empower
Watch Users Solving It For You
❖ users are your community

9. (-C) Outsource & Empower
Let Others Do Your Work
❖ innersourced Big Query pipeline & Presto proxy

10. (-C) Outsource & Empower
Lean on Your Vendors. Hard.

11. (+V) Automate and Generalize
Climb The Ladder of Automation

12. (+V) Automate and Generalize
Treat Use Cases as Future Services
❖ pilot with one team
❖ parametrize and turn into a service

13. (-C) Protect and Simplify
Build The Wall

14. (-C) Protect and Simplify
Maintain Identity and Vision
“I strongly believe that the future
prosperity of the American people
depends on how well
each data infrastructure team
understands what NOT to build.
#bigdata #yolo”
Abraham Lincoln in a letter to Congress, 1845

15. Thank you!
Michal Gancarski, Zalando SE

16. Introduction
The Infrastructure
❖ thousands of datasets, hundreds of users, tens of clusters

21. (-C) Protect and Simplify
Provide Dedicated Support
x
❖ focus for the rest
of the team
❖ driver of future
automation

22. (-C) Protect and Simplify
Retire Less Valuable Services
“Rather than construction,
programming is more like gardening.”
The Pragmatic Programmer
https://pragprog.com/book/tpp/the-pragmatic-programmer
“Migrations matter because they are usually
the only available avenue to make
meaningful progress on technical debt.”
Will Larson
https://lethain.com/migrations/

23. (-C) Outsource & Empower
Provide Self-Service

24. (+V) Reach Out
Share Knowledge

25. (-C) Outsource & Empower

26. Introduction

27. (+V) Reach Out

28. (+V) Automate & Generalize

29. (+V) Automate and Generalize
Provide 24/7 Monitoring

30. (+V) Automate and Generalize
Turn Components into Commits

31. (-C) Protect & Simplify

32. (+V) Reach Out
(-C) Outsource & Empower
(+V) Automate & Generalize
(-C) Protect & Simplify