Em Campbell-Pretty gave a presentation on scaling agile practices to a large enterprise data warehousing program. The data warehouse contained over 80 TB of business data across 28 servers processing 400 million records daily. Campbell-Pretty described establishing agile teams, organizing them into a release train, and using SAFe principles and practices to coordinate their work. This resulted in delivery cycle times decreasing from 12 months to 3 months with weekly deployments, cost reductions of 70%, and improvements in quality, on-time delivery, and team satisfaction.

1. Em Campbell-Pretty
Partner, Context Matters
@PrettyAgile
www.prettyagile.com
au.linkedin.com/in/ejcampbellpretty/
em@contextmatters.com.au
Scaling Agile Data
Warehousing with SAFe
Agile Denver Meetup
8th October 2015

2. THE ENTERPRISE DATA WAREHOUSE
80 TB of business data
28 Teradata nodes
3 TB of new data per month
350 SQL users
47 business systems feeding into the
EDW
400 million records loaded daily
13 billion records extracted daily
43 billion records in largest table
4 tables with over 35 billion records
124 different client apps
6.8 million queries daily
16K tables, 28K views
and 2K macros
7K jobs running daily
21K processes

3. THE EDW DELIVERY TEAM

5. • Amazon search
AGILE DATA WAREHOUSING?

6. WIDE PIPES

7. Image Source: http://ivyleagueinsecurities.com/2010/07/missing-pieces/

8. Image Source: http://phandroid.com/2011/05/24/unlock-your-samsung-galaxy-s-2-for-use-on-att-or-t-mobile/

9. Photo (https://flic.kr/p/FryLL) by Harold Walker CC BY-NC 2.0

10. THE FIRST “AGILE” TEAM

11. FEBRUARY
MAY
AUGUST
OCTOBER
NOVEMBER
SCALING “WAGILE”

12. Photo Source: https://flic.kr/p/ndZRQj

13. Project
Manager
Product Owner
Business
Analyst Change Lead Technical Lead
Logical Data
Modeller
Physical Data
Modeller
Physical Data
Modeller
Vender Project
Manger
On Site ETL On Site ETL
On Site ETL Off shore ETL
Off shore ETL Off shore ETL
DBA
Business
Intelligence Test Lead
Test Analyst
Enterprise
Architect Data SME
Scrum Master
A 22 PERSON AGILE TEAM???

14. Team
Size
Throughput

15. Scrum Master
Technical Lead Test Lead Developer Developer Developer Developer Developer
SMALL CROSS FUNCTION TEAMS

17. The Agile
Release Train
Agile Technical
Practices

18. Photo scoured from: https://flic.kr/p/8ejM1e

20. GM
EDW Program
Solution
Design & Build
BAU Program

21. SHAPING THE EDW AGILE RELEASE TRAIN

22. Transition
Services
SHAPING THE EDW AGILE RELEASE TRAIN
Deployment
Services
Development Services
Pipeline
Services

24. SET THE DATE

28. WHAT ABOUT PROJECTS?

29. VISUALIZE EVERYTHING

32. EPIC OWNER
FEATURE
OWNER
PRODUCT
OWNER
CONTENT AUTHORITY

33. SCALING DOWN FROM 18 PROJECT MANAGERS

34. FUNDING THE AGILE RELEASE TRAIN (ART)
COST OF ART
----------------------
ART VELOCITY
COST PER STORY POINT =

35. PREPARING THE PSI BACKLOG

36. 8 week PSI

37. 8 week PSI
6 week PSI

38. 8 week PSI
6 week PSI
4 week PSI

39. 8 week PSI
6 week PSI
4 week PSI
2 week PSI????

40. UNITY DAY (PSI-LITE)

42. BAKE IN TIME TO IMPROVE

43. KEEPING THE TRAIN ON THE TRACKS

44. COMMUNICATE, COMMUNICATE, COMMUNICATE
9:00am 9:15am 930am
9:45am

45. BUBBLE UP ISSUES OUTSIDE TEAM CONTROL

46. USE SPECIALIST CHAPTERS

47. APPLY LEAN LEADERSHIP

48. REINFORCE TEAM & ART IDENTITY
World Leaders
in Agile Data
Warehousing

49. IMPLEMENT EXTREME PROGRAMMING

50. INVEST IN ENABLING INFRASTRUCTURE

51. A LITTLE TASTE OF OUR CULTURE

52. A LITTLE TASTE OF OUR CULTURE

53. ü Average delivery cycle time down from 12 month to 3 months
ü Frequency of deployment increased from quarterly to weekly
ü Cost to deliver down 70%
ü 95% decrease in product defects
ü 99% projects delivered on time and on budget
ü Happy project sponsors (NPS +50)
ü Happy teams (NPS +56)
THE RESULTS
“This is probably the most well-scaled agile data
warehousing group I’ve ever observed, and they are
still getting better”
Ken Collier
Director of Agile Analytics at Thoughtworks
& author of Agile Analytics

54. Em Campbell-Pretty
Partner, Context Matters
@PrettyAgile
www.prettyagile.com
au.linkedin.com/in/ejcampbellpretty/
em@contextmatters.com.au
QUESTIONS?
WANT TO KNOW MORE?
Check out my blog: www.prettyagile.com
You can also download a free copy of my article on
“Scaling Agile Data Warehousing”
from the Cutter IT Journal using this link:
http://www.cutter.com/offers/campbellpretty.html