Session Name: Our DevOps Journey is Incomplete Without Data Every company is a “Software Company”, “Software is eating the world”, in these similar lines I recently heard every company regardless of size it is a “Data Company”. True, somewhere or other every organisation produces, consumes, analyses, report Data & makes a decision, promotes, buy, sell, acquire, expand, down-size, so on based on Data. DevOps momentum has seen a rapid growth rate of new tools in the space of CI/CD, ARA(Application Release Automation), frameworks for enabling application delivery at pace. When it comes to Continuous Delivery, modern architecture patterns, practices like Microservices, our delivery teams face challenges with data. I want to discuss some of the challenges that I have gone through, ideas and some concrete pointers to help further to understand the “Data” problem in the DevOps space. If we could go as fast as only our weakest link, then Data, Data Management, Data Architecture, and associated practices need our attention and love.

1. TRACK: CI/CD CONTINUOUS EVERYTHING
NOVEMBER 12, 2020
BMK LAKSHMINARAYANAN(@LBMKRISHNA)
Value Stream Architect @BNZ
Our DevOps Journey
is Incomplete
Without Data
Our Data, Architecture & Data Management Practices
Need Love & Attention

2. TRACK: CI/CD CONTINUOUS EVERYTHING
Data is the lifeblood of
the system of the
business
@LBMKRISHNA

3. TRACK: CI/CD CONTINUOUS EVERYTHING
Problem Context
Challenges
Solution Approach
Recommendations
Remaining Challenges
Context, Statistics, Impact
Ownership, Silo, Locking, Standards
Education, Conventions, Architecture, Refactor
Useful Resources
Call for Action
Flow
@LBMKRISHNA

4. TRACK: CI/CD CONTINUOUS EVERYTHING

5. TRACK: CI/CD CONTINUOUS EVERYTHING
Problem Context
Understanding Our Problem
Continuous Delivery, DevOps & Data
@LBMKRISHNA

6. TRACK: CI/CD CONTINUOUS EVERYTHING
• Manual database deployments continue to be a
Software delivery bottleneck
• Speeding up database releases are most difficult
• Neglecting the data in our DevOps, Continuous Delivery
processes has significant “Business Impact”
• Data issues continue to slow us down, trouble our
efforts to continuously integrate, test & deploy
Problem Context
@LBMKRISHNA

7. TRACK: CI/CD CONTINUOUS EVERYTHING
Statistics
46%
20%
23%
57%
Extremely or very difficult to
speed database releases
We are slow due to
traditional siloed database
development practices
High risk of deployment
failure & extra downtime due
to database changes
Application changes require
corresponding database
changes
Source :https://www.devopsdigest.com/the-state-of-database-deployments-in-application-delivery-2019 @LBMKRISHNA

8. TRACK: CI/CD CONTINUOUS EVERYTHING
@LBMKRISHNA

9. TRACK: CI/CD CONTINUOUS EVERYTHING
State of DevOps Report 2019
@LBMKRISHNAhttps://services.google.com/fh/files/misc/state-of-devops-2019.pdf

10. TRACK: CI/CD CONTINUOUS EVERYTHING
State of DevOps Report 2019
@LBMKRISHNAhttps://services.google.com/fh/files/misc/state-of-devops-2019.pdf

11. TRACK: CI/CD CONTINUOUS EVERYTHING
Challenges
Understanding Our Challenges
Continuous Delivery, DevOps & Data
@LBMKRISHNA

12. TRACK: CI/CD CONTINUOUS EVERYTHING
Enterprise Data Challenges
@LBMKRISHNA
🔻
🔻
🔻
🔻
How do we think about data?
Data lock-in
Inherited data management
practices
Decade old standards &
conventions
Centralize data team to
design, model, deploy &
operate

13. TRACK: CI/CD CONTINUOUS EVERYTHING
Our system complexities are because of our
thinking, thinking particularly about the “Data”. We
tend to compromise a lot as we could not get the
“Data” in the right way, at the right time, in the
proper format, to the right place
Who is Responsible?
@LBMKRISHNA

14. TRACK: CI/CD CONTINUOUS EVERYTHING
Systems of Record
@LBMKRISHNA
Systems of Intelligence
Systems of Engagement

15. TRACK: CI/CD CONTINUOUS EVERYTHING
Architecture Mismatch
@LBMKRISHNA
Microservices Era & Big Ball of “Data” mud

16. TRACK: CI/CD CONTINUOUS EVERYTHING
Solution Approach
Small Steps, But in Right Direction
Education, Conventions, Architecture, Shift Left
@LBMKRISHNA

17. TRACK: CI/CD CONTINUOUS EVERYTHING
Solution Approach
@LBMKRISHNA
➽
➽
➽
➽
Data Management Team
Decentralize
Educate, Upskill, Tools,
Processes
Conventions & Standards
Unlock the Data
Adopting Modern Practices

18. TRACK: CI/CD CONTINUOUS EVERYTHING
Solution Approach
@LBMKRISHNA
http://assets.red-gate.com/solutions/database-devops/state-of-database-devops-2020.pdf?v=2

19. TRACK: CI/CD CONTINUOUS EVERYTHING
• Make the data first-class citizen
• Source control - Database schema
• Simplify processes
• Review changes early
• CI/CD pipeline
Help Our Data Folks - Practices
@LBMKRISHNA

20. TRACK: CI/CD CONTINUOUS EVERYTHING
• “T” & “Comb” shaped
• Data Engineers
• Educator & Mentors
• Platform Builders
Help Our Data Folks - Futuristic
@LBMKRISHNA

21. TRACK: CI/CD CONTINUOUS EVERYTHING
• Data Governance, Lineage
• Compliance (PCI DSS, Data Privacy)
• Regulations (GDPR, CCPA)
• Data Security (PII)
• Standards & Conventions (Naming, model, dictionary, catalogue)
• Enable Self-Service (Database request, Schema change request)
Help Our Dev & Ops Folks - Educate
@LBMKRISHNA

22. TRACK: CI/CD CONTINUOUS EVERYTHING
Recommendations
Useful Resources
Upskill, Learn, Spread, Share
@LBMKRISHNA

23. TRACK: CI/CD CONTINUOUS EVERYTHING
Recommendations
@LBMKRISHNA
We need to level up our game
Data, Data Architecture, Data Management &
Associated Practices need our love & attention

24. TRACK: CI/CD CONTINUOUS EVERYTHING
• Understanding current state (Tools, processes, roles, flow)
• Making work visible (current workloads, workloads in pipeline)
• Value Stream Management (New feature, Tech debt, Incident)
• Upskilling (DevOps, CI/CD Practices, Automation)
• Investment (Education, Time, Tools, Self-service)
• Role play (Data Engineer/Developer for a day)
Recommended Approach
@LBMKRISHNA

25. TRACK: CI/CD CONTINUOUS EVERYTHING
Remaining Challenges
Collectively We Need To Come Together, Work Together, Collaborate
Call for Action
@LBMKRISHNA

26. TRACK: CI/CD CONTINUOUS EVERYTHING
• Automated Governance (Including Data)
• Policies via automation and in pipeline
• Compliance as code
• Everyone is custodian of the Data
Help – Let Us Come Together
@LBMKRISHNA

27. TRACK: CI/CD CONTINUOUS EVERYTHING
NOVEMBER 12, 2020
BMK LAKSHMINARAYANAN(@LBMKRISHNA)
Value Stream Architect @BNZ
Our DevOps Journey
is Incomplete
Without Data
Our Data, Architecture & Data Management Practices
Need Love & Attention

28. TRACK: CI/CD CONTINUOUS EVERYTHING
THANK YOU TO OUR SPONSORS