The document discusses bringing operations considerations into the development process earlier, referred to as "shifting left." It advocates designing applications with operations in mind from the beginning. This includes understanding operational objectives, constraints, and service level agreements. Application telemetry and monitoring are also important to incorporate from the start. The document provides examples of how to implement operational practices like deployments, health checks, and incident response processes in a shifted left model where development and operations work more closely together.

1. Classificatie: vertrouwelijk
Making the Shift Left –
Bringing Ops to Dev
before bringing applications to
production

2. Classificatie: vertrouwelijk
Making the Shift
Left
Bringing Ops to Dev
before bringing
applications to
production
DigitalXchange – 3rd June 2023
Lucas Jellema, CTO & Architect Conclusion
Application
Platform

3. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023
Architect on many integration
initiatives with dozens of
large and small organizations
in The Netherlands
Oracle ACE Director,
Java Rockstar, author of
two books on integration
frequent speaker on
conferences & active
blogger
Lucas Jellema
Cloud Solution Architect & CTO
lucas.jellema@conclusion.nl | technology.amis.nl | @lucasjellema | lucas-jellema
3

4. Classificatie: vertrouwelijk
• 25 years
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 4

5. Classificatie: vertrouwelijk
• Light Bulb moment
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 5

6. Classificatie: vertrouwelijk
• Production!
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 6

7. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 7
business
process
Application Application Application
Platform Platform Platform Platform
Infrastructure Infrastructure Infrastructure Infrastructure Infrastructure

8. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 8
business
process
Application Application Application
Platform Platform Platform Platform
Infrastructure Infrastructure Infrastructure Infrastructure Infrastructure

9. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 9

10. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 10
• you make it run after you have built it
• you keep it running

11. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 11
• you make it run after you have built it
• you keep it running

12. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 12
• you make it run after you have built it
• you keep it running

13. Classificatie: vertrouwelijk
DevOps Cycle
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 13
https://medium.com/t%C3%BCrk-telekom-bulut-teknolojileri/devops-lifecycle-continuous-integration-and-development-e7851a9c059d

14. Classificatie: vertrouwelijk
Shift Left – Operations & Development without hand-over
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 14

15. Classificatie: vertrouwelijk
Shift Left – Operations and Development without hand-over
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 15

16. Classificatie: vertrouwelijk
Shift Left – Operations and Development without hand-over
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 16
Operations
Application
Platform & Infra
Development

17. Classificatie: vertrouwelijk
What is Ops?
• Make Up, Make Perform (as needed)
• Handle Exceptions
• Ensure Safe & Secure
• Watch | Control | Reduce Costs
• Optimize for Sustainable/Green run
• Grease the wheels & Clean the floors
• clean, prune, small technical
maintenance, odd little jobs
• (prepare for) Disaster Recovery
• Report on Day to Day operations
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 17

18. Classificatie: vertrouwelijk
Objectives & Constraints for Operations
• Service Level Agreement (contract)
• Business Requirements
• Principles, Guidelines, Rules, Constraints
• architecture,
• security,
• regulatory
• Professional responsibility, ethical
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 18

19. Classificatie: vertrouwelijk
What to Measure and what to Aim for?
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 19

20. Classificatie: vertrouwelijk
What level of care?
• Application + Platform + Infra need to
allow for the required level of care
• observe status (& trend )
• compare with goals
• work within constraints
• decide act upon action
• have protocols to follow and
instruments to take action
• This requires early preparation
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 20

21. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 21

22. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 22
Basware
invoices
record,
approve/deny
daily approved
invoices batch
MS Business
Central - Finance
integration
€
€
€
€

23. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 23

24. Classificatie: vertrouwelijk
Common thread
Requirements Development
Build, Test,
Q&A
Release &
Deploy
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 25

25. Classificatie: vertrouwelijk
Common thread
Requirements Development
Build, Test,
Q&A
Release &
Deploy
Operations
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 26

26. Classificatie: vertrouwelijk
Shift Left
Requirements Development
Build, Test,
Q&A
Release &
Deploy
Operations
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 27
Ops by Design

27. Classificatie: vertrouwelijk
Required for successful operations
• Definition of what is success
• What are non-functional constraints and conditions?
• Security, availability, performance, costs, CO2
• KPIs and targets?
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 28
Whenever the monitoring
indicated that a service was
down – the entire portal was
stopped and restarted
One particularly unstable service
determined whether today was
the user’s birthday.
It triggered multiple restarts per day …

28. Classificatie: vertrouwelijk
Application “Fingerprint”
• Describe for each Application:
• business value and owner/stakeholders
• priority / criticality
• business process
• trigger (when/why)
• result (& verification)
• source and destination systems
• data structures & filtering | mapping
• non happy flows
• quality controls
• security
• operational targets
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 29
Foundation for Solution Design and
Test plan, basis for Operational
mechanisms & processes

29. Classificatie: vertrouwelijk
Dev => Ops
Dev
Intake /
Ops
Acceptance
Deploy /
Rollout /
Activate
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 30
Observe Interpret Act
Learn &
Improve
Routine Management
Certificate, credential, user,
endpoints; Backup, Archive, Purge,
Patching, Cost Allocation
Support Desk
Fire drills / Chao(p)s Testing
Report & Improve
Recovery, Fail Over,
Hot Fix, Rollback of
Rollout, Scale Out, …
Improve instrumentation,
scalability, recovery, non happy
flow handling, configurability
Ops

30. Classificatie: vertrouwelijk
Intake: Check on Operability of Application
• Fingerprint
• Definition of Success
• QA-ed
• Operability of Application
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 31
Dev
Intake /
Ops
Acceptance
Photo by Priscilla Du Preez on Unsplash

31. Classificatie: vertrouwelijk
Ops Acceptance Test – confirm that operations can be done
• Simulate realistic production runtime scenarios
• Areas of interest:
• detection | analysis | instruction | facilities | action | result
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 32

32. Classificatie: vertrouwelijk
Deployment | Roll Out | Activation
• Automated
• Configurable environment specific dependencies at Application, Platform & Infra
• Verifiable through post-rollout Smoke Tests
• Activate for subset of workload: Blue/Green, Canary
• Scenario to Rollback
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 33
Dev
Intake /
Ops
Acceptance
Deploy /
Rollout /
Activate

33. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 34
API

34. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 35
• Saturday Midnight
• The loss of service was experienced on Monday morning
Catalog Service
API
Products Service
Products Service
?

35. Classificatie: vertrouwelijk
Beyond Smoke Test: Health Checks & In-Production Testing
• Periodically check availability of owned and called endpoints
• In Production Testing: dummy business objects
“business side-effect free” but otherwise very real
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 36
API
ping
No side effect –
test all the way accessibility
API
create
API
validate
Process business objects that are known to
be dummies – not sent to external systems

36. Classificatie: vertrouwelijk
Instrument & Emit - Observe – Interpret - Act
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 37
application
application
Platform & Infra
telemetry
Collect
process
& interpret
Alert
Report
Visualize
Analyze
Act
AIOps

37. Classificatie: vertrouwelijk
Telemetry – M(E)LT
• The need to know
• what happens
• what is the status
• Dimensions
• real time and after-the-fact
• fine grained and coarse grained
• business & functional, security, cost, CO2 and technical
• Types of Telemetry: MELT
• Events, Profiles & Exceptions
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 38

38. Classificatie: vertrouwelijk
The Invisible Route
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 39

39. Classificatie: vertrouwelijk
Tracing provides CCTV-like insight:
when, where, what, why, who, how long
• Track individual session | flow
• Powerful aggregation across traces
• group by component, status, origin,
trace context attributes
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 41

40. Classificatie: vertrouwelijk
• Log event (typically when something happens):
logs can be associated
with a job | request | transaction:
• Logs provide drill down details for trace-span
• Consolidated logs – from across application landscape – are powerful analysis tool
• timestamp is crucial to sort and correlate
platform
& infra
logs
Logs – fine grained reports
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 42

41. Classificatie: vertrouwelijk
Metrics – periodic mini status reports
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 43
platform
& infra
metrics
• Timestamped measurements
• Sometimes pushed, often
polled/scraped

42. Classificatie: vertrouwelijk
OpenTelemetry
• Cross industry standard for collection of Telemetry
• Virtually all tools for monitoring, analyzing, visualizing
telemetry support OpenTelemetry as source
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 44

43. Classificatie: vertrouwelijk
Events – telemetry context for analyzing and predicting
• Black Friday
• Horizontal Scale Out (double capacity for Service X)
• Rollout/activation of new version of Service Y
• Start of batch job Z
• Snowstorm in the Köln area
• Upgrade of Library P
• New certificate for Component Q
• Purged 2 TB of data from Time Series Database R
• Outage in Azure Data Center (Frankfurt region)
• Dortmund .. Bayern … Borussia .. nein, München meister
• New engineer joined the DevOps team
• Recovery test on database C
• Ticket logged regarding poor performance
• Expiry of Certificate K
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 45

44. Classificatie: vertrouwelijk
Profiling
• Dynamic analysis at runtime
• Periodic system snapshots (sampling)
• Metrics
• CPU & Memory usage
per process & child process | thread
• Application Call tree
• Duration & frequency of function calls
• Detailed bottleneck analysis
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 46

45. Classificatie: vertrouwelijk
Collect, Correlate, Interpret, Alert
• Create the overall picture
• Detect
• Find what’s mssng – unexpected unevents
• Automate
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 47

46. Classificatie: vertrouwelijk
Ops Obs is a veritable BI challenge
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 48
platform
infra
Integration
WebApp
Function
Job
API
Service
Traces, Log entries,
Metrics, Context
Events
Instrumentation
& Configuration
determines
telemetry
Telemetry Lake

47. Classificatie: vertrouwelijk
The Ideal Dashboard …
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 49

48. Classificatie: vertrouwelijk
The Ideal Dashboard …
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 50
Dashboard Past & Future
Now

49. Classificatie: vertrouwelijk
The Ideal Dashboard …
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 51
Dashboard - Now Past & Future
Now
Alerts Notifications Actions
Heartbeat – Proof of Health

50. Classificatie: vertrouwelijk
The Ideal Dashboard …
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 52
Dashboard – Past & Future Past & Future
Now
Trend & Pattern Charts Predictions Simulations / What-If explorations

51. Classificatie: vertrouwelijk
Act upon Alert
• Need to know what to do when
and how to do it then
• how important is it? what is priority?
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 53
?

52. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 54

53. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 55

54. Classificatie: vertrouwelijk
1202
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 56

55. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 57

56. Classificatie: vertrouwelijk
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 58

57. Classificatie: vertrouwelijk
Means & Mandate to act – Application & Platform/Infra
• Inspect
• Reconfigure
• Scale
• Restart
• Resubmit
• Apply Patch
• (Re)schedule job
• Perform recovery
• Provision & Update Infra
• Acquire (temporary) elevated privileges
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 59
Observe Interpret Act
Learn &
Improve
Routine Management
Fire drills / Chao(p)s Testing
Report & Improve

58. Classificatie: vertrouwelijk
Operations Firedrills
• Testing Resilience of
• systems & automated procedures
• processes
• people
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023
60

59. Classificatie: vertrouwelijk
Improvements over time
• Observe and extrapolate
• Explore What-If scenarios
• Automate
• Calibrate
• Refine schedules
• Fine-tune Tools and Procedures
• Extend configurability
• Automate (AI), dynamic, predictive configuration
• Refine instrumentation
• Improve telemetry processing
• Decommission
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 61

60. Classificatie: vertrouwelijk
Summary
• Operations for
successful business functionality
• Define success
• Design & Build
Observability & Operability
Test Operations
• Shift Left – “Ops by Design”
• Not DevOops but DevOps
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 62
Application
Platform

61. Classificatie: vertrouwelijk
Thank you
for your attention
I hope
this was
useful
Making the Shift Left - Bringing Ops to Dev - June 3rd 2023
lucas.jellema@conclusion.nl | technology.amis.nl | @lucasjellema | lucas-jellema
63