The business case and ROI for MicroServices, DevOps / Agile, adopting CI/CD, and Kubernetes with best practices. (Final draft)

1. ROI For DevOps Culture and Practices
ACCELERATED DELIVERY
A Business Case for DevOps Culture and Practices:
Lean Management, Agile, DevOps, CI/CD, and
Microservices
BUSINESS CASE: AGILE
DEVOPS, CI/CD,
MICROSERVICES

2. – Jeffrey Immelt, CEO of GE
“Every industry and company that is not bringing
software to the core of their business will be
disrupted.”
2

3. What We Are After
❖ High performing IT organizations:
❖ 200x more frequent deployments
❖ 24x faster recovery from failure
❖ 3x lower change failure rate
❖ 22% less time on unplanned work
❖ 50% less time remediating security issues
3
DevOps The Big Picture, written by Richard Seroter (Pluralsigt)

4. Why We Care
❖ Elite DevOps performers:
❖ 2604x faster time to recover from incidents
❖ 46x more frequent code deployments
❖ 1.53x more likely to meet of exceed business goals
❖ 1.8x more likely to recommend their team to others
4
CI/CD: The Big Picture, written by Barry Luijbregts (Pluralsigt)

5. [
5
Author of best-selling agile development book
Early adopter of TDD, DevOps, Agile, etc.
Successfully ran development organizations
Developed open source software used by millions
• Java Champion 2018
Early adopter and advocate of microservices
• Worked on Vert.x, QBit, Reakt, Groovy, Boon,
etc.
• Speaker on microservices at JavaOne
• Designed/implemented microservices-based
systems that scale to 100M users
RICK HIGHTOWER

6. [
6
‣ Past experience
‣ Latest trends
‣ Shoulders of giants
SLIDE DECK BASED ON MANY BOOKS, AND MORE

7. [
7
‣ How we got here
‣ History of
• MicroServices
• DevOps / Agile
• CI/CD
• Kubernetes
‣ Business proposition
‣ CD/CD - DevOps practices
‣ Continuous delivery
• Continuous integration
• Lean management and monitoring /
KPIs
• SCM / Version Control / GitOps /
Immutable infrastructure
• Trunk-based development
‣Live Demos: Concrete best practices
OUTLINE

8. 8
DevOps is not a task;
it’s a culture.

9. DevOps not just the name of a team
❖ DevOps is more than having a team called DevOps
❖ DevOps is a mindset and development culture
❖ Culture of DevOps, Agility, Lean, MVP, etc. is a clear win
❖ Case studies
❖ Continuous learning/Continuous improvement and growth
❖ There are guides, books, practices, and information
❖ It is a journey

10. Acceleration in Practice
❖ Make customers happy
❖ Deliver more
❖ Less burnout
❖ Grow the value of the
company
❖ Make more money
10

11. Acceleration in Practice
❖ Empower developers
❖ Docker, MiniKube, Kind, K8s
❖ Create environments on the fly
❖ Eliminate manual processes
❖ Reduce people-intensive tasks
❖ Add quality to the schedule
❖ Test-driven development
❖ Avoid long lead times
❖ Avoid long-lived branches
❖ Hard to merge
❖ Hard to deploy
11

12. How we got here
History of Microservices,
Agile, and CI/CD
Brief history of time

13. How we got here
❖ Web pages that were brochures
❖ eCommerce
❖ Legacy integration
❖ Rush to ‘webify’ businesses
❖ SOA: wrap legacy systems as services to use from the web
❖ Virtualization, Virtualization 2.0, Cloud, Containers, and
now Container orchestration

14. Microservices: INCEPTION and Natural Evolution
❖ Now you can run a Java Virtual Machine in a Docker image
❖ Which is just a process pretending to be an OS
❖ Which is running in an OS that is running in the cloud
❖ Which is running inside of a virtual machine
❖ Which is running in Linux server that you don’t own that you share with
people whom you don’t know
❖ Servers are not giant refrigerator boxes that you order from Sun and wait three
months for (circa 2000)..… Goal was to run a lot of things on same server
❖ Did you develop code in the 90s with punch cards?
❖ Microservices recognize trend
14

15. Microservices
❖ Focus is building small, reusable, scalable services
❖ Adopt the Unix single-purpose utility approach to service development
❖ Small and malleable so they can be released more often
❖ Easier to write
❖ Easier to change
❖ Go hand in hand with continuous integration and continuous delivery
❖ Heavily REST-based and message oriented
❖ Focus on business capability
❖ Refocus on object oriented programming roots
❖ Organize code around business domains.
❖ Data and business rules colocated in the same process or set of processes.
What is microservice architecture?

16. Microservices: Key ingredients
❖ Independently deployable, small, domain-driven
services
❖ Own their data (no shared databases)
❖ Communication through a well-defined wire protocol
usually JSON over HTTP (curl-able interfaces)
❖ Well defined interfaces and minimal functionality
❖ Avoiding cascading failures and synchronous calls -
reactive design for failure

17. [
17
CDN INVALIDATOR MICROSERVICE – INTELLIJ

18. [
18
“It’s not the daily
increase but daily
decrease. Hack
away at the
unessential.”
– Bruce Lee
SOA EVOLVED

19. Microservices: Evolution of SOA
❖ SOA and Microservices have common goals and purposes
❖ Refinement to meet goals of polyglot devices and 3rd
generation virtualization
❖ Cloud, container, container orchestration
❖ Parts of SOA that worked well
❖ Microservices Web technologies based (or streams or messages)
❖ To provide scalability, modular, domain-driven, small, and
continuously deployable cloud-based services

20. [
20
“Microservices Architecture … perhaps started out as SOA and
applying lessons learned as well as pressure to:
support polyglot devices,
deploy more rapidly
And adopt
architecture liquidity that cloud computing and virtualization/
containerization provide” —Microservices Architecture Paper
SOA VS. MICROSERVICES

21. [
21
‣ Philosophy behind microservices mirrors Unix
‣ Unix’s inventor, Ken Thompson, defined its philosophy:
• One tool, one job.
‣ Emphasizes building short, simple, clear, modular, and extendable code
• Easily maintained and repurposed by other developers
MICROSERVICES: UNIX PHILOSOPHY
What is microservice arc

22. MicroServices: Achieving Resilience
❖ Avoid synchronous calls to avoid cascading failures
❖ Instead embrace:
❖ Streams, queues,
❖ Actor systems
❖ Event loops
❖ Other async calls.
❖ Spend more time with distributed logging/log aggregation w/MDC
❖ Now distributed tracing: Jaeger, Service Mesh
22

23. MicroServices: Monitoring and KPIs
❖ Customer/User experience KPIs
❖ Debugging (requests per second, # threads, #
connections, failed auth, expired tokens, etc.)
❖ Circuit breaker (monitor health, restarts, act/react based
on KPIs)
❖ Cloud orchestration (monitor load, spin up instances)
❖ Health checks and observable KPIs
23

24. MicroServices: Continuous Deployment
❖ Microservices are continuously deployable services
❖ Focus of microservices is on breaking applications into small (micro),
reusable services that might be useful to other services or other
applications.
❖ ‘micro’ part of microservices comes to denote small
❖ Services can be deployed independently.
❖ Can be tweaked and then redeployed independently.
❖ Microservice vs monolith when deploying
What is microservice a

25. Evolution of Agile Development
DevOps, Agile,
TDD, CI/CD
Brief history of time

26. Devops: Lean, Scrum, XP, Agile, TDD, CI/CD, 12 factor
❖ TDD, CI and CI/CD
❖ Test Driven Development and Agile
❖ XP, Agile, Scrum
❖ CI/CD (Jenkins and the tools that came before)
❖ CI/CD needs automated testing
❖ DevOps aka DevSecOps
❖ Heroku and the birth of 12 factor deployment, DevOps, KPIs, SRE
❖ YBYOI vs. SRE vs. DevOps and where do you fit?
❖ SRE: Observability, Log aggregation, KPIs/Metrics, Distributed/Trace logging
❖ What is GitOps?
❖ What is immutable infrastructure?
❖ What is cloud native?

27. [
27
KUBERNETES

28. Kubernetes (K8s)
❖ Services, stateful sets, namespaces, tags, ingress, egress
❖ Helm/Kustomize for packaging an app or set of related MicroServices and deploy to K8s
❖ Multi-cloud support and just cloud support
❖ Monitoring built-in (or at least easily pluggable)
❖ Easy to ramp up (or easier)
❖ Supports flexible deployment models
❖ Integrates with Cloud providers services or runs standalone (on prem or cloud)
❖ Container Orchestration: Yarn, Mesos, Marathon, Nomad, Borg, ECS and now
Kubernetes
❖ What came before and now: Heroku/PaaS/IAAS/EC2, Docker, Docker Swarm, Mesos/
Marathon, Nomad, ECS, EKS, etc. –– K8s is the current mindshare champ.
28

29. No Margin. No Mission.
Business case:
Winning the race
Why we want to do it? Why it is
important!

30. Organizational performance
❖ High performers 2x the rate will exceed organizational performance goals as low performers:
❖ 2x profitability
❖ 2x productivity
❖ 2x market share
❖ 2x number of customers
❖ High performers twice as likely to exceed non-commercial performance goals as low performers
❖ 2x better quantity of products and services
❖ 2x operating efficiency
❖ 2x customer satisfaction
❖ 2x quality of products/services
❖ 2x achieving organizational/mission goals
❖ 50% increase in market capitalization compared to low performers!
30

31. Culture
❖ Bad delivery performance major contributor to burnout
❖ Deployment pain and poor software delivery practices cause organizational burnout
❖ Good technical practices predict continuous delivery
❖ DevOps Culture: Improve organizational culture, job satisfaction, delivery performance,
less burnout, less deployment pain, and less time spent on rework!
❖ High performers spend 50% less time remediating security issues than low performers
❖ Trunk-based Development (like Github flow)
❖ High performers have shortest integration times and branch lifetimes
❖ Branch life and integration typically lasting hours or a day
❖ Low performers have longest integration times and branch lifetimes
❖ Branch life and integration typically lasting days or weeks
31

32. [
32
DEVOPS ACCELERATING DELIVERY
Source: Forsgren PhD, Nicole, Jez Humble, Gene Kim, Accelerate. IT Revolution Press, Kindle Edition.

33. Software delivery performance
❖ Delivery performance improves organizational performance and quality/
customer satisfaction
❖ Delivery performance is driven by investment in DevOps
❖ To improve overall software delivery performance Improve these
❖ deploy frequency, lead time, mean time to restore (MTTR), and change fail
percentage
❖ Lead time improves with good version control and automated testing
❖ MTTR improves with good version control and monitoring
❖ Deploy frequency improves with continuous delivery
❖ Version control best practices needed for continuous delivery

34. Continuous delivery
❖ The ability to deliver
❖ Build quality in
❖ Work in small batches
❖ Automate repetitive tasks including
❖ testing & deployments
❖ Pursue continuous improvement
❖ Ownership
❖ Comprehensive configuration management
❖ Continuous integration
❖ Continuous testing
You can’t skip steps.
There is investment up
front.
Today’s speed up can
be tomorrows painted
yourself
In a corner.

35. What are the steps
How we get there
What must we ensure to hit these
amazing outcomes!

36. Accelerate DevOps
❖ Continuous delivery
❖ Architecture
❖ Product and process
❖ Lean Management and monitoring
❖ Cultural

37. Continuous delivery capabilities
❖ Implement continuous delivery / continuous deployment
❖ Version control all production artifacts
❖ Automate your deployment pipeline
❖ Implement continuous integration
❖ Use trunk-based development methods (like Github flow
instead of git flow)
❖ Implement test automation
❖ Shift left on security

38. [
38
Culture eats
strategy for
breakfast.
– Peter Drucker
CULTURE

39. Culture Capabilities
❖ Support a generative culture
❖ Encourage and support learning
❖ Encourage collaboration
❖ Make work as meaningful as possible
❖ Support and encourage transformational leadership

40. [
40
What gets
measured gets
improved.
– Peter Drucker
MEASUREMENT

41. Lean management and Monitoring Capabilities
❖ Lightweight change approval process
❖ Monitor application and system KPIs to inform business decisions
❖ Proactively check system health
❖ Preemptively detect and mitigate problems
❖ Improve process and work within WIP limits
❖ Need visible dashboards to monitor/communicate WIP, quality,
applications and systems perf
❖ Continuous improvement

42. Example Monitoring
❖ Hygieia
❖ Productivity dashboards
❖ Jenkins dashboards
❖ Runtime KPIs
❖ Customer KPIs
❖ Developer KPIs
❖ Grafana, Kibana, Prometheus, InfluxDB, StatsD, etc.

43. [
43
JENKINS BUILD SYSTEM

44. [
44
CODE COVERAGE DASHBOARD

45. [
45
PROMETHEUS – TRACKING KPIS

46. [
46
KUBERNETES DASHBOARD

47. [
47
HYGIEIA

48. Lean Management
❖ Process: Small Batches
❖ Decompose into features that allow for rapid development
❖ MVP - prototype with just enough features to proved business value or enable validated
learning
❖ Quickly gather customer requirements (A/B testing, customer satisfaction surveys, etc.)
❖ Team experimentation
❖ Lean Management
❖ Change approval
❖ Proactive notification
❖ Monitoring and KPIs
❖ WIP limits, visualizing work

49. Delivery Vs. Deployment
Continuous Delivery
Software can be deployed to production at any time
Continuous Deployment
Software is automatically deployed to production all the
time
49

50. Version control – SCM
❖ GitOps - keeping application code, system configuration,
application configuration, and scripts for automating
build and configuration in version control.
❖ Factors together predict IT performance
❖ Key component of continuous delivery
❖ Immutable infrastructure
❖ GitOps - keeping system and application config in git
(versioned) correlates high with delivery performance
50

51. Deployment Automation
❖ Deployment automation
❖ Containers, config, immutable infrastructure
❖ Comprehensive configuration management (automation
scripts), continuous integration and continuous testing
❖ Key metric of GitOps is how many diffs exists in system
config from git to deploy
51

52. Continuous Integration Needs
❖ Continuous integration
❖ Relies on SCM and Deployment automation
❖ Good version control, GitOps, Immutable Infrastructure
❖ Relies on automated tests
❖ Unit
❖ Integration
❖ Acceptance
❖ Functional
❖ Performance
❖ Synthetic
❖ Code coverage
❖ Static analysis
52

53. Trunk-based development
❖ Trunk-based development
❖ Like GitHub Flow but shorter lived branches
❖ Fewer active branches that never outlive a sprint
❖ Branch-off master per feature, bug fix, etc.
❖ More PRs more often
❖ No code freezes; integration periods less than a day
❖ Polar opposite of git flow
❖ Easy to see WIP
53

54. Shift left on security
❖ Integrating security into design and testing phases
❖ Security reviews of applications, including the infosec
team in design and demo process
❖ Using pre-approved security libraries and packages,
and testing security features as part of automated
testing suite
54

55. Concrete steps
Knowing the road
Best practices applied

56. Prefer Microservices…
❖ Monoliths can speed up MVP and prototypes but at a cost
❖ Hockey Stick
❖ J-Curve
❖ Monoliths make make CI/CD slower
❖ Monoliths make automated tests suites harder to build and they are needed for CI/CD
❖ Smaller monoliths and SOA is a move in the right direction, but monoliths should be
considered technical debt
❖ Refactoring to microservices is a journey
❖ Adoption is a Journey
❖ Fits the CI/CD well
❖ Fits small batch well (Lean Management)

57. Embrace Observability from the start
❖ Log aggregation
❖ Time series data base
❖ Log all KPIs for clusters
❖ Log all KPIs for applications and services
❖ Alerting
❖ Know when and how to employ distributed tracing
❖ Distributed/Trace logging
❖ Service Mesh

58. [
58
Cloud Native DevOps with Kubernetes
John Arundel and Justin Domingus
“The goal of an observability team
is not to collect logs, metrics or
traces. It is to build a culture of
engineering based on facts and
feedback, and then spread that
culture within the broader
organization.”
– Brian Knox (Digital Gear)
RECOMMENDED READING
View on Web

59. What is a PR? And how to ensure quality with it
❖ A PR is a pull request
❖ PR gives other developers a chance to review code before it
committed to master
❖ PR via small batch (why small? JIRA story or even a task or two)
❖ With GitHub and WebHooks you can block PRs from merging
❖ Code coverage met, build works, unit tests run, other checks
via Jenkins
❖ Reviewed and approved by at least two people
59

60. [
60
‣ Unit tests
‣ Perf testing JMH
‣ Functional tests
• At the HTTP layer
• At the Spring Boot layer
‣ Acceptance tests
‣ Smoke integration tests
‣ Full integration tests
‣ Synthetic testing
‣ Code Coverage (sonarqube)
‣ Security/Vulnerability dependency
license check
• Aqua, Fortify
‣ Full integration perf testing Aqua, Fortify
‣ Full integration perf testing
TESTS TO CREATE: TDD

61. [
61
‣ CI/CD
‣ Deploy often (daily or more)
‣ Test often (after every checkin run all automated tests)
‣ Block PRs from merging until they pass tests
‣ Block PRs until they are reviewed
‣ Block PRs until they reach a certain code coverage
‣ You can't do CI/CD without automated testing
‣ Testing allows you to move quickly with confidence
CI/CD TO ENFORCE QUALITY

62. Embrace small batch work
❖ Goal of three PRs per week
❖ Goal of one to two tasks from Jira per PR
❖ Use Jira # in commits and PR comments
❖ Break stories and features up into tasks
❖ Check in interim stories
❖ Use feature flags if it is hard to break up a feature or
story

63. Using Docker, Helm, Kubernetes
❖ Uber dev tools
❖ Persistent cluster set up
❖ Helm install Kubernetes
❖ Easy to integrate locally
❖ Docker, docker-deploy, helm, etc.
❖ Local integration possible and repeatable
❖ Same Docker containers in integration, development and prod
63

64. Example Automated deployment
❖ Merging into master triggers a deploy to integration and sends a Team message
❖ Approval from Product Manager pushes code to Prod or Demo
❖ Checking into main branch from PR (Trunk based development, i.e., GitHub flow)
❖ Artifacts and scripts for deployment checked into git and should not be modified
(GitOps)
❖ Puts code staging area to be checked by Product Manager into containers and
deployed to cluster (some ephemeral some not)
❖ Once checked by Infosec and Product Manager:
❖ Canary Deploys to 3 to 5% of traffic and is monitored (end goal) (Service Mesh,
Itsio)
❖ Then more and more as it is monitored and is ok

65. [
65
DEVELOPMENT PROCESS
Developer Feature Work
CI/CD Job
Groom a Jira Feature
Create Feature
Branch from Master
Work on Feature
Commit and Push
WIP
Create Feature PR
Respond to Peer
Reviews
Respond to Failed
Tests
Sync from Master Merge to Master
Build and Test
Deploy

66. [
66
CI/CD PROCESS
CI/CD Deployment
Developer Feature Work
Tag Release
Deploy to Integration
Run Integration Tests
Jenkins Demo
Workflow
Jenkins Approval
Workflow
Deploy to Demo
Environment
Product Manager
Ensure Changes
Approve Changes w/
Jenkins Workflow
CI/CD Build
Run Build and Unit
Tests
Ensure Build and Unit
Tests
Ensure Code
Coverage
Ensure Semantic
Version
Ensure Release Notes
Deploy to Ephemeral
Integration Env.
Run Integration Tests
Ensure Security
Checks
Ensure Two Peer
Reviews
Ready to Merge
Merge to Master Create Feature PR

67. Demo
❖ Jenkins
❖ Minikube/Kind
❖ Spring Boot
❖ Konfigure
❖ Kubernetes

68. Conclusion

69. Main Takeaways
❖ You can’t afford not to transform
❖ Transformation requires a deep understanding of practices
❖ ROI is there
❖ Faster to market, less burnout, more scalable, higher market cap, faster recovery, etc.
❖ Having a team called DevOps is not doing DevOps per se
❖ Continuous learning/Continuous improvement and growth
❖ Everyone gets into the DevOps mindset around TDD and integration tests
❖ Deploy small iterations continuously- Start by striving for daily deployments
❖ Automated deployment pipelines
❖ TDD and behavior driven development a MUST
❖ Kubernetes, Docker, Feature flags, Trunk based development and Microservices support these efforts
❖ DevOps is not a task; it is a culture

70. Appendix
Matt Damon slides

71. [
71
1. Microservices by Martin Fowler and James Lewis
2. Microservices Architecture by Chris Richardson
3. Micro services – Java, the Unix Way by James Lewis
4. Microservices, or How I Learned To Stop Making Monoliths and Love Conway’s Law by Cliff Moon
5. Micro service architecure by Fred George
6. Microservices are not a free lunch by Benjamin Wootton
7. Antifragility and Microservices by Russ Miles
8. The Unix Philosophy
9. Conway’s Law
10. Amazon Architecture
11. Migrating to microservices by Adrian Cockroft
12. Microservices with Spring Boot
13. Microservices for the Grumpy Neckbeard
14. Microservices definition on Wikipedia
15. Microservices and DevOps by Adrian Cockcroft
16. Building and Deploying Microservices - Bart Blommaerts
17. Microservices on the JVM - Alexander Heusingfeld
18, Microservices Shaun Abrams
19. Microservices Rick Hightower
20. Microservices Monitoring Rick Hightower
21. Microservices vs. SOA Rick Hightower
22. Microservices and Docker Rick Hightower
REFERENCES

72. [
72
‣ Not a new thing
‣ Not necessarily cool or hip
‣ Obvious next step in evolution
‣ Web, cloud, mobile (polyglot devices), server virtualization
‣ OS containerization, container orchestration, multi-core servers
‣ Cheaper and cheaper RAM
‣ 64 bit computing
‣ 10GB Ethernet, 100GB Ethernet, etc.
MICROSERVICES ADDRESS INDUSTRY EVOLUTION

73. [
73
QUALITY - UNPLANNED WORK
Source, Forsgren PhD, Nicole. Jez Humble, Gene Kim, Accelerate . IT Revolution Press. Kindle Edition.

74. [
74
‣ Gather and implement customer feedback
‣ Make the flow of work through the system visible
‣ Work in small batches
‣ Foster and enable team experimentation
PRODUCT AND PROCESS CAPABILITIES

75. [
75
‣ Use loosely coupled architecture
• Release new services on demand without outages
‣ Empower the team to select tools; trust team to pick the best tools
ARCHITECTURAL CAPABILITIES TO ACCELERATE

76. [
76
‣ Loosely coupled, well-encapsulated architecture drives IT performance.
• 2017 dataset biggest contributor to continuous delivery was loosely coupled, well-
encapsulated architecture
ARCHITECTURE

77. [
77
‣ Experimental approach to product development highly correlates with continuous
delivery
‣ Lean product development capabilities predict improvements in organizational culture
like reduced burnout higher software delivery performance and overall organizational
performance
LEAN PRODUCT MANAGEMENT CAPABILITIES

78. [
78
1. Microservices by Martin Fowler and James
Lewis
2. Microservices Architecture by Chris
Richardson
3. Micro services – Java, the Unix Way by
James Lewis
4. Microservices, or How I Learned To Stop
Making Monoliths and Love Conway’s Law
by Cliff Moon
5. Micro service architecure by Fred George
6. Microservices are not a free lunch by
Benjamin Wootton
7. Antifragility and Microservices by Russ
Miles
8. The Unix Philosophy
9. Conway’s Law
10. Amazon Architecture
11. Migrating to microservices by Adrian
Cockroft
12. Microservices with Spring Boot
13. Microservices for the Grumpy Neckbeard
14. Microservices definition on Wikipedia
15. Microservices and DevOps by Adrian
Cockcroft
16. Building and Deploying Microservices -
Bart Blommaerts
17. Microservices on the JVM - Alexander
Heusingfeld
18. Microservices Shaun Abrams
ADDITIONAL READING