So, you want to update the software for your user, be it the nodes in your K8s cluster, a browser on user’s desktop, an app in user’s smartphone or even a user’s car. What can possibly go wrong? In this talk, we’ll analyze real-world software update fails and how multiple DevOps patterns, that fit a variety of scenarios, could have saved the developers. Manually making sure that everything works before sending an update and expecting the user to do acceptance tests before they update is most definitely not on the list of such patterns. Join us for some awesome and scary continuous update horror stories and some obvious (and some not so obvious) proven ideas for improvement and best practices you can start following tomorrow.

1. DevOps Patterns & Antipatterns
for Continuous Software Updates
“What can possibly go wrong?!”

2. Why software updates?

3. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

6. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

7. “As everycompanybecome asoftware company,
Security vulnerabilities are the new oil spills”
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

9. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
Identify Fix Deploy

10. Identify
Immediately
Fix
OS upgrade
Deploy
years

11. Identify
2 months
Fix
Struts upgrade
Deploy
2 months

12. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

13. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

14. Identify
As fast as possible
Fix
As fast as possible
Deploy
As fast as possible
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

18. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

19. This is not
a new idea!
XP: short feedback
Scrum: reducing cycle time
to absolute minimum
TPS: Decide as late as
possible and Deliver as fast
as possible
Kanban: Incremental
change
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

21. @jbaruch #dockercon jfrog.com/shownotes@ErinMeyerINSEAD’s “Culture Map”
🎩

22. shownotes
http://jfrog.com/shownotes
Slides
Video
Links
Comments, Ratings
Raffle
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

24. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

25. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

26. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

29. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

30. Do we want
it?
Let’s
update!
Are there
any high
risks?
How about no
Do we trust
the
update?
Update
available
No
Yes
Yes
Yes
NoNo

31. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

32. number of artifacts as a symptom of complexity
@jbaruch @jfrog #LiquidSoftware www.liquidsoftware.com
IoT
Serverless
Docker
Infrastructure as Code
Continuous Delivery
Continuous Integration
Agile
Microservices
2000
Today

33. The problem is not the code, it’s the data. Big data.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

34. #emptyenvelopefromchina
@jbaruch #LiquidSoftware @devopsdotcom
http://jfrog.com/shownotes

35. Do we want
it?
Let’s
update!
Are there
any high
risks?
Can we
verify the
update?
How about no
Do we trust
the
update?
Update
available
No
Yes
Yes
Yes
Yes
NoNoNo
Time
consuming
verification

37. Features
that we
want
Acceptance
tests costs
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

39. Do we want
it?
Let’s update!
Are there
any high
risks?
Update
available
No
Yes
No one
asked you
(auto update)
Your browser
Twitter in your browser
Twitter on your smartphone
Your smartphone OS?!

40. What can possibly
go wrong?

41. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

43. Continuous
updates pattern:
Local rollback
Problem: update went
catastrophically wrong and
an over the-air patch can’t
reach the device
Solution: Have a previous
version saved on the device
prior to update. Rollback in
case problem occurred
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

44. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

48. Continuous
updates pattern:
OTA software
updates
Problem: physical recalls
are costly. Extremely costly.
Also, you can’t force an
upgrade.
Solution: Implement over
the air software updates,
preferably, continuous
updates.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

49. continuous OTA updates are like normal OTA updates,
but better
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

52. Continuous
updates pattern:
continuous
updates
Problem: In batch updates
important features wait for
non-important features.
Solution: Implement
continuous updates.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

53. You thought your problems are hard?
Things under your control Server-side Updates IoT (Mobile, Automotive,
Edge) Updates
The availability of the target
✓ ✕
The state of the target
✓ ✕
The version on the target
✓ ✕
The access to the target
✓ ✕
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

55. KNIGHT-MARE
New system reused old APIs
1 out of 8 servers was not
updated
New clients sent requests to
machine contained old code
Engineers undeployed
working code from updated
servers, increasing the load on
the not-updated server
No monitoring, no alerting, no
debugging
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

56. Continuous
updates pattern:
Automated
deployment
Problem: People suck at
repetitive tasks.
Solution: Automate
everything.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

57. Continuous
updates pattern:
frequent updates
Problem: Seldom
deployments generate
anxiety and stress, leading
to errors.
Solution: Update
frequently to develop skill
and habit.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

58. Continuous
updates pattern:
state awareness
Problem: Target state can
affect the update process
and the behavior of the
system after the update.
Solution: Know and
consider target state when
updating.
Reverting might require
revering the state.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

60. Cloud-dark
New rules are deployed
frequently to battle attacks
Deployment of a single
misconfigured rule
Included regex to spike
CPU to 100%
“Affected region: Earth”
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

61. Continuous
updates pattern:
Progressive
Delivery
Problem: Releasing a bug
affects ALL the users.
Solution: Release to a
small number of users first
effectively reducing the
blast radius and observe.
If a problem occurs, stop
the release, revert or
update the affected users.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

62. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

63. Continuous
updates pattern:
observability
Problem: Some problems
are hard to trace relying on
user feedback only
Solution: Implement
tracing, monitoring and
logging
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

64. Continuous
updates pattern:
Rollbacks
Problem: Fixes might take
time, users suffer in a
meanwhile
Solution: Implement
rollback, the ability to
deploy a previous version
without delay
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

65. Continuous
updates pattern:
feature flags
Problem: Rollbacks are not
always supported by the
deployment target platform
Solution: Embed 2 versions
of the features in the app
itself and trigger them with
API calls
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

67. Continuous
updates pattern:
zero downtime
updates
Problem: You will probably
loose all your users if you
shut down for 5 weeks to
perform an update.
Solution: Perform zero-
downtime OTA small and
fequent continuous
updates.
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

68. Continuous
updates
Frequent
Automatic
Tested
Progressively delivered
State-aware
Observability
*Local Rollbacks
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

69. Do we want
it?
Let’s update!
Are there
any high
risks?
Do we trust
the
update?
Update
available
No
Yes
Yes
Yes
Sure, why not?
(auto update)

70. Our goal is to transition from bulk and rare software updates to
extremely tiny and extremely frequent software updates;
so tiny and so frequent that they provide an illusion of
software flowing from development to the update target.
We call it the Liquid Software vision.
”
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

71. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

72. Corner cases?
@jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

75. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes

76. Q&A and twitter ads
@jbaruch
#LiquidSoftware
#DevNexus
https://liquidsoftware.com
https://jfrog.com/shownotes