This document discusses immutable infrastructure and how it can help address issues with mutable infrastructure configurations. It defines immutable infrastructure as pre-configuring systems before deployment so running systems are delivered as complete units. This enables delegating management to platforms like Kubernetes. It presents three patterns for immutable provisioning: 1) using a baseline image and configuration, 2) live booting and configuration, and 3) deploying pre-built images. Image creation ideally involves configuring a live system and capturing the results. Immutable infrastructure can provide benefits like speed, safety and scalability compared to traditional mutable approaches.

1. Immutable Infrastructure
Rethinking Configuration in the Age of Easy Redeployment
RackN, Inc
February, 2018
Note: Graphics mainly from http://pexels.com
<<< Shift Left <<<

2. 2
@zehicle #immutable
Involved in Open Ops Software:
Digital Rebar Project
Kubernetes ClusterOps SIG
OpenStack Board
Your Humble Presenter
I’m all about automating infrastructure.
Rob Hirschfeld (aka @zehicle)
Co-Founder of RackN
rob@rackn.com

3. 3
@zehicle #immutable
Storytime! “Self-Bootstrapping Kubernetes”
Kubecon in Nov 2017 we created this demo
Simple “immutable” Idea:
1) In Memory Boot Machines
2) Install Docker
3) Elect Leader
4) Run Kubeadm on Leader
5) Run Kubeadm on Remainder
But….it’s shockingly hard to maintain.
Dependencies breaks the installation
And they are constantly changing.

4. 4
@zehicle #immutable
Storytime! “Self-Bootstrapping Kubernetes”
So, while it’s pretty cool,
it’s not “real” immutability
Presentation & Demo
https://youtu.be/OowxF6GqK4I
sa !

5. Why is configuration fragile?

6. Why is configuration fragile?
mu on
V

7. 7
@zehicle #immutable
But… I Infrastructure as Code?!
Sorry. Mutability adds complexity
Traditional “build-in place” approaches
● Have hidden dependency graphs
● Create variation between environments
● Are harder to “lock down” due to config
AND OMG… updates and patches are even harder
● Idempotent operations are difficult
● Roll backward is next to impossible!
● Creating indeterminate state

8. 8
@zehicle #immutable
Traditional “build-in place” approaches
● Have hidden dependency graphs
● Create variation between environments
● Are harder to “lock down” due to config
AND OMG… updates and patches are even harder
● Idempotent operations are difficult
● Roll backward is impossible
● Creating indeterminate state
But… I Infrastructure as code?!
Sorry. Mutability adds complexity
Let’s o k w !

9. What is Immutable Infrastructure?

10. What is Immutable Infrastructure?Pre-de y fi d
V

11. 11
@zehicle #immutable
Traditional Deploy and Configure
System is configured in situ from
a least common denominator
baseline.
This can be “immutable-like”
under the right conditions.
We’ll come back to that...
Delivery Pipeline
Deployment
Code
Build
Integrate
Run
Configure

12. 12
@zehicle #immutable
Shifting Configuration BEFORE Deployment
In our ideal delivery pipeline,
configuration is before
deployment.
Running systems are delivered as
a complete runnable unit for
deployment.
Delivery Pipeline
Deployment
Code
Build
Integrate
Run
Configure

13. 13
@zehicle #immutable
Shifting Configuration BEFORE Deployment
In reality, it’s very hard to create a
distinct artifact for every running
instance; instead, we create
incremental versions.
So we do some initialization of the
reusable versioned instance.
Cloud init is the most commonly
known pattern for this.
Delivery Pipeline
Deployment
Code
Build
Integrate
Run
Configure
Ini iz !
V

14. Cloud Native
Infrastructure
CNIbook.info
Justin Garrison & Kris Nova
“Infrastructure as software”

15. 15
@zehicle #immutable
Which Enables… Delegating Operations
If you can make your artifacts
immutable then you can delegate
management of them to a
platform like Kubernetes.
Kubernetes does not configure
infrastructure. It maintains state
based on a manifest.
StateManager
(e.g.Kubernetes)
Code
Build
Integrate
Run
Configure
Delivery Pipeline

16. 16
@zehicle #immutable
Which Enables… Delegating Operations
If you can make your artifacts
immutable then you can delegate
management of them to a
platform like Kubernetes.
Kubernetes does not configure
infrastructure. It maintains state
based on a manifest.
StateManager
(e.g.Kubernetes)
Code
Build
Integrate
Run
Configure
Delivery Pipeline
Kub te ? W !
Is ut > K8s?

17. Immutable is a DevOps Pattern
<<< Shift Left & Create/Delete

18. 18
@zehicle #immutable
The Problem
Immutability <<< Shifting Left
package
server
image
provision
server
initial
config

19. 19
@zehicle #immutable
The Problem
Immutability <<< Shifting Left
patch 1
package
server
image
provision
server
initial
config

20. 20
@zehicle #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
package
server
image
provision
server
initial
config

21. 21
@zehicle #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
t e d s o n't
s o t c 2!
package
server
image
provision
server
initial
config

22. 22
@zehicle #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
t e d s o n't
s o t c 2!
What Madness?
● We have to maintain root access
● Patches assume system state
● Patches create dependency graphs
● Coordination? Should we halt work?
● Drift is inevitable!
package
server
image
provision
server
initial
config

23. 23
@zehicle #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
SA !!package
server
image
provision
server
initial
config

24. 24
@zehicle #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
package
server
image
provision
server
initial
config
destroy!!

25. 25
@zehicle #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
destroy!!
destroy!!patch 1
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config

26. 26
@zehicle #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
destroy!!
destroy!!patch 1
depatch 2
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config

27. 27
@zehicle #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
destroy!!
destroy!!patch 1
patch N
depatch 2
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config

28. Cloud like behavior …

29. 29
Immutable Provisioning systems
treat infrastructure as a black box
Cloud-like Integration and Staged Workflow
Provisioning
System
Requested
State
Returned
State
REST
API
Event
Hook

30. 30
Cloud-like Integration and Staged Workflow
Immutable Provisioning systems
treat infrastructure as a black box
Provision requests are for a system state
with optional parameters.
The intermediate changes to achieve the
state are not exposed to the requester.
Provisioning
System
Reset Join
Install
Config
Test
Requested
State
Returned
State
REST
API
Event
Hook

31. 31
Cloud-like Integration and Staged Workflow
Immutable Provisioning systems
treat infrastructure as a black box
Provision requests are for a system state
with optional parameters.
The intermediate changes to achieve the
state are not exposed to the requester.
REMEMBER: Operators of the
provisioning system require high
transparency, stages and control.
Provisioning
System
Reset Join
Install
Config
Test
Requested
State
Returned
State
REST
API
Event
Hook
No h n
op i n !

32. Immutable Patterns
1) Baseline + Configuration
2) Live Boot + Configuration
3) Image Deploy

33. 33
Provision
1: Baseline + Configuration
Benefit: Easiest to achieve with current tools, Safer than Patching
Challenge: Lots of Post-Configuration, Not Really “Immutable”, Slow
Instead of relying on patches, rely on starting from a pristine image
ResetBaseline Configure Run
Additional Reference https://thenewstack.io/immutable-hardware-ops-hygiene-security-efficiency/

34. 34
Benefit: Fast reset times, forces good behavior
Challenge: Provisioning becomes critical path, still have dependency graph
Like #1 but clean-up is simply a reboot. Favors smaller footprint O/S.
2: Live Boot + Configuration
Provision RebootBaseline Configure Run

35. 35
3: Image Deploy
Benefit: Shorter time to ready, highly controlled (“shift left”), rollback
Challenge: Harder to create and deploy images
Image is deployed from source instead of Baseline + Configure
Provision
Deploy
Image
Run Provision
Deploy
Image
Run

36. 36
3: Image Deploy
Benefit: Shorter time to ready, highly controlled (“shift left”), rollback
Challenge: Harder to create and deploy images
Image is deployed from source instead of Baseline + Configure
Provision
Deploy
Image
Run Provision
Deploy
Image
Run
Ini iz !
V
Ini iz !
V

37. 37
So… Let’s talk Image Creation
Ideally in an automation build process.
You DO THE CONFIGURATION on a
live system (so you still need
configuration tools) and then capture
the image into a portable format.
Tools like Hashicorp Packer, Image
Builder, WBIC or raw images are used
to create source files (e.g. AMI, OVS).

38. 38
So… Let’s talk Image Creation
Ideally in an automation build process.
You DO THE CONFIGURATION on a
live system (so you still need
configuration tools) and then capture
the image into a portable format.
Tools like Hasicorp Packer, Image
Builder, WBIC or raw images are used
to create source files (e.g. AMI, OVS).
Tha n li ot
of k & re y ow!

39. Yes, But…
It’s faster, safer & more scalable.

40. 40
Build Pipeline
Immutable Demo
Prep: Image is pre-created from reference system.
Reference
System
Image
Read

41. 41
Build Pipeline Deploy
Immutable Demo
Prep: Image is pre-created from reference system.
Stage: Boot RAM image and write image to disk(s)
Reference
System
Image
Target System
RAM BOOT
WriteRead

42. 42
Build Pipeline Deploy
Immutable Demo
Prep: Image is pre-created from reference system.
Stage: Boot RAM image and write image to disk(s)
Deploy: Reboot and run
Reference
System
Image
Target System
RAM BOOT
Target System
RUNNING
Reboot
Write RunRead

43. Thank you!
Questions?
Interested in IMMUTABLE METAL?
It’s complicated, but we can get you there.
Start at http://portal.rackn.io
• Quickstart takes about 30 minutes
• Use your own hardware, VirtualBox or Packet.net
account
– use “RACKN100” on Packet.net for credit