The influence of “distributed
platforms” on devops
Kris Buytaert
@krisbuytaert

@KrisBuytaert@KrisBuytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● CTO and Open Source Consultant @CTO and Open Source Consultant @inuits.euinuits.eu
● Everything is a DNS ProblemEverything is a DNS Problem
● Evangelizing devopsEvangelizing devops
● Organiser of #devopsdays, #cfgmgmtcamp, #loadays,Organiser of #devopsdays, #cfgmgmtcamp, #loadays,
….….

What's thisWhat's this
devops thingdevops thing
anyhow ?anyhow ?

World , 200X-2009World , 200X-2009
Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, JezzPatrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz
Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and lotsHumble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and lots
of others ..of others ..
Gent , October 2009Gent , October 2009
Mountain View , June 2010Mountain View , June 2010
Hamburg, Goteborg, Rome, ...Hamburg, Goteborg, Rome, ...
5th aniversary in Gent 20145th aniversary in Gent 2014
25+ events in 201625+ events in 2016

C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

The Old DaysThe Old Days
● ““Put this Code Live, here's a tarball” NOW!Put this Code Live, here's a tarball” NOW!
● What dependencies ?What dependencies ?
● No machines available ?No machines available ?
● What database ?What database ?
● Security ?Security ?
● High Availability ?High Availability ?
● Scalability ?Scalability ?
● My computer can't install this ?My computer can't install this ?

10 days into operations10 days into operations

11 days into operations11 days into operations

Some HistorySome History
LessonsLessons

openMosixopenMosix (+/- 2001-2005)(+/- 2001-2005)
● 2.4 Kernel patch,2.4 Kernel patch,
● Loadbalance multiple similar processesLoadbalance multiple similar processes
● Both short (compilations) and longlived (calculations)Both short (compilations) and longlived (calculations)
processesprocesses
● Parts of processes migrated to other nodesParts of processes migrated to other nodes
● OMFS allowed identical access to data from all nodesOMFS allowed identical access to data from all nodes

openMosix LimitationsopenMosix Limitations
● shared memoryshared memory
•
Limited type of applications could migrateLimited type of applications could migrate
•
Patches available but not successfulPatches available but not successful
● Filesystem accessFilesystem access
oMFS : unstable => removedoMFS : unstable => removed

Lessons from openMosixLessons from openMosix
● Applications need to be adaptedApplications need to be adapted
● Files need to be accessible (oMFS)Files need to be accessible (oMFS)
● Huge gap between developers and consumersHuge gap between developers and consumers
=> Limited working use cases=> Limited working use cases

Linux-HALinux-HA (2005- now)(2005- now)
● Hearbeat, Heartbeat v2, Pacemaker , CorosyncHearbeat, Heartbeat v2, Pacemaker , Corosync
● Define resource (OCF)Define resource (OCF)
● Define constraints, clones, colocation, …Define constraints, clones, colocation, …
● Long running services that should not stopLong running services that should not stop

Pacemaker ArchitecturePacemaker Architecture
● Stonithd : The Heartbeat fencing subsystem.
● Lrmd : Local Resource Management Daemon. Interacts
directly with resource agents (scripts).
● pengine Policy Engine. Computes the next state of the
cluster based on the current state and the configuration.
● cib Cluster Information Base. Contains definitions of all
cluster options, nodes, resources, their relationships to one
another and current status. Synchronizes updates to all
cluster nodes.
● crmd Cluster Resource Management Daemon. Largely a
message broker for the PEngine and LRM, it also elects a
leader to co-ordinate the activities of the cluster.
● openais messaging and membership layer.
● heartbeat messaging layer, an alternative to OpenAIS.
● ccm Short for Consensus Cluster Membership. The
Heartbeat membership layer.

Working with developersWorking with developers
● State vs StatelessState vs Stateless
•
Most applications have state somewhereMost applications have state somewhere
•
Discuss how to share/access stateDiscuss how to share/access state
● Data accessData access
•
Local filesystem is kingLocal filesystem is king
•
We needed distributed / shared filesystemsWe needed distributed / shared filesystems
● HealthHealth
● MetricsMetrics

Isolating ServicesIsolating Services
● 1 application per VM1 application per VM
•
Easy configurationEasy configuration
•
No accidental changesNo accidental changes
Anno 2005 – 2010 people called us crazyAnno 2005 – 2010 people called us crazy
● Multiple VM's with the same functions (A,B,001,002,256)Multiple VM's with the same functions (A,B,001,002,256)
● Isolate code from configIsolate code from config

Lessons from Linux-HALessons from Linux-HA
● Applications need to be adaptedApplications need to be adapted
● Files need to be accessible (shared/distributed FS)Files need to be accessible (shared/distributed FS)
● Monitoring strategy needs changeMonitoring strategy needs change
● Good collaboration between developers and ops folksGood collaboration between developers and ops folks
=> Many general purpose use cases=> Many general purpose use cases
● Works better with an Open Source MindsetWorks better with an Open Source Mindset

Impact on Monitoring/MetricsImpact on Monitoring/Metrics
● HA , monitoring a serviceHA , monitoring a service
•
Monitoring the individual serviceMonitoring the individual service
•
Monitoring/Ensuring the global serviceMonitoring/Ensuring the global service
● Health status of an applicationHealth status of an application
● Metric collectionMetric collection
● Alerting based on metrics rather than only status checksAlerting based on metrics rather than only status checks

Back to 2016Back to 2016

Cloud Adoption is (s)lowCloud Adoption is (s)low
● Enterprises are afraid cloudEnterprises are afraid cloud
•
Security, Cost, Control, Stability, (insert more fud here)Security, Cost, Control, Stability, (insert more fud here)
● ““Private cloud” will solve thisPrivate cloud” will solve this
● Please fill in 4 word documents for for each VM you wantPlease fill in 4 word documents for for each VM you want

Failed (Private) Cloud ProjectsFailed (Private) Cloud Projects
● Identical copies of Bare Metal or vm's moved to the cloudIdentical copies of Bare Metal or vm's moved to the cloud
● No config managementNo config management
● No monitoringNo monitoring
● No resilienceNo resilience
● Legacy procedures (still 5 week wait for a VM)Legacy procedures (still 5 week wait for a VM)
● No API usageNo API usage

These DaysThese Days
● ““Put this Code Live, here's Docker Container” NOW!Put this Code Live, here's Docker Container” NOW!
● No machines available ?No machines available ?
● What database ?What database ?
● Security ? What distro is this even ?Security ? What distro is this even ?
● High Availability ?High Availability ?
● Scalability ?Scalability ?
● How do we monitor his ?How do we monitor his ?
● How did you build this ?How did you build this ?

A typical Enterprise ContainerA typical Enterprise Container
● No different from a full vmNo different from a full vm
● Multiple services running inMultiple services running in
on containeron container
● Ssh is the default connectionSsh is the default connection

Why ?Why ?

IT Departments have not adapted,IT Departments have not adapted,
'Shadow'-IT is winning'Shadow'-IT is winning

Elephant , MonolithElephant , Monolith
vsvs
MicroservicesMicroservices

Can we Slice the Elephants ?Can we Slice the Elephants ?
{Devops/Cloud/Container}{Devops/Cloud/Container} adoption requires changeadoption requires change
•
In software architectureIn software architecture
•
In codeIn code
•
In infrastructureIn infrastructure
•
In mindsetIn mindset
•
In organisationIn organisation
•
In managementIn management

ContactContact
Kris Buytaert Kris.Buytaert@inuits.euKris Buytaert Kris.Buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
Inuits.euInuits.eu
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221