Slides for my #devopsdays Riga talk "The Return of the Dull Stack Engineer"

1. The Return of the DullThe Return of the Dull
Stack EngineerStack Engineer
Kris Buytaert
@krisbuytaert

2. Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● CTO and Open Source ConsultantCTO and Open Source Consultant
@@inuits.euinuits.eu
● Everything is a freaking DNSEverything is a freaking DNS
ProblemProblem
● Evangelizing devopsEvangelizing devops
● Organiser of #devopsdays,Organiser of #devopsdays,
#cfgmgmtcamp, #loadays, ….#cfgmgmtcamp, #loadays, ….

3. C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and JohnDamon Edwards and John
Willis

4. A global movement to improve theA global movement to improve the
quality of software delivery leveragingquality of software delivery leveraging
Open Source experience, started in GentOpen Source experience, started in Gent
in 2009in 2009

5. Because the old days:Because the old days:
● ““Put this Code Live, here's a tarball”Put this Code Live, here's a tarball”
NOW!NOW!
● What dependencies ?What dependencies ?
● No machines available ?No machines available ?
● What database ?What database ?
● Security ?Security ?
● High Availability ?High Availability ?

6. 10 days into operations10 days into operations

7. Topic Evolution : CultureTopic Evolution : Culture
● Agile, Kanban, LeanAgile, Kanban, Lean
● Startups as examplesStartups as examples
● UnicornsUnicorns
● Poster ChildsPoster Childs
● Agile, Fake Agile, Less,Agile, Fake Agile, Less,
Safe,Safe,
● Enterprises asEnterprises as
examplesexamples
● Engage managementEngage management
● BurnoutBurnout

8. Topic Evolution : AutomationTopic Evolution : Automation
● Cfengine, Puppet, ChefCfengine, Puppet, Chef
● Mcollective, Noah,Mcollective, Noah,
RundeckRundeck
● Hudson, JenkinsHudson, Jenkins
● EC2, OpenStack,EC2, OpenStack,
Cloudstack,Cloudstack,
Eucalyptus,OpenNebulEucalyptus,OpenNebul
a, openQRMa, openQRM
● Bash in Yaml “Basic”Bash in Yaml “Basic”
● Ansible, Rundeck,Ansible, Rundeck,
ChoriaChoria
● Jenkins, GitlabCI,Jenkins, GitlabCI,
CircleCI, TravisCICircleCI, TravisCI
● EC2, GCE, Azure,EC2, GCE, Azure,
OpenStack,OpenStack,

9. Topic Evolution : AutomationTopic Evolution : Automation
● Xen, KVM, openVZ, lxcXen, KVM, openVZ, lxc
● VagrantVagrant
● Linux-HA, CoroSyncLinux-HA, CoroSync
● Ceph, Gluster, DRBD,Ceph, Gluster, DRBD,
● Docker , Docker,Docker , Docker,
Docker, Docker, Moby,Docker, Docker, Moby,
Rkt, lxcRkt, lxc
● CR-IOCR-IO
● Docker , Docker,Docker , Docker,
Dockrer,Mesos,Dockrer,Mesos,
KubernetesKubernetes
● NFSNFS

10. Topic Evolution : MonitoringTopic Evolution : Monitoring
● NagiosNagios
● CactiCacti
● MuninMunin
● RRDRRD
● Zenoss, ZabbixZenoss, Zabbix
● #monitoringsucks#monitoringsucks
● IcingaIcinga
● ELKELK
● Graphite + GrafanaGraphite + Grafana
● CollectdCollectd
● PrometheusPrometheus
● #monitoringlove#monitoringlove

11. Culture vs ToolsCulture vs Tools

12. These DaysThese Days
● ““Put this Code Live, here's a DockerPut this Code Live, here's a Docker
Container ”Container ”
● No machines available ?No machines available ?
● What database ? Where to store theWhat database ? Where to store the
data ?data ?
● Security ? What distro is this even ?Security ? What distro is this even ?
Bad Cows ?Bad Cows ?
● How do we monitor his ?How do we monitor his ?
● Backups ?Backups ?
● How did you build this ?How did you build this ?

13. Tools will not fix yourTools will not fix your
broken culturebroken culture
Tooling hype is not helpingTooling hype is not helping

14. Hype SeasonHype Season
● Conference Visitors numbers skyrocketConference Visitors numbers skyrocket
•
OpenStack 3000+ marketeersOpenStack 3000+ marketeers
•
DockerCon : 1000+ devs & unicornsDockerCon : 1000+ devs & unicorns
•
Kube Con : 1500+ unicorns & vendorsKube Con : 1500+ unicorns & vendors
● Everybody claims to be doing it , nobodyEverybody claims to be doing it , nobody
is actually doing it.is actually doing it.

15. Why adopt shiny new stuff ?Why adopt shiny new stuff ?
● Attention Span ?Attention Span ?
● Resume building ?Resume building ?
•
0.5 year = novice in 20 techs0.5 year = novice in 20 techs
vsvs
•
7 year = expert in 4 techs7 year = expert in 4 techs

16. Works on My MachineWorks on My Machine

17. New != BetterNew != Better
● It’s called Bleeding edge for aIt’s called Bleeding edge for a
reasonreason
● Missing FeaturesMissing Features
● OperatabilityOperatability
● StabilityStability

18. Hello world todayHello world today
● Micro service architectureMicro service architecture
● Displaying content in a fileDisplaying content in a file
● Kubernetes 2.6+ StorageKubernetes 2.6+ Storage
● OpenStackOpenStack

19. Complexity is theComplexity is the
enemy of reliabilityenemy of reliability

20. U are not facebookU are not facebook

21. U are not LinkedinU are not Linkedin
● Kafka is awesomeKafka is awesome
● But you have 100 messages an hourBut you have 100 messages an hour

22. Welcome to EuropeWelcome to Europe

23. Why don’t ops adopt ?Why don’t ops adopt ?
● They do ..They do ..
•
It just takes more timeIt just takes more time
•
More worriesMore worries
•
More careMore care

24. Improve your sleepImprove your sleep
Put your {devs,managers,qa}Put your {devs,managers,qa}
ON CALL !ON CALL !

25. Why don’t ops adopt ?Why don’t ops adopt ?
● We already fixed that problemWe already fixed that problem
● We have a different use caseWe have a different use case
● That won’t work becauseThat won’t work because
•
AA
•
BB
•
CC
● Cool lets try it ..Cool lets try it ..

26. Puppet AdoptionPuppet Adoption
● Focus on the wrong partFocus on the wrong part
•
Expert in PuppetExpert in Puppet
•
Doesn’t understand apache configDoesn’t understand apache config

27. Operations trying to adoptOperations trying to adopt

28. $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
● We don’t have timeWe don’t have time
● Initial Budget was too low,Initial Budget was too low,
unrealisticunrealistic
● Existing Technical DebtExisting Technical Debt
● FirefightingFirefighting
● Time pressured projectTime pressured project

29. Operations
Developers
Security

30. Workload is invertedWorkload is inverted
● X devs work on 1-2 concurrentX devs work on 1-2 concurrent
projectsprojects
● X/20 ops/security work on all theX/20 ops/security work on all the
projects the devs touched over theprojects the devs touched over the
past 5 yearspast 5 years
● AND need to build the next genAND need to build the next gen
platformplatform

31. ● GrumpyGrumpy
•
Woken up oneWoken up one
to many timesto many times
for the wrongfor the wrong
reasonreason
● Old , akaOld , aka
ExperiencedExperienced
Typical Ops DudeTypical Ops Dude

34. Devops = CoolDevops = Cool
● All the cool kids are doing thisAll the cool kids are doing this
● We want to be coolWe want to be cool
● People only want to work for coolPeople only want to work for cool
companiescompanies
● Banks etc are boringBanks etc are boring

35. FOMOFOMO
● We need to do something CoolWe need to do something Cool
•
Docker Docker DockerDocker Docker Docker
•
Bi ModalBi Modal

36. Broken StrategiesBroken Strategies
● Hiring Devops EngineersHiring Devops Engineers
● Building Devops TeamsBuilding Devops Teams
● Docker Docker Docker => VM’s asDocker Docker Docker => VM’s as
ContainersContainers
s/ops/oops/g;s/ops/oops/g;

37. Tools will not fix yourTools will not fix your
broken culturebroken culture
Tooling hype is not helpingTooling hype is not helping

39. YOLOYOLO
● StartupStartup
● VCVC
● Exit StrategyExit Strategy
● BusinessBusiness
● CustomerCustomer
● SurvivalSurvival

40. Late AdoptionLate Adoption
● You need to do IACYou need to do IAC
● We don’t need IACWe don’t need IAC
● You really need to do IACYou really need to do IAC
● IAC is too difficult for us ..IAC is too difficult for us ..
● Really you need to do IACReally you need to do IAC
● We’re doing Ansible now …We’re doing Ansible now …

42. 0 day software0 day software
0900: “Whow Hashicorp released0900: “Whow Hashicorp released
Consul X.Y”Consul X.Y”
Let’s have this in prod by thisLet’s have this in prod by this
afternoonafternoon
Ego driven developmentEgo driven development

43. R&D vs ProdR&D vs Prod
● Schedule time to Research new topicsSchedule time to Research new topics
● Not all of them are irrelevantNot all of them are irrelevant
● But limit the number of patterns you need to manageBut limit the number of patterns you need to manage
● e.g Limited number of OS Variants, Deploymente.g Limited number of OS Variants, Deployment
Patterns, Storage BackendsPatterns, Storage Backends

44. Stop DreamingStop Dreaming
There is no way your 3 person ops team can manageThere is no way your 3 person ops team can manage
and scale VMWare, EC2, Azure, OpenStack, RHEV,and scale VMWare, EC2, Azure, OpenStack, RHEV,
Gluster, DRBD, LibVirt,NFS, Ceph, Mongo, MySQL,Gluster, DRBD, LibVirt,NFS, Ceph, Mongo, MySQL,
PgSQL, Cassandra, mcollective, Rabbit, collectd,PgSQL, Cassandra, mcollective, Rabbit, collectd,
dhcpd, sonar, aptly,ActiveMQ, Kafka, Nomad,dhcpd, sonar, aptly,ActiveMQ, Kafka, Nomad,
zookeeper, haproxy, iptables, Consul, Etcd, Puppet,zookeeper, haproxy, iptables, Consul, Etcd, Puppet,
Cacti, corosync, Docker Distribution, vault, bind,Cacti, corosync, Docker Distribution, vault, bind,
owncloud, openswan,Nexus, Redmine, diamond,owncloud, openswan,Nexus, Redmine, diamond,
Jenkins,Mule, GoCD, Chef, memcached, foreman, PHP,Jenkins,Mule, GoCD, Chef, memcached, foreman, PHP,
Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot,Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot,
Ansible, Django, Apache Storm, redis, Nginx, ApacheAnsible, Django, Apache Storm, redis, Nginx, Apache
httpd, Jboss, Jetty, Hadoop, Elastic, Graphite, Grafana,httpd, Jboss, Jetty, Hadoop, Elastic, Graphite, Grafana,
freeswan, Graylog, Apache Spark, Bamboo, Jirafreeswan, Graylog, Apache Spark, Bamboo, Jira

45. Stop DreamingStop Dreaming
There is no way your 3 person ops team can manageThere is no way your 3 person ops team can manage
and scaleand scale VMWare,VMWare, EC2,EC2, Azure,Azure, OpenStack,OpenStack, RHEV,RHEV,
Gluster, DRBD, LibVirt,Gluster, DRBD, LibVirt,NFS,NFS, Ceph, Mongo, MySQL,Ceph, Mongo, MySQL,
PgSQL,PgSQL, Cassandra,Cassandra, mcollective, Rabbit, collectd,mcollective, Rabbit, collectd,
dhcpd, sonar, aptly,ActiveMQ,dhcpd, sonar, aptly,ActiveMQ, KafkaKafka,, Nomad,Nomad,
zookeeper, haproxy, iptables, Consul, Etcd, Puppet,zookeeper, haproxy, iptables, Consul, Etcd, Puppet,
Cacti,Cacti, corosync,corosync, Docker Distribution,Docker Distribution, vault, bind,vault, bind,
owncloud, openswan,Nexus, Redmine,owncloud, openswan,Nexus, Redmine, diamond,diamond,
Jenkins,Jenkins,Mule, GoCD, Chef, memcached,Mule, GoCD, Chef, memcached, foreman, PHP,foreman, PHP,
Java,Ruby, nextcloud, Kubernetes, pulp,Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot,Spring Boot,
Ansible, Django, Apache Storm, redis, Nginx, ApacheAnsible, Django, Apache Storm, redis, Nginx, Apache
httpd,httpd, Jboss, Jetty,Jboss, Jetty, Hadoop,Hadoop, Elastic, Graphite, Grafana,Elastic, Graphite, Grafana,
freeswan, Graylog, Apache Spark, Bamboo, Jirafreeswan, Graylog, Apache Spark, Bamboo, Jira

46. Full Stack EngineerFull Stack Engineer
● Knows the MEAN StackKnows the MEAN Stack
● Is still confused aboutIs still confused about
the Javascriptthe Javascript
framework he wants toframework he wants to
use yesterdayuse yesterday
● Is familiar with all theIs familiar with all the
previously mentionedpreviously mentioned
toolstools
● Understands the LinuxUnderstands the Linux
KernelKernel
● Debugs Java CodeDebugs Java Code
● Understands NetworkingUnderstands Networking
and Securityand Security
● Understands deploymentUnderstands deployment
● Tunes DatabasesTunes Databases

47. Dull Stack EngineerDull Stack Engineer

48. Dull Stack EngineerDull Stack Engineer
● Knows the stack in depthKnows the stack in depth
● Invests in peopleInvests in people
● Knows his coworkers in depthKnows his coworkers in depth
● Is there for the business, not for theIs there for the business, not for the
fun of the new toolfun of the new tool

49. $value$value
● Devops is about generating valueDevops is about generating value
for your organisation , togetherfor your organisation , together
● What’s the value of shiny $tool vsWhat’s the value of shiny $tool vs
boring tool ?boring tool ?

50. Back to the rootsBack to the roots
● Collaborating for a better platformCollaborating for a better platform
● Open SourceOpen Source
● EmpathyEmpathy
● Common SenseCommon Sense

51. ContactContact
Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
Find Inuits inFind Inuits in
Brasschaat,GhentBrasschaat,Ghent
,,
Rotterdam,PraguRotterdam,Pragu
e,e,