Slides for my Buildstuff.lt conference , #devops, #automation, #cams , Undeployable Software

1. Deploy Stuff, Run Stuff
Buildstuff.lt 2016
Kris Buytaert
@krisbuytaert

2. Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @Consultant @inuits.euinuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Some books, some papers, some blogsSome books, some papers, some blogs
● Evangelizing devopsEvangelizing devops
● Organiser of #devopsdays, #cfgmgmtcamp,Organiser of #devopsdays, #cfgmgmtcamp,
#loadays, ….#loadays, ….

3. What's this devopsWhat's this devops
thing anyhow ?thing anyhow ?

4. C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

5. Culture,Culture,
automation,automation,
Measturement,Measturement,
sharingsharing

6. the “old” daysthe “old” days
● ““Put this Code Live, here's a tarball” NOW!Put this Code Live, here's a tarball” NOW!
● What dependencies ?What dependencies ?
● No machines available ?No machines available ?
● What database ?What database ?
● Security ?Security ?
● High Availability ?High Availability ?
● Scalability ?Scalability ?
● My computer can't install this ?My computer can't install this ?

7. Blamefull OrganisationsBlamefull Organisations

8. HistoricallyDifferent GoalsHistoricallyDifferent Goals
DevelopmentDevelopment
● New releasesNew releases
● New FeaturesNew Features
● New platformsNew platforms
● New architecturesNew architectures
● Functional ReqFunctional Req
OperationsOperations
● Stable PlatformStable Platform
● No DowntimeNo Downtime
● Scalable PlatformScalable Platform
● Non Functional ReqNon Functional Req

9. MethodologiesMethodologies
● WaterfallWaterfall
● WatermillWatermill
● SAFESAFE
● Scrum
● Kanban

10. DOD in ScrumDOD in Scrum
•
DoD is a checklist of valuable activities requiredDoD is a checklist of valuable activities required
to produce software.to produce software.
•
Definition of Done is a simple list of activitiesDefinition of Done is a simple list of activities
(writing code, coding comments, unit testing,(writing code, coding comments, unit testing,
integration testing, release notes, designintegration testing, release notes, design
documents, etc.) that adddocuments, etc.) that add
verifiable/demonstrable value to the product.verifiable/demonstrable value to the product.
•
DoD is the primary reporting mechanism forDoD is the primary reporting mechanism for
team members.team members.

11. DoD in Scrum (2)DoD in Scrum (2)
a tool for bringing transparency to the work aa tool for bringing transparency to the work a
Scrum Team is performing. It is related more toScrum Team is performing. It is related more to
the quality of a product, rather than itsthe quality of a product, rather than its
functionality.functionality.

12. EnglishEnglish

13. Done means testedDone means tested

14. Done means deployedDone means deployed
In productionIn production

15. If it isn't monitored it isn't inIf it isn't monitored it isn't in
productionproduction
Done = In production andDone = In production and
MonitoredMonitored

16. Product vs ProjectProduct vs Project
● Frequent releasesFrequent releases
● UpdatesUpdates
● UpgradesUpgrades
● Frequent HandoversFrequent Handovers
● Fire & ForgetFire & Forget
● One HandoverOne Handover

17. A software project is not done untilA software project is not done until
your last enduser is in his grave !your last enduser is in his grave !
Kris Buytaert, #devopsdays Amsterdam 2013Kris Buytaert, #devopsdays Amsterdam 2013

18. Culture,Culture,
Automation,Automation,
Measurement,Measurement,
SharingSharing

19. What r u rambling bout ?What r u rambling bout ?
● Typical problems:Typical problems:
•
Non functional requirements are not part of aNon functional requirements are not part of a
sprintsprint

20. the un-buildablethe un-buildable
● The libraries you depend on have beenThe libraries you depend on have been
removed by the upstream authorremoved by the upstream author
•
Too oldToo old
•
AbandonnedAbandonned
● You haven't specified the versions and theYou haven't specified the versions and the
'latest' version is'latest' version is
•
BrokenBroken
•
incompatibleincompatible
● Rubybems.org,Github.com is downRubybems.org,Github.com is down

21. the un-packageablethe un-packageable

22. the un-packageablethe un-packageable

26. the un-deployablethe un-deployable
● You require manual changes to filesYou require manual changes to files
•
Files in different locationsFiles in different locations
● Your haven't versionned all the filesYour haven't versionned all the files
•
Plenty of files are missingPlenty of files are missing
•
Missing librariesMissing libraries
● Do you really expect everyone to know how toDo you really expect everyone to know how to
deploy your Flavour of the week stackdeploy your Flavour of the week stack

27. ““If my computerIf my computer
can't install it,can't install it,
the installer isthe installer is
broken”broken”
Luke Kanies atLuke Kanies at
Fosdem (2007)Fosdem (2007)

28. the un-configurablethe un-configurable
● Sometimes the preconfigured oneSometimes the preconfigured one
● Config is inside the build artifactConfig is inside the build artifact
•
.war, binary, code.war, binary, code
● Redeploy / restart is required to reconfigureRedeploy / restart is required to reconfigure
● Log on with the credentials generated inLog on with the credentials generated in
/tmp/blahX2312/tmp/blahX2312
● Click on the 3rd button on the left, then downClick on the 3rd button on the left, then down
and first on the right.and first on the right.

29. As an Ops personAs an Ops person
““As a system administrator, I can tell whenAs a system administrator, I can tell when
software vendors hate me. It shows in theirsoftware vendors hate me. It shows in their
products.”products.”
““DON'T make the administrative interface aDON'T make the administrative interface a
GUI. System administrators need aGUI. System administrators need a
command-line tool for constructingcommand-line tool for constructing
repeatable processes. Procedures are bestrepeatable processes. Procedures are best
documented by providing commands thatdocumented by providing commands that
we can copy and paste from the procedurewe can copy and paste from the procedure
document to the command line. We cannotdocument to the command line. We cannot
achieve the same repeatability when theachieve the same repeatability when the
instructions are: "Checkmark the 3rd andinstructions are: "Checkmark the 3rd and
5th options, but not the 2nd option, then5th options, but not the 2nd option, then
click OK." Sysadmins do not want a GUI thatclick OK." Sysadmins do not want a GUI that
requires 25 clicks for each new user.”requires 25 clicks for each new user.”
Thomas A. Limoncelli in ACM Queue December 2010Thomas A. Limoncelli in ACM Queue December 2010
http://queue.acm.org/detail.cfm?id=1921361http://queue.acm.org/detail.cfm?id=1921361

30. the un-runnablethe un-runnable
● You shipped a .exe fileYou shipped a .exe file
•
We run on LinuxWe run on Linux
● You shipped a Docker fileYou shipped a Docker file
•
We have no containerized infrastructureWe have no containerized infrastructure
● Your application connects to 10.0.0.1:3306Your application connects to 10.0.0.1:3306
● Your application connects toYour application connects to
myapp.ourcompany.commyapp.ourcompany.com
● Your code requires Python 5Your code requires Python 5
•
We are on EL 7We are on EL 7

31. the un-clusterablethe un-clusterable
● Where is your state ?Where is your state ?
● How do you know where to connect to ?How do you know where to connect to ?
● /tmp is not a distributed filesystem/tmp is not a distributed filesystem

32. the un-cloudablethe un-cloudable
● Where is your state ?Where is your state ?

33. the un-securedthe un-secured
● Clear text passwords in config filesClear text passwords in config files
● Passwords as parametersPasswords as parameters
● No encryptionNo encryption
● Vague ssl version build inVague ssl version build in
● No authentication at allNo authentication at all
● Users are stored locallyUsers are stored locally

34. Culture,Culture,
Automation,Automation,
Measurement :Measurement :
measure all the thingsmeasure all the things
SharingSharing

35. the un-monitorablethe un-monitorable
● Is this thing even on ?Is this thing even on ?
● Logs ?Logs ?
● Log verbosity ?Log verbosity ?
● Log corelation ?Log corelation ?
● Consistent output for testingConsistent output for testing
● If the enduser calls, your monitoring has failedIf the enduser calls, your monitoring has failed

36. the un-measurablethe un-measurable
● No metrics,No metrics,
● No log entries to derive metrics from,No log entries to derive metrics from,
● Please providePlease provide
•
MetricsMetrics
•
Health / status page / apiHealth / status page / api
•
LogsLogs

37. Culture,Culture,
Automation,Automation,
Measurement,Measurement,
SharingSharing

39. This new 'D' hypeThis new 'D' hype
● New kid on the blockNew kid on the block
● Vagrant-lxc with a nice cliVagrant-lxc with a nice cli
● The Ultimate “devops tool”The Ultimate “devops tool”
● ““Unseen” growthUnseen” growth
● Docker is the new cool thing to doDocker is the new cool thing to do

40. DockerDocker
DockerDocker
DockerDocker
DockerDocker
DockerDocker
DockerDocker

41. A typical EnterpriseA typical Enterprise
ContainerContainer● No different from aNo different from a
full vmfull vm
● Multiple servicesMultiple services
running in onerunning in one
containercontainer
● Ssh is the defaultSsh is the default
connectionconnection

42. Why ?Why ?
● I want a VMI want a VM
•
Please fill in these 4 formsPlease fill in these 4 forms
•
Wait 2 weeks.Wait 2 weeks.
● RepeatRepeat
IT Departments have not adapted,IT Departments have not adapted,
'Shadow'-IT is winning'Shadow'-IT is winning

43. These DaysThese Days
● ““Put this Code Live, here's a DockerPut this Code Live, here's a Docker
Container ”Container ”
● No machines available ?No machines available ?
● What database ? Where to store theWhat database ? Where to store the
data ?data ?
● Security ? What distro is this even ?Security ? What distro is this even ?
Bad Cows ?Bad Cows ?
● How do we monitor his ?How do we monitor his ?
● Backups ?Backups ?
● How did you build this ?How did you build this ?

44. 11 days into operations11 days into operations
● ““Put this Code Live, here'sPut this Code Live, here's
Dockerfile”Dockerfile”
● What corporate proxy ?What corporate proxy ?
● Oh I missed 2 other containersOh I missed 2 other containers
● Security ? What distro is this even ?Security ? What distro is this even ?
Bad Cows ?Bad Cows ?
● What do you mean “We are a RHELWhat do you mean “We are a RHEL
shop ?”shop ?”

45. Closing the gaps between dev and ops,Closing the gaps between dev and ops,
AGAIN !!AGAIN !!
● Where do your containers come from ?Where do your containers come from ?
● Who build it ?Who build it ?
● Can you rebuild it ?Can you rebuild it ?
● Do you even need a containerDo you even need a container
● How do you build the hosts that run theHow do you build the hosts that run the
containers ?containers ?
● Infrastructure as code ++Infrastructure as code ++

46. Image Build by devs,Image Build by devs,
maintained by nobodymaintained by nobody

47. Can you ?Can you ?
● When GitHub is down ?When GitHub is down ?
● When rubygems.org isWhen rubygems.org is
down ?down ?
● When someone removesWhen someone removes
a Node.js library ?a Node.js library ?
● Fix critical SecurityFix critical Security
Issues ?Issues ?
● Can your business suriveCan your business surive
if you answer NO toif you answer NO to
these questions ?these questions ?

48. ● I love docker as aI love docker as a
technologytechnology
● I hate that it too oftenI hate that it too often
put us back 5 years withput us back 5 years with
regards to Cultureregards to Culture
adoptionadoption
● Docker is an easy victim,Docker is an easy victim,
there's other tools thatthere's other tools that
create similar situationscreate similar situations
● It's still mostly aboutIt's still mostly about
CultureCulture
It's still aboutIt's still about
collaborationcollaboration

49. It's not about the toolsIt's not about the tools
It's about changeIt's about change
It's about the peopleIt's about the people

50. ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221