Lessons learned from deploying an operational SAAS stack On Prem at a Customer

1. Building and Deploying a Saas platform
On Prem
Kris Buytaert
@krisbuytaert
Slides by Michel van de Ven and
Julien Pivotto

2. Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Some books, some papers, some blogsSome books, some papers, some blogs
● Evangelizing devopsEvangelizing devops

3. Inuits
● Inuits is an Open Source company
•
We contribute backWe contribute back
● +85 people in 4 countries (BE, NL, UA,CZ)
● One language: English
● We offer
•
ConsultingConsulting
•
DevelopmentDevelopment
•
DevOopsDevOops
•
Ops ..Ops ..
•
and multiple niche Saas Platformsand multiple niche Saas Platforms

4. Use Case
=
MediaMosa as a Service

5. MediaSalsa infrastructure (simplified)
● For each environment (DTAP)
•
Backends: Core service (MediaMosa)Backends: Core service (MediaMosa)
•
Frontends: OptionalFrontends: Optional
•
Web serversWeb servers
•
Database serversDatabase servers
•
Solr serversSolr servers
•
Transcoding serversTranscoding servers
•
Streaming Servers ..Streaming Servers ..
•
……..

6. Culture
Automation
Measurement
Sharing

7. Puppet
Puppet automates all the things
→ mcollective orchestrates all the things

8. CD
● Continuous Delivery vs Continuous Deployment
•
““Continuous Delivery doesn't mean every change is deployed toContinuous Delivery doesn't mean every change is deployed to
production ASAP. It means every change is proven to be deployable atproduction ASAP. It means every change is proven to be deployable at
any time” (@ccaum)any time” (@ccaum)
● Puppet code
•
Deployed to dev environmentDeployed to dev environment
•
Same puppet code for each environmentSame puppet code for each environment
•
User-triggered deployments to UAT & ProdUser-triggered deployments to UAT & Prod
•
Feature flags in Puppet code per environment (switchable architecture)Feature flags in Puppet code per environment (switchable architecture)
● Application code
•
Continuous integration in devContinuous integration in dev
•
Continously deploy to UATContinously deploy to UAT
•
Continuously deploy on our prod stackContinuously deploy on our prod stack
•
Continuously wait for customer permission on PremContinuously wait for customer permission on Prem

9. Testing
● Developers test a lot, but
•
The tests don’t workThe tests don’t work
•
It works on my machineIt works on my machine™™
•
Wrong platformWrong platform
•
Wrong PHP versionWrong PHP version
Fixed now, thanks to Jenkins!Fixed now, thanks to Jenkins!
Fixed now, thanks to JenkinsFixed now, thanks to Jenkins
JobDSL / Pipeline as CodeJobDSL / Pipeline as Code

10. Version Control
● Git
● Code is under revision control
•
Prefer small commitsPrefer small commits
•
Trunk based developmentTrunk based development
•
Branches are considered EVILBranches are considered EVIL
•
Release management = Git submodulesRelease management = Git submodules
● Infrastructure as code → git / hiera

11. Using OS packaging system
● Consistency, security, dependencies
● Uniquely identify where files are coming from
● Source repo may not be reachable
● Little overhead when you automate
● Configuration does not belong in a package
● PCS Pattern

12. Repository ManagementRepository Management

13. Pipelines
● A collection of jobs
● Run in sequence
● Start on checkout, end on deployment
● From the developers’ side:
→ Git push
← Mail with changes + link to deploy

14. Pipelines steps

15. Scaling PipelinesScaling Pipelines
● Create a Pipeline,Create a Pipeline,
● For job in PipelineFor job in Pipeline
•
Create new Job Based on OldJobCreate new Job Based on OldJob
● Update One JobUpdate One Job
● Never refactor the restNever refactor the rest

16. Complex PipelinesComplex Pipelines

17. Jenkins Job DSLJenkins Job DSL
● GroovyGroovy
● FlexibleFlexible
● Well SupportedWell Supported
● Suitable for more complex PipelinesSuitable for more complex Pipelines
https://jenkinsci.github.io/job-dsl-https://jenkinsci.github.io/job-dsl-
plugin/plugin/

18. SeedjobsSeedjobs
● GroovyGroovy
● GitGit
● Rebuild jobs on commitRebuild jobs on commit
● Projects in foldersProjects in folders

19. Same Pipelines, Tools, Patterns
are used by both devs and ops

20. Culture
Automation
Measurement
Sharing

21. Icinga
● Monitor everything
•
vhostsvhosts
•
databasesdatabases
•
cronjobscronjobs
•
unit test suitesunit test suites
● Automated using Puppet
•
Exported resources / Stored configsExported resources / Stored configs
● If it is deployed it monitored

22. Logstash
● Collect all the logs
•
Drupal logsDrupal logs
•
Apache logsApache logs
•
Deployment logsDeployment logs
•
System logsSystem logs
● Interprete, filter and correlate them
● Logstash, ElasticSearch, Kibana, statsd, Graphite,
Grafana

23. Collectd+Graphite+Grafana
● Measure all the things
•
cpu/memcpu/mem
•
databasesdatabases
•
Application metricsApplication metrics
•
Derived from logsDerived from logs
•
Business MetricsBusiness Metrics

24. Culture
Automation
Measurement
Sharing

25. Open Source
● Mediamosa is Fully Open Source
● Lots of the PuppetCode to deploy it is
● Our passwords etc aren't

26. e.g MediaSalsa Deployments
● Initially
•
1 instance Academic usage @SurfNet1 instance Academic usage @SurfNet
•
1 Instance Commercial DC for non-edu1 Instance Commercial DC for non-edu
● Today
•
2 academic instances2 academic instances
•
1 commercial Saas instance1 commercial Saas instance
•
2 on prem deployments2 on prem deployments

27. Why multiple Deployments
● “Security”
•
Academic Customer wanted a private tenant for securityAcademic Customer wanted a private tenant for security
and privacyand privacy
● Initial hardware investement done already
•
Public Tender , $customer bought huge amount of storagePublic Tender , $customer bought huge amount of storage
•
Saas solution charges per TBSaas solution charges per TB
•
Asked for custom manual deploymentAsked for custom manual deployment
● CIO’s still don’t believe in Cloud/SAAS (2017 !!!!)

28. Saas vs OnPrem
We have automated everything,
Infrastructure as Code , Pipeline as Code,
Continuous delivery , so deploying this stack
another time should be trivial !!

29. WRONG

30. Biased Automation
● Works in our infra , our constraints, our expectations
● We expect to have access to our infra
•
Puppet, monitoring, metrics, repos , jenkins, dnsPuppet, monitoring, metrics, repos , jenkins, dns

31. VM Provisioning
● Different Technologies
•
Open vs ProprietaryOpen vs Proprietary
•
Guess which one is more problematicGuess which one is more problematic
● No access to Internal repositories
•
(Pulp Mirrors)(Pulp Mirrors)
● Network topologies
● Having to ask to reboot a host
● Having to ask to grow a VM

32. Security
● IPSec links to all stacks
•
Our own network complexity has grown exponentiallyOur own network complexity has grown exponentially
•
Overlapping networksOverlapping networks
•
Introducing BGP etcIntroducing BGP etc
● Our network = Trusted
● Their network = Hostile
•
Different approach in host vs network based firewallingDifferent approach in host vs network based firewalling
● User management
•
Only our accounts in our stack , our ldapOnly our accounts in our stack , our ldap
•
They want accountsThey want accounts

33. Variants
● We don’t want exceptions
● They do want exceptions
● Old purchasing mentality
•
Custom FeaturesCustom Features
•
Additional ComponentsAdditional Components
•
It’s “Their” stackIt’s “Their” stack
● Exceptions need to be codified in our infra

34. Continuous Deployment Delivery
● Deployment isn’t our decision anymore
● Back to fixed deployment windows :(
● Coordination with $customer on when to deploy
● Even for Security Fixes
● For every single instance (except the public SAAS
one)

35. Extreme Cost Difference
● The effort to run 5 stacks in your own infrastructure within your team is
smaller than running 1 additional stack on prem at a customer
● Your pragmatic approach does not fit their infrastructure
● You will need to implement features (security/ storage support) that you
do not need for your SAAS platform.
● Less movement room to fix stuff .
•
e.g no “freely” available extra resources disk / memory / cpue.g no “freely” available extra resources disk / memory / cpu
•
More complex solutions (e.g DBA work..)More complex solutions (e.g DBA work..)

36. It could have been worse
● We are an Open Source company
● All of our Choices are Open Source by default
•
We could deploy full stacks On PremWe could deploy full stacks On Prem
•
Including metrics, log analytics and monitoringIncluding metrics, log analytics and monitoring
•
We had no external dependenciesWe had no external dependencies
•
No additional license costsNo additional license costs

37. If you can choose ..
don’t ...don’t ...

38. ContactContact
Kris Buytaert Kris.Buytaert@inuits.euKris Buytaert Kris.Buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
Inuits.euInuits.eu
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221