Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deploying your SaaS stack OnPrem

155 views

Published on

Lessons learned from deploying an operational SAAS stack On Prem at a Customer

Published in: Technology
  • Be the first to comment

Deploying your SaaS stack OnPrem

  1. 1. Building and Deploying a Saas platform On Prem Kris Buytaert @krisbuytaert Slides by Michel van de Ven and Julien Pivotto
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Some books, some papers, some blogsSome books, some papers, some blogs ● Evangelizing devopsEvangelizing devops
  3. 3. Inuits ● Inuits is an Open Source company • We contribute backWe contribute back ● +85 people in 4 countries (BE, NL, UA,CZ) ● One language: English ● We offer • ConsultingConsulting • DevelopmentDevelopment • DevOopsDevOops • Ops ..Ops .. • and multiple niche Saas Platformsand multiple niche Saas Platforms
  4. 4. Use Case = MediaMosa as a Service
  5. 5. MediaSalsa infrastructure (simplified) ● For each environment (DTAP) • Backends: Core service (MediaMosa)Backends: Core service (MediaMosa) • Frontends: OptionalFrontends: Optional • Web serversWeb servers • Database serversDatabase servers • Solr serversSolr servers • Transcoding serversTranscoding servers • Streaming Servers ..Streaming Servers .. • ……..
  6. 6. Culture Automation Measurement Sharing
  7. 7. Puppet Puppet automates all the things → mcollective orchestrates all the things
  8. 8. CD ● Continuous Delivery vs Continuous Deployment • ““Continuous Delivery doesn't mean every change is deployed toContinuous Delivery doesn't mean every change is deployed to production ASAP. It means every change is proven to be deployable atproduction ASAP. It means every change is proven to be deployable at any time” (@ccaum)any time” (@ccaum) ● Puppet code • Deployed to dev environmentDeployed to dev environment • Same puppet code for each environmentSame puppet code for each environment • User-triggered deployments to UAT & ProdUser-triggered deployments to UAT & Prod • Feature flags in Puppet code per environment (switchable architecture)Feature flags in Puppet code per environment (switchable architecture) ● Application code • Continuous integration in devContinuous integration in dev • Continously deploy to UATContinously deploy to UAT • Continuously deploy on our prod stackContinuously deploy on our prod stack • Continuously wait for customer permission on PremContinuously wait for customer permission on Prem
  9. 9. Testing ● Developers test a lot, but • The tests don’t workThe tests don’t work • It works on my machineIt works on my machine™™ • Wrong platformWrong platform • Wrong PHP versionWrong PHP version Fixed now, thanks to Jenkins!Fixed now, thanks to Jenkins! Fixed now, thanks to JenkinsFixed now, thanks to Jenkins JobDSL / Pipeline as CodeJobDSL / Pipeline as Code
  10. 10. Version Control ● Git ● Code is under revision control • Prefer small commitsPrefer small commits • Trunk based developmentTrunk based development • Branches are considered EVILBranches are considered EVIL • Release management = Git submodulesRelease management = Git submodules ● Infrastructure as code → git / hiera
  11. 11. Using OS packaging system ● Consistency, security, dependencies ● Uniquely identify where files are coming from ● Source repo may not be reachable ● Little overhead when you automate ● Configuration does not belong in a package ● PCS Pattern
  12. 12. Repository ManagementRepository Management
  13. 13. Pipelines ● A collection of jobs ● Run in sequence ● Start on checkout, end on deployment ● From the developers’ side: → Git push ← Mail with changes + link to deploy
  14. 14. Pipelines steps
  15. 15. Scaling PipelinesScaling Pipelines ● Create a Pipeline,Create a Pipeline, ● For job in PipelineFor job in Pipeline • Create new Job Based on OldJobCreate new Job Based on OldJob ● Update One JobUpdate One Job ● Never refactor the restNever refactor the rest
  16. 16. Complex PipelinesComplex Pipelines
  17. 17. Jenkins Job DSLJenkins Job DSL ● GroovyGroovy ● FlexibleFlexible ● Well SupportedWell Supported ● Suitable for more complex PipelinesSuitable for more complex Pipelines https://jenkinsci.github.io/job-dsl-https://jenkinsci.github.io/job-dsl- plugin/plugin/
  18. 18. SeedjobsSeedjobs ● GroovyGroovy ● GitGit ● Rebuild jobs on commitRebuild jobs on commit ● Projects in foldersProjects in folders
  19. 19. Same Pipelines, Tools, Patterns are used by both devs and ops
  20. 20. Culture Automation Measurement Sharing
  21. 21. Icinga ● Monitor everything • vhostsvhosts • databasesdatabases • cronjobscronjobs • unit test suitesunit test suites ● Automated using Puppet • Exported resources / Stored configsExported resources / Stored configs ● If it is deployed it monitored
  22. 22. Logstash ● Collect all the logs • Drupal logsDrupal logs • Apache logsApache logs • Deployment logsDeployment logs • System logsSystem logs ● Interprete, filter and correlate them ● Logstash, ElasticSearch, Kibana, statsd, Graphite, Grafana
  23. 23. Collectd+Graphite+Grafana ● Measure all the things • cpu/memcpu/mem • databasesdatabases • Application metricsApplication metrics • Derived from logsDerived from logs • Business MetricsBusiness Metrics
  24. 24. Culture Automation Measurement Sharing
  25. 25. Open Source ● Mediamosa is Fully Open Source ● Lots of the PuppetCode to deploy it is ● Our passwords etc aren't
  26. 26. e.g MediaSalsa Deployments ● Initially • 1 instance Academic usage @SurfNet1 instance Academic usage @SurfNet • 1 Instance Commercial DC for non-edu1 Instance Commercial DC for non-edu ● Today • 2 academic instances2 academic instances • 1 commercial Saas instance1 commercial Saas instance • 2 on prem deployments2 on prem deployments
  27. 27. Why multiple Deployments ● “Security” • Academic Customer wanted a private tenant for securityAcademic Customer wanted a private tenant for security and privacyand privacy ● Initial hardware investement done already • Public Tender , $customer bought huge amount of storagePublic Tender , $customer bought huge amount of storage • Saas solution charges per TBSaas solution charges per TB • Asked for custom manual deploymentAsked for custom manual deployment ● CIO’s still don’t believe in Cloud/SAAS (2017 !!!!)
  28. 28. Saas vs OnPrem We have automated everything, Infrastructure as Code , Pipeline as Code, Continuous delivery , so deploying this stack another time should be trivial !!
  29. 29. WRONG
  30. 30. Biased Automation ● Works in our infra , our constraints, our expectations ● We expect to have access to our infra • Puppet, monitoring, metrics, repos , jenkins, dnsPuppet, monitoring, metrics, repos , jenkins, dns
  31. 31. VM Provisioning ● Different Technologies • Open vs ProprietaryOpen vs Proprietary • Guess which one is more problematicGuess which one is more problematic ● No access to Internal repositories • (Pulp Mirrors)(Pulp Mirrors) ● Network topologies ● Having to ask to reboot a host ● Having to ask to grow a VM
  32. 32. Security ● IPSec links to all stacks • Our own network complexity has grown exponentiallyOur own network complexity has grown exponentially • Overlapping networksOverlapping networks • Introducing BGP etcIntroducing BGP etc ● Our network = Trusted ● Their network = Hostile • Different approach in host vs network based firewallingDifferent approach in host vs network based firewalling ● User management • Only our accounts in our stack , our ldapOnly our accounts in our stack , our ldap • They want accountsThey want accounts
  33. 33. Variants ● We don’t want exceptions ● They do want exceptions ● Old purchasing mentality • Custom FeaturesCustom Features • Additional ComponentsAdditional Components • It’s “Their” stackIt’s “Their” stack ● Exceptions need to be codified in our infra
  34. 34. Continuous Deployment Delivery ● Deployment isn’t our decision anymore ● Back to fixed deployment windows :( ● Coordination with $customer on when to deploy ● Even for Security Fixes ● For every single instance (except the public SAAS one)
  35. 35. Extreme Cost Difference ● The effort to run 5 stacks in your own infrastructure within your team is smaller than running 1 additional stack on prem at a customer ● Your pragmatic approach does not fit their infrastructure ● You will need to implement features (security/ storage support) that you do not need for your SAAS platform. ● Less movement room to fix stuff . • e.g no “freely” available extra resources disk / memory / cpue.g no “freely” available extra resources disk / memory / cpu • More complex solutions (e.g DBA work..)More complex solutions (e.g DBA work..)
  36. 36. It could have been worse ● We are an Open Source company ● All of our Choices are Open Source by default • We could deploy full stacks On PremWe could deploy full stacks On Prem • Including metrics, log analytics and monitoringIncluding metrics, log analytics and monitoring • We had no external dependenciesWe had no external dependencies • No additional license costsNo additional license costs
  37. 37. If you can choose .. don’t ...don’t ...
  38. 38. ContactContact Kris Buytaert Kris.Buytaert@inuits.euKris Buytaert Kris.Buytaert@inuits.eu Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.eu/http://www.inuits.eu/ Inuits.euInuits.eu Essensteenweg 31Essensteenweg 31 BrasschaatBrasschaat BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×