Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Docker Meetup San Francisco: Radical Agility with Docker & AWS

6,372 views

Published on

This slidedeck is about Zalando's open-source PaaS framework STUPS (stups.io), which was built in-house to enable multiple teams to use the full power of AWS without scarifying vital aspects like security, trace-ability and architectural standards. Docker plays a key role in this setup and helps us to realize an easy and robust deployment process.

The corresponding meetup took place at Oct 28 2015 at Microsoft Reactor Space in San Francisco organized by Docker.

Published in: Technology

Docker Meetup San Francisco: Radical Agility with Docker & AWS

  1. 1. Radical Agility with Docker & AWS Docker San Fransisco Meetup 2015-10-28 volker.pilz@zalando.de @volkr tech.zalando.com
  2. 2. ONE OF EUROPE’S LARGEST ONLINE FASHION RETAILERS ★ 15 countries ★ 3 fulfillment centers ★ 16+ mn active customers ★ 2.2+ bn € ($ ~2.5 bn) revenue 2014 ★ 135+ mn visits per month ★ 9.000+ employees ★ IPO in Oct 2014 ★ ~7 bn € current valuation (Oct 2015)
  3. 3. … WITH A STRONG FOCUS ON TECHNOLOGY ★ 800+ in Tech Dept ★ 5 main tech locations in Europe ○ Berlin ○ Dortmund ○ Hamburg ○ Dublin ○ Helsinki Visit us our tech page: tech.zalando.com
  4. 4. Mobile first company: > 50% traffic from mobile devices
  5. 5. A BRIEF HISTORY OF ZALANDO TECHNOLOGY
  6. 6. PLATFORM THE CHALLENGE Platform team request servers deploy Datacenter
  7. 7. THE CHALLENGE 90+ delivery teams Platform team deploy request servers request storage PLATFORM Datacenter
  8. 8. RADICAL AGILITY
  9. 9. DELIVER AMAZING PRODUCTS EFFICIENTLY AT SCALE, AND FEELING GREAT ABOUT IT.
  10. 10. PURPOSE AUTONOMY MASTERY Read further if you want: Daniel H. Pink “Drive”
  11. 11. ✓ API FIRST ✓ REST ✓ SAAS ✓ MICRO SERVICES ✓ CLOUD DESIGN PRINCIPLES / RULES OF PLAY
  12. 12. AN ARCHITECTURE FOR INNOVATION
  13. 13. Compliance Innovation
  14. 14. STUPS STUPS To Unleash Penguin Swarms
  15. 15. ➊ One AWS account per Team ➋ Deployment with Docker ➌ Managed SSH Access ➍ REST/OAuth 2.0 mandatory ➎ Traceability of changes IN A NUTSHELL STUPS
  16. 16. AWS STUPS DOCKER DEPLOY SSH ACCESS AUDIT REPORTS FULL AWS ACCESS A PLATFORM ON TOP OF AMAZON WEB SERVICES
  17. 17. “We provide maximum freedom for developers while enabling near-real- time audit compliance for every single application.” -- Zalando STUPS Delivery Team
  18. 18. Apache License Version 2.0STUPS github.com/zalando-stups
  19. 19. STUPS COMPONENT LANDSCAPE
  20. 20. OUR FOCUS FOR TODAY ✓ PierOne Docker Registry with S3 storage ✓ Senza Deploy Tool using CloudFormation ✓ Taupage AMI Amazon Machine Image with Docker Runtime http://docs.stups.io/en/latest/user-guide/standalone-deployment.html
  21. 21. ACCOUT ISOLATION
  22. 22. Public Internet *.jacob.example.org *.edward.example.orgTeam “Jacob” Team “Edward” ELB ELB ISOLATED AWS ACCOUNTS & OAUTH 2.0 & SECURITY Data CenterLB AWS
  23. 23. DEPLOYMENT
  24. 24. AWS DEPLOYMENT Senza CLI Deploy Tool Pier One Docker Registry docker pull docker push Taupage AMI
  25. 25. PIER ONE DOCKER REGISTRY ✓ S3 backend to store images ✓ OAuth2 integration ✓ Team repositories ✓ Immutable tags ✓ JVM-based (Clojure) ✓ Command Line Interface (Python)
  26. 26. TAUPAGE AMI ✓ Start a Docker Container on boot ✓ Enables managed SSH access ✓ Audit Logging ✓ Application Logging (LogEntries, Scalyr, CloudWatch Logs) ✓ Monitoring ✓ Reviewed security additions
  27. 27. IMMUTABLE STACKS ELB myapp-v1 myapp.example.org EC2 + Docker EC2 + Docker EC2 + Docker 100%
  28. 28. IMMUTABLE STACKS ELB myapp-v1 EC2 + Docker EC2 + Docker EC2 + Docker ELB myapp-v2 EC2 + Docker EC2 + Docker myapp.example.org 90% 10% $ senza traffic myapp v2 10
  29. 29. IMMUTABLE STACKS ELB myapp-v1 EC2 + Docker EC2 + Docker EC2 + Docker ELB myapp-v2 EC2 + Docker EC2 + Docker myapp.example.org 0% 100% $ senza traffic myapp v2 100
  30. 30. FROM zalando/openjdk:8u40-b09-4 EXPOSE 8080 COPY target/hello-world.jar / COPY target/scm-source.json / CMD java $(java-dynamic-memory-opts) ↲ -jar /hello-world.jar DOCKERFILE
  31. 31. $ docker build -t ↲ pierone.example.org/myteam/hello-world:0.2 . $ pierone login Getting OAuth2 token "pierone".. OK Storing Docker client configuration in ~/.dockercfg.. OK $ docker push pierone.example.org/myteam/hello-world:0.2 DOCKER BUILD & PUSH
  32. 32. $ pierone tags myteam hello-world Team |Artifact |Tag |Created|By | myteam hello-world 0.1-andre-test 13d ago ahartmann myteam hello-world 0.1 3d ago ahartmann myteam hello-world 0.2 3m ago hjacobs VERIFY IMAGE UPLOAD $ pierone scm myteam hello-world 0.2 Tag|Author |URL |Revision |Status|Created|By | 0.2 hjacobs git:git@github.. 442b7502 10m ago hjacobs
  33. 33. SENZA: DEFINITION YAML SenzaInfo: StackName: hello-world Parameters: - ImageVersion: Description: "Docker image version of Hello World." SenzaComponents: - Configuration: Type: Senza::StupsAutoConfiguration # auto-detect network setup - AppServer: # will create a launch configuration and ASG with scaling triggers Type: Senza::TaupageAutoScalingGroup InstanceType: t2.micro SecurityGroups: [app-hello-world] ElasticLoadBalancer: AppLoadBalancer TaupageConfig: runtime: Docker source: "pierone.example.org/stups/hello-world:{{Arguments.ImageVersion}}" ports: 8080: 8080
  34. 34. SENZA: STACK DEPLOYMENT $ senza create hello-world.yaml v1 0.2 Generating Cloud Formation template.. OK Creating Cloud Formation stack hello-world-v1.. OK $ senza events hello-world.yaml v1 Stack Name|Ver.|Resource Type |Resource ID |Status |Status Reason |Event Time hello-world v1 CloudFormation::Stack hello-world-v1 CREATE_IN_PROGRESS User Initiated 10m ago ... hello-world v1 CloudFormation::Stack hello-world-v1 CREATE_COMPLETE 6m ago
  35. 35. SENZA: MANAGE STACKS
  36. 36. MANAGED SSH ACCESS
  37. 37. SSH ACCESS: TIME-LIMITED ACCESS TO ANY TEAM SERVER
  38. 38. TRACABILITY
  39. 39. Pier One Docker Registry build approved EC2 instance Docker Container Application Version “1.0” artifact: docker/myart:1.0 Taupage AMI Ticket system Kio Application Registry SCM Image “docker/myart:1.0” commit: afb123Issue “ABC-123” Commit “afb123” msg: ABC-123.. ✓ Specification ✓ Artefact tested deploy
  40. 40. DOCKER? DOCKER!
  41. 41. EXAMPLE STACK AWS EC2 Taupage AMI Docker Container Application ✓ In isolated team account ✓ Created by senza through Cloud Formation ✓ Docker Runtime ✓ Managed SSH access ✓ Audit Logging ✓ Log Collection ✓ Monitoring ✓ Reviewed security additions ✓ Ubuntu ✓ OpenJDK ✓ Zalando CA certificate ✓ scm-source …
  42. 42. ● Ubuntu & OpenJDK base image https://github.com/zalando/docker-openjdk ● Log to STDOUT ● Config via KMS encrypted env vars ● AMI runs Docker daemon out-of-the-box ● Non-root execution ● Persistence via EBS mounts ● Immutable stacks, no orchestration ● DNS endpoints RECAP: DOCKER IN STUPS
  43. 43. docker run -d --log-driver=syslog ↲ --restart=on-failure:10 ↲ -e DB_SUBNAME=.. ↲ -v /meta:/meta:ro ↲ -e CREDENTIALS_DIR=/meta/credentials ↲ -p 8080:8080 -p 7979:7979 ↲ -u 999 ↲ pierone.example.org/stups/pierone:0.5 TAUPAGE: DOCKER COMMAND LINE
  44. 44. STUPS Frontpage http://stups.io STUPS Documentation http://docs.stups.io GitHub Repositories https://github.com/zalando-stups Trying out Senza and Taupage http://docs.stups.io/en/latest/user-guide/standalone-deployment.html MORE?
  45. 45. CONTACT Volker Pilz volker.pilz@zalando.de linkedin.com/in/vpilz
  46. 46. FURTHER INFORMATION stups.io tech-stups-pr@zalando.de tech.zalando.com
  47. 47. THANK YOU! Find us here at the meetup ➔ Rani ➔ Lauri ➔ Ako ➔ Volker
  48. 48. BACKUP
  49. 49. MONITORING
  50. 50. TODO: Screenshot ZMON
  51. 51. ZMON APPLIANCE *.foo.example.org *.bar.example.org Team “Foo” Team “Bar” EC2 Instance EC2 InstanceEC2 Instance EC2 Instance ZMON Appliance ZMON Appliance KairosDB EC2 Instance EC2 Instance ZMON Controller ELB ELB
  52. 52. LOGGING
  53. 53. docker run .. --log-driver=syslog .. /etc/rsyslog.d/24-application.conf :syslogtag, startswith, "docker" ↲ /var/log/application.log /etc/logrotate.d/.. Don’t forget log rotation.. TAUPAGE: DOCKER SYSLOG
  54. 54. APPLICATION LOGS: TAUPAGE SUPPORTS LOGENTRIES AND SCALYR
  55. 55. STUPS COMPONENT NAMING
  56. 56. asciinema.org/a/25668 DEPLOYMENT

×