Radical Agility with Docker & AWS
•
15 likes•7,223 views
This document describes how Zalando, one of Europe's largest online fashion retailers, uses Docker and AWS to enable radical agility in deploying applications. It discusses how Zalando developed STUPS (STUPS To Unleash Penguin Swarms), an internal platform built on AWS that provides developers maximum freedom while enabling near-real-time audit compliance. STUPS utilizes Docker, PierOne Docker registry, Senza deploy tool, and Taupage AMIs to allow teams to deploy applications in isolated AWS accounts with full AWS access, immutable infrastructure, and managed SSH access for compliance and security.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to Radical Agility with Docker & AWS
Similar to Radical Agility with Docker & AWS (20)
Recently uploaded
Recently uploaded (20)
Radical Agility with Docker & AWS
- 1. Radical Agility with Docker & AWS Docker San Fransisco Meetup 2015-10-28 volker.pilz@zalando.de @volkr tech.zalando.com
- 2. ONE OF EUROPE’S LARGEST ONLINE FASHION RETAILERS ★ 15 countries ★ 3 fulfillment centers ★ 16+ mn active customers ★ 2.2+ bn € ($ ~2.5 bn) revenue 2014 ★ 135+ mn visits per month ★ 9.000+ employees ★ IPO in Oct 2014 ★ ~7 bn € current valuation (Oct 2015)
- 3. … WITH A STRONG FOCUS ON TECHNOLOGY ★ 800+ in Tech Dept ★ 5 main tech locations in Europe ○ Berlin ○ Dortmund ○ Hamburg ○ Dublin ○ Helsinki Visit us our tech page: tech.zalando.com
- 4. Mobile first company: > 50% traffic from mobile devices
- 8. PLATFORM THE CHALLENGE Platform team request servers deploy Datacenter
- 9. THE CHALLENGE 90+ delivery teams Platform team deploy request servers request storage PLATFORM Datacenter
- 10. RADICAL AGILITY
- 11. DELIVER AMAZING PRODUCTS EFFICIENTLY AT SCALE, AND FEELING GREAT ABOUT IT.
- 12. PURPOSE AUTONOMY MASTERY Read further if you want: Daniel H. Pink “Drive”
- 13. ✓ API FIRST ✓ REST ✓ SAAS ✓ MICRO SERVICES ✓ CLOUD DESIGN PRINCIPLES / RULES OF PLAY
- 16. STUPS STUPS To Unleash Penguin Swarms
- 17. ➊ One AWS account per Team ➋ Deployment with Docker ➌ Managed SSH Access ➍ REST/OAuth 2.0 mandatory ➎ Traceability of changes IN A NUTSHELL STUPS
- 18. AWS STUPS DOCKER DEPLOY SSH ACCESS AUDIT REPORTS FULL AWS ACCESS A PLATFORM ON TOP OF AMAZON WEB SERVICES
- 19. “We provide maximum freedom for developers while enabling near-real- time audit compliance for every single application.” -- Zalando STUPS Delivery Team
- 22. OUR FOCUS FOR TODAY ✓ PierOne Docker Registry with S3 storage ✓ Senza Deploy Tool using CloudFormation ✓ Taupage AMI Amazon Machine Image with Docker Runtime http://docs.stups.io/en/latest/user-guide/standalone-deployment.html
- 23. ACCOUT ISOLATION
- 24. Public Internet *.jacob.example.org *.edward.example.orgTeam “Jacob” Team “Edward” ELB ELB ISOLATED AWS ACCOUNTS & OAUTH 2.0 & SECURITY Data CenterLB AWS
- 25. DEPLOYMENT
- 26. AWS DEPLOYMENT Senza CLI Deploy Tool Pier One Docker Registry docker pull docker push Taupage AMI
- 27. PIER ONE DOCKER REGISTRY ✓ S3 backend to store images ✓ OAuth2 integration ✓ Team repositories ✓ Immutable tags ✓ JVM-based (Clojure) ✓ Command Line Interface (Python)
- 28. TAUPAGE AMI ✓ Start a Docker Container on boot ✓ Enables managed SSH access ✓ Audit Logging ✓ Application Logging (LogEntries, Scalyr, CloudWatch Logs) ✓ Monitoring ✓ Reviewed security additions
- 29. IMMUTABLE STACKS ELB myapp-v1 myapp.example.org EC2 + Docker EC2 + Docker EC2 + Docker 100%
- 30. IMMUTABLE STACKS ELB myapp-v1 EC2 + Docker EC2 + Docker EC2 + Docker ELB myapp-v2 EC2 + Docker EC2 + Docker myapp.example.org 90% 10% $ senza traffic myapp v2 10
- 31. IMMUTABLE STACKS ELB myapp-v1 EC2 + Docker EC2 + Docker EC2 + Docker ELB myapp-v2 EC2 + Docker EC2 + Docker myapp.example.org 0% 100% $ senza traffic myapp v2 100
- 32. FROM zalando/openjdk:8u40-b09-4 EXPOSE 8080 COPY target/hello-world.jar / COPY target/scm-source.json / CMD java $(java-dynamic-memory-opts) ↲ -jar /hello-world.jar DOCKERFILE
- 33. $ docker build -t ↲ pierone.example.org/myteam/hello-world:0.2 . $ pierone login Getting OAuth2 token "pierone".. OK Storing Docker client configuration in ~/.dockercfg.. OK $ docker push pierone.example.org/myteam/hello-world:0.2 DOCKER BUILD & PUSH
- 34. $ pierone tags myteam hello-world Team |Artifact |Tag |Created|By | myteam hello-world 0.1-andre-test 13d ago ahartmann myteam hello-world 0.1 3d ago ahartmann myteam hello-world 0.2 3m ago hjacobs VERIFY IMAGE UPLOAD $ pierone scm myteam hello-world 0.2 Tag|Author |URL |Revision |Status|Created|By | 0.2 hjacobs git:git@github.. 442b7502 10m ago hjacobs
- 35. SENZA: DEFINITION YAML SenzaInfo: StackName: hello-world Parameters: - ImageVersion: Description: "Docker image version of Hello World." SenzaComponents: - Configuration: Type: Senza::StupsAutoConfiguration # auto-detect network setup - AppServer: # will create a launch configuration and ASG with scaling triggers Type: Senza::TaupageAutoScalingGroup InstanceType: t2.micro SecurityGroups: [app-hello-world] ElasticLoadBalancer: AppLoadBalancer TaupageConfig: runtime: Docker source: "pierone.example.org/stups/hello-world:{{Arguments.ImageVersion}}" ports: 8080: 8080
- 36. SENZA: STACK DEPLOYMENT $ senza create hello-world.yaml v1 0.2 Generating Cloud Formation template.. OK Creating Cloud Formation stack hello-world-v1.. OK $ senza events hello-world.yaml v1 Stack Name|Ver.|Resource Type |Resource ID |Status |Status Reason |Event Time hello-world v1 CloudFormation::Stack hello-world-v1 CREATE_IN_PROGRESS User Initiated 10m ago ... hello-world v1 CloudFormation::Stack hello-world-v1 CREATE_COMPLETE 6m ago
- 39. SSH ACCESS: TIME-LIMITED ACCESS TO ANY TEAM SERVER
- 40. TRACABILITY
- 41. Pier One Docker Registry build approved EC2 instance Docker Container Application Version “1.0” artifact: docker/myart:1.0 Taupage AMI Ticket system Kio Application Registry SCM Image “docker/myart:1.0” commit: afb123Issue “ABC-123” Commit “afb123” msg: ABC-123.. ✓ Specification ✓ Artefact tested deploy
- 42. DOCKER? DOCKER!
- 43. EXAMPLE STACK AWS EC2 Taupage AMI Docker Container Application ✓ In isolated team account ✓ Created by senza through Cloud Formation ✓ Docker Runtime ✓ Managed SSH access ✓ Audit Logging ✓ Log Collection ✓ Monitoring ✓ Reviewed security additions ✓ Ubuntu ✓ OpenJDK ✓ Zalando CA certificate ✓ scm-source …
- 44. ● Ubuntu & OpenJDK base image https://github.com/zalando/docker-openjdk ● Log to STDOUT ● Config via KMS encrypted env vars ● AMI runs Docker daemon out-of-the-box ● Non-root execution ● Persistence via EBS mounts ● Immutable stacks, no orchestration ● DNS endpoints RECAP: DOCKER IN STUPS
- 45. docker run -d --log-driver=syslog ↲ --restart=on-failure:10 ↲ -e DB_SUBNAME=.. ↲ -v /meta:/meta:ro ↲ -e CREDENTIALS_DIR=/meta/credentials ↲ -p 8080:8080 -p 7979:7979 ↲ -u 999 ↲ pierone.example.org/stups/pierone:0.5 TAUPAGE: DOCKER COMMAND LINE
- 46. STUPS Frontpage http://stups.io STUPS Documentation http://docs.stups.io GitHub Repositories https://github.com/zalando-stups Trying out Senza and Taupage http://docs.stups.io/en/latest/user-guide/standalone-deployment.html MORE?
- 49. THANK YOU! Find us here at the meetup ➔ Rani ➔ Lauri ➔ Ako ➔ Volker
- 50. BACKUP
- 51. MONITORING
- 53. ZMON APPLIANCE *.foo.example.org *.bar.example.org Team “Foo” Team “Bar” EC2 Instance EC2 InstanceEC2 Instance EC2 Instance ZMON Appliance ZMON Appliance KairosDB EC2 Instance EC2 Instance ZMON Controller ELB ELB
- 54. LOGGING
- 55. docker run .. --log-driver=syslog .. /etc/rsyslog.d/24-application.conf :syslogtag, startswith, "docker" ↲ /var/log/application.log /etc/logrotate.d/.. Don’t forget log rotation.. TAUPAGE: DOCKER SYSLOG
- 56. APPLICATION LOGS: TAUPAGE SUPPORTS LOGENTRIES AND SCALYR