Docker Meetup San Francisco: Radical Agility with Docker & AWS

1. Radical Agility with Docker & AWS Docker San Fransisco Meetup 2015-10-28 volker.pilz@zalando.de @volkr tech.zalando.com

2. ONE OF EUROPE’S LARGEST ONLINE FASHION RETAILERS ★ 15 countries ★ 3 fulfillment centers ★ 16+ mn active customers ★ 2.2+ bn € ($ ~2.5 bn) revenue 2014 ★ 135+ mn visits per month ★ 9.000+ employees ★ IPO in Oct 2014 ★ ~7 bn € current valuation (Oct 2015)

3. … WITH A STRONG FOCUS ON TECHNOLOGY ★ 800+ in Tech Dept ★ 5 main tech locations in Europe ○ Berlin ○ Dortmund ○ Hamburg ○ Dublin ○ Helsinki Visit us our tech page: tech.zalando.com

4. Mobile first company: > 50% traffic from mobile devices

5. A BRIEF HISTORY OF ZALANDO TECHNOLOGY

6. PLATFORM THE CHALLENGE Platform team request servers deploy Datacenter

7. THE CHALLENGE 90+ delivery teams Platform team deploy request servers request storage PLATFORM Datacenter

8. RADICAL AGILITY

9. DELIVER AMAZING PRODUCTS EFFICIENTLY AT SCALE, AND FEELING GREAT ABOUT IT.

10. PURPOSE AUTONOMY MASTERY Read further if you want: Daniel H. Pink “Drive”

11. ✓ API FIRST ✓ REST ✓ SAAS ✓ MICRO SERVICES ✓ CLOUD DESIGN PRINCIPLES / RULES OF PLAY

12. AN ARCHITECTURE FOR INNOVATION

13. Compliance Innovation

14. STUPS STUPS To Unleash Penguin Swarms

15. ➊ One AWS account per Team ➋ Deployment with Docker ➌ Managed SSH Access ➍ REST/OAuth 2.0 mandatory ➎ Traceability of changes IN A NUTSHELL STUPS

16. AWS STUPS DOCKER DEPLOY SSH ACCESS AUDIT REPORTS FULL AWS ACCESS A PLATFORM ON TOP OF AMAZON WEB SERVICES

17. “We provide maximum freedom for developers while enabling near-real- time audit compliance for every single application.” -- Zalando STUPS Delivery Team

18. Apache License Version 2.0STUPS github.com/zalando-stups

19. STUPS COMPONENT LANDSCAPE

20. OUR FOCUS FOR TODAY ✓ PierOne Docker Registry with S3 storage ✓ Senza Deploy Tool using CloudFormation ✓ Taupage AMI Amazon Machine Image with Docker Runtime http://docs.stups.io/en/latest/user-guide/standalone-deployment.html

21. ACCOUT ISOLATION

22. Public Internet *.jacob.example.org *.edward.example.orgTeam “Jacob” Team “Edward” ELB ELB ISOLATED AWS ACCOUNTS & OAUTH 2.0 & SECURITY Data CenterLB AWS

23. DEPLOYMENT

24. AWS DEPLOYMENT Senza CLI Deploy Tool Pier One Docker Registry docker pull docker push Taupage AMI

25. PIER ONE DOCKER REGISTRY ✓ S3 backend to store images ✓ OAuth2 integration ✓ Team repositories ✓ Immutable tags ✓ JVM-based (Clojure) ✓ Command Line Interface (Python)

26. TAUPAGE AMI ✓ Start a Docker Container on boot ✓ Enables managed SSH access ✓ Audit Logging ✓ Application Logging (LogEntries, Scalyr, CloudWatch Logs) ✓ Monitoring ✓ Reviewed security additions

27. IMMUTABLE STACKS ELB myapp-v1 myapp.example.org EC2 + Docker EC2 + Docker EC2 + Docker 100%

28. IMMUTABLE STACKS ELB myapp-v1 EC2 + Docker EC2 + Docker EC2 + Docker ELB myapp-v2 EC2 + Docker EC2 + Docker myapp.example.org 90% 10% $ senza traffic myapp v2 10

29. IMMUTABLE STACKS ELB myapp-v1 EC2 + Docker EC2 + Docker EC2 + Docker ELB myapp-v2 EC2 + Docker EC2 + Docker myapp.example.org 0% 100% $ senza traffic myapp v2 100

30. FROM zalando/openjdk:8u40-b09-4 EXPOSE 8080 COPY target/hello-world.jar / COPY target/scm-source.json / CMD java $(java-dynamic-memory-opts) ↲ -jar /hello-world.jar DOCKERFILE

31. $ docker build -t ↲ pierone.example.org/myteam/hello-world:0.2 . $ pierone login Getting OAuth2 token "pierone".. OK Storing Docker client configuration in ~/.dockercfg.. OK $ docker push pierone.example.org/myteam/hello-world:0.2 DOCKER BUILD & PUSH

33. SENZA: DEFINITION YAML SenzaInfo: StackName: hello-world Parameters: - ImageVersion: Description: "Docker image version of Hello World." SenzaComponents: - Configuration: Type: Senza::StupsAutoConfiguration # auto-detect network setup - AppServer: # will create a launch configuration and ASG with scaling triggers Type: Senza::TaupageAutoScalingGroup InstanceType: t2.micro SecurityGroups: [app-hello-world] ElasticLoadBalancer: AppLoadBalancer TaupageConfig: runtime: Docker source: "pierone.example.org/stups/hello-world:{{Arguments.ImageVersion}}" ports: 8080: 8080

34. SENZA: STACK DEPLOYMENT $ senza create hello-world.yaml v1 0.2 Generating Cloud Formation template.. OK Creating Cloud Formation stack hello-world-v1.. OK $ senza events hello-world.yaml v1 Stack Name|Ver.|Resource Type |Resource ID |Status |Status Reason |Event Time hello-world v1 CloudFormation::Stack hello-world-v1 CREATE_IN_PROGRESS User Initiated 10m ago ... hello-world v1 CloudFormation::Stack hello-world-v1 CREATE_COMPLETE 6m ago

35. SENZA: MANAGE STACKS

36. MANAGED SSH ACCESS

37. SSH ACCESS: TIME-LIMITED ACCESS TO ANY TEAM SERVER

38. TRACABILITY

39. Pier One Docker Registry build approved EC2 instance Docker Container Application Version “1.0” artifact: docker/myart:1.0 Taupage AMI Ticket system Kio Application Registry SCM Image “docker/myart:1.0” commit: afb123Issue “ABC-123” Commit “afb123” msg: ABC-123.. ✓ Specification ✓ Artefact tested deploy

40. DOCKER? DOCKER!

41. EXAMPLE STACK AWS EC2 Taupage AMI Docker Container Application ✓ In isolated team account ✓ Created by senza through Cloud Formation ✓ Docker Runtime ✓ Managed SSH access ✓ Audit Logging ✓ Log Collection ✓ Monitoring ✓ Reviewed security additions ✓ Ubuntu ✓ OpenJDK ✓ Zalando CA certificate ✓ scm-source …

42. ● Ubuntu & OpenJDK base image https://github.com/zalando/docker-openjdk ● Log to STDOUT ● Config via KMS encrypted env vars ● AMI runs Docker daemon out-of-the-box ● Non-root execution ● Persistence via EBS mounts ● Immutable stacks, no orchestration ● DNS endpoints RECAP: DOCKER IN STUPS

43. docker run -d --log-driver=syslog ↲ --restart=on-failure:10 ↲ -e DB_SUBNAME=.. ↲ -v /meta:/meta:ro ↲ -e CREDENTIALS_DIR=/meta/credentials ↲ -p 8080:8080 -p 7979:7979 ↲ -u 999 ↲ pierone.example.org/stups/pierone:0.5 TAUPAGE: DOCKER COMMAND LINE

44. STUPS Frontpage http://stups.io STUPS Documentation http://docs.stups.io GitHub Repositories https://github.com/zalando-stups Trying out Senza and Taupage http://docs.stups.io/en/latest/user-guide/standalone-deployment.html MORE?

45. CONTACT Volker Pilz volker.pilz@zalando.de linkedin.com/in/vpilz

46. FURTHER INFORMATION stups.io tech-stups-pr@zalando.de tech.zalando.com

47. THANK YOU! Find us here at the meetup ➔ Rani ➔ Lauri ➔ Ako ➔ Volker

48. BACKUP

49. MONITORING

50. TODO: Screenshot ZMON

51. ZMON APPLIANCE *.foo.example.org *.bar.example.org Team “Foo” Team “Bar” EC2 Instance EC2 InstanceEC2 Instance EC2 Instance ZMON Appliance ZMON Appliance KairosDB EC2 Instance EC2 Instance ZMON Controller ELB ELB

52. LOGGING

53. docker run .. --log-driver=syslog .. /etc/rsyslog.d/24-application.conf :syslogtag, startswith, "docker" ↲ /var/log/application.log /etc/logrotate.d/.. Don’t forget log rotation.. TAUPAGE: DOCKER SYSLOG

54. APPLICATION LOGS: TAUPAGE SUPPORTS LOGENTRIES AND SCALYR

55. STUPS COMPONENT NAMING

56. asciinema.org/a/25668 DEPLOYMENT