Download to read offline
![LB tricks
error_page 500 = /errors/internal-server-error;
location ~ ^/errors/(?<code>[a-z-])+ {
proxy_pass http://error.uken.com:80/titans/$code;
}
Reload away.
No more redirects!](https://image.slidesharecdn.com/sheepit-150421100133-conversion-gate02/85/Sheep-it-46-320.jpg)
![[OpenInfra Days Korea 2018] Day 2 - E5-1: "Invited Talk: Kubicorn - Building ...](https://cdn.slidesharecdn.com/ss_thumbnails/e60955buildingsimplekubernetes-180705043459-thumbnail.jpg?width=640&height=640&fit=bounds)
![[HKOSCON][20180616][Containerized High Availability Virtual Hosting Deploymen...](https://cdn.slidesharecdn.com/ss_thumbnails/hkoscon20180616containerizedhighavailabilityvirtualhostingdeploymentwithkubernetesdockerandansible-180616144342-thumbnail.jpg?width=640&height=640&fit=bounds)
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Sheep it
- 1. Deploying with Consul +Docker
- 2.
- 3.
- 4. Hello! I am LucasFontes Services Team Lead at Uken Studios I take screenshots Spirit Animal: Left Shark internets: @lxfontes
- 5. Uken Studios Rails shop HTML5and Unity3d ~ 150 servers (300 peak) - ~ 40 Databases - ~ 70 App servers - SOA with nsq.io
- 7. Static server list Lock-stepcommands on all servers HAProxy config via Chef Snowflake servers. (DB operations/scheduled tasks) ~15 clusters Shared HAProxy cluster
- 8. Bare Metal DBs (cpuspeed / memory / ssd) Web (cpu count / nic bonding) Tons of unused network transfer Monthly billing upgrades idle hardware (redundancy)
- 9. PowerUP ~ 2013 Build- Heroku buildpacks lxfontes/slugbuilder Orchestration - Redis / Rails (private) Run - Docker/lxc hosts lxfontes/slugrunner-rb Dynamic LB - knuckles uken/knuckles
- 11.
- 12. ~4 - 10hours provisioning time Virtual Servers meh Softlayer -> AWS
- 13. Switch - 4hours cut-over
- 14.
- 15. What if auto-scalingkicks in mid-deploy? What if instance dies mid-deploy? Trust chef nodes will register themselves! Static server list Lock-step commands on all servers HAProxy config via Chef Snowflake servers. (DB operations/scheduled tasks)
- 16. HAProxy + Chef MakeSensu remove dead nodes move on.
- 17. PowerUP much BuildPacks such Scaling wowDynamic LB (knuckles) Poor adoption Manual Database migrations No love for buildpacks No more idle hardware
- 19. Ops Requirements - Nottied to AWS - Containers / packs - Either straight LXC, docker, rkt - Dynamic Load Balancing - ELB only for SSL termination - Easy to troubleshoot - Heavy logging, unique-ids everywhere - Configuration Decoupling - Not tied to chef/ansible/puppet - DB location, worker count - Deploy anything - Not just rails
- 20.
- 21. Dev Requirements - Runpre/post deploy scripts - slack notifications - close jira tickets - CDN friendly - GitHub - 'Can haz ephemeral shell?' - run rake tasks manually - ‘Can haz production shell?’ - troubleshooting - One click deploy
- 22. Out there or… “whyyou should not do it yourself”
- 23. Out there - Clusters -deis, kubernetes, shipbuilder, coreos, mesos - Config management - chef, ansible - Deploy tools - capistrano, mina, fabric, packer - Services - ECS, CodeDeploy, Elastic Beanstalk - shippable, ezautoscaling (!?)
- 24. They all work! Notthe way we want!
- 25.
- 26. BUILD How? Dockerfile oneach repo docker build and done Store? Docker Registry Target Image? repo + prod|staging registry.uken.int/uken/titans_prod Target Version? Git revision registry.uken.int/uken/titans_prod:cc3e68a
- 27. RUN Sealed container only docker--rm=true Pre release Database Migrations / Seeds CDN Asset upload Post release Ephemeral shell sessions
- 28. RELEASE Upload image todocker registry Notify hosts registry.uken.int/uken/titans_prod:cc3e68a Go download this version! Docker pull + restart on hosts
- 29. ‘’ But Lucas… really... Howhosts reload ? Load balancing ? This looks complicated
- 30. Distributed Quorum Agents onall hosts / load balancers Key Value Pairs titans_prod/version: registry.uken.int/uken/titans_prod:cc3e68a titans_prod/env/DB_HOST: db1.uken.int DNS + HTTP API kinda important Service Registration + Health Checks web1.node.consul => 10.28.3.1 titans_prod.service.consul => 10.28.3.1 port 5001 Consul
- 31.
- 32.
- 33. ‘’ But Lucas… How hostsreload ? Load balancing ? This looks complicated
- 34. Use Consul DNSAPI I said it was important Lookup by repo + environment titans_prod.service.consul srv1.node.consul port 5100 srv2.node.consul port 4200 srv3.node.consul port 9000 proxy_pass http://$target; gist
- 35. ‘’ But Lucas… How hostsreload ? Load balancing ? This looks complicated
- 36. Not really Build Dockerfile docker build Run/ shells docker pull docker run Release docker push consul set kv pair Frontend Provisioning ec2 snapshots Load Balancers ec2 snapshots
- 37.
- 38.
- 39. Consul + DNS Runa local cache + consul chain (unbound) stub-zone: name: "consul" stub-addr: 127.0.0.1@8600 CNAME databases / caches on alt domain
- 40. Docker Filesystem Drivers overlay /aufs / btrfs / devicemapper STDOUT logging Bad for long lived containers - fixed in 1.6! Container linking Meh. --net=host Need volumes during build? docker run && docker commit
- 41. Dockerfiles: Base Image ubuntu uken/base uken/rubyuken/djangouken/go shim
- 42. Dockerfiles: Passing parameters #Inject SSH key ADD http://127.0.0.1:9090/ssh_key /root/.ssh/id_rsa RUN chmod 600 /root/.ssh/id_rsa # Revision + deploy vars ADD http://127.0.0.1:9090/env /etc/profile.d/custom_env.sh No ‘official’ way HTTP trick
- 43. Dockerfiles: Caching FROM uken/ruby_20:latest ENVRAILS_ENV design ENV WORKERS 4 ADD ./Gemfile /app/Gemfile ADD ./Gemfile.lock /app/Gemfile.lock RUN cd /app && bundle install --jobs 4 --deployment ADD ./ /app RUN cd /app && bundle exec rake assets:precompile
- 44. Procfiles Foreman on desktop ddollar/foreman Foregoon servers ddollar/forego web: bundle exec unicorn -c config/unicorn.rb -p ${PORT:-3000} -E ${RAILS_ENV:- development} realtime: ./realtime/realtime_server.linux -ws $PORT sidekiq: bundle exec sidekiq -v -c 3 cron: cron -f
- 45. LB tricks HTTP/1.1 200OK ... X-U-Fe: 10.1.0.248:5003 add_header X-U-Fe $target; proxy_pass http://$target;
- 46. LB tricks error_page 500= /errors/internal-server-error; location ~ ^/errors/(?<code>[a-z-])+ { proxy_pass http://error.uken.com:80/titans/$code; } Reload away. No more redirects!
- 47.