Chicago Docker Meetup Presentation - Mediafly

1,921 views
1,856 views

Published on

Bryan Murphy's presentation from the 2nd Chicago Docker meetup on March 12, 2014 at Mediafly HQ. In his presentation, Bryan explains how we use Docker right now at Mediafly in production.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,921
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
17
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Chicago Docker Meetup Presentation - Mediafly

  1. 1. Docker  at  Mediafly Bryan  Murphy 2014.03.12 1  /  34
  2. 2. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? Bryan  Murphy Mediafly,  Inc. Senior  Technical  Architect  (7  Years) Specialize  in  back-­end  services,  video  processing, scaling  and  architecture. Mobitrac,  Inc. Senior  Developer  (2  Years) Specialized  in  travelling  salesman  problem,  routing  algorithms,  and  visualization for  logistics  software. RBC/Centura  Mortgage Lead  Web  Developer  (5  years) Created  and  maintained  online  customizable  loan  officer  hosting  platform  and  rate search  engine. HCST,  Inc.  (ISP  in  Dayton,  OH) Developer/Administrator  (3  years) Web  and  systems  development,  server  and  network  administration. 2  /  34
  3. 3. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? What  do  we  do? Content  Management  System 3  /  34
  4. 4. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? What  do  we  do? Secure  Content  Delivery  Platform 4  /  34
  5. 5. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? What  do  we  do? Document  Conversion DOC,  PDF,  PPT,  XLS Video  Processing AVI,  F4M,  F4V,  M3U8,  MKV,  MOV,  MP4,  MXF,  WMV Customizable  UI Cross  device  HTML5+JavaScript  UI. Secure  Delivery Encryption,  DRM,  Real-­Time  Watermarking 5  /  34
  6. 6. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? What  makes  us  interesting? Multi-­Device Multi-­Tenant Major  customers  are  deployed  to  their  own  isolated  environment. Service  Oriented Multiple  services  coordinated  using  messaging  and  http  APIs. Distributed Asynchronous  message  driven  achitecture  using  RabbitMQ. 6  /  34
  7. 7. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? A  Sampling  the  Technologies  Use Programming  Languages Bash,  C#,  Java,  Python,  Ruby,  PHP,  BrightScript,  Objective-­C Platforms Android,  Flash,  iOS,  JDK6,  JDK7,  Mono,  .NET,  Python,  Roku,  Ruby Application  Servers Flask,  IIS,  Tomcat,  Rails Operating  Systems Ubuntu  10.04  /  12.04  /  14.04,  CentOS,  Windows  2008,  Windows  2012 Software Apache,  Nginx,  IIS,  Memcached,  RabbitMQ,  PostgreSQL,  Salt,  Puppet,  FFmpeg, SoX And  more! 7  /  34
  8. 8. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? Some  Numbers Millions  of  API  requests/day Thousands  of  cloud  servers  provisioned/day Thousands  of  video  transcode  jobs/day Thousands  of  document  processing  jobs/day 100's  of  Millions  of  files  stored  on  S3 Hundreds  of  Terabytes  of  online  video  storage 8  /  34
  9. 9. Mediafly Who  am  I? What  do  we  do? What  makes  us interesting? Multitenant  Secure  Distributed  Asynchronous Service  Oriented  Polyglot  Architecture  Under  Load What  have  we  gotten  ourselves  into?? 9  /  34
  10. 10. Enter  the  Docker 10  /  34
  11. 11. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Developer  Friendly Simple  Syntax Mediafly  Monitoring  Web  Site: FROM mediafly-ubuntu-12.04-base-flask ADD ./src/web.py /srv/web.py EXPOSE 8888 WORKDIR /srv CMD gunicorn web:app --workers 8 --bind 0.0.0.0:8888 --log-level=DEBUG Source  Control Branch?  Merge?  No  problem! $ hg commit -m "updated database cluster configuration" Dockerfile Infrastructure  as  Code RUN add-apt-repository ppa:nginx/development RUN apt-get update RUN apt-get install -y nginx-extras==1.1.19-1ubuntu0.6 11  /  34
  12. 12. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Iterate Fast  Start  Time $ time docker run -i -t ubuntu:12.04 date real 0m0.177s user 0m0.000s sys 0m0.036s Incremental  Builds $ docker build -t elasticsearch . ...snip... Step 9 : RUN chown -R root.root /opt/elasticsearch-1.0.0 ---> Using cache ---> 8bb1225ea753 Successfully built 8bb1225ea753 real 0m0.132s user 0m0.020s sys 0m0.024s Consistent  Environment  (Repeatability) $ docker run -i -t ubuntu:12.04 touch /tmp/file.txt $ echo $? 0 $ docker run -i -t ubuntu:12.04 test -f /tmp/file.txt $ echo $? 1 Faster  Development Don't  waste  time  waiting! 12  /  34
  13. 13. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Testing Test  Your  Code  in  Context $ docker run -i -t appserver python unittests.py $ echo $? 0 Test  Your  Infrastructure $ docker run -i -t appserver test -e /usr/sbin/nginx $ echo $? 0 Test  Your  Configuration $ docker run -i -t appserver grep -q "root /var/lib/appserver;" /etc/nginx/conf.d/appserver.conf $ echo $? 0 Deploy  What  Was  Tested $ export VERSION=r2014.03.12b01 $ docker build -t appserver . && docker run -i -t appserver python unittests.py && docker run -i -t appserver test -e /usr/sbin/nginx && docker run -i -t appserver grep -q "root /var/lib/appserver;" /etc/nginx/conf.d/appserver.conf && docker tag appserver registry:443/appserver:$VERSION && docker push registry:443/appserver:$VERSION $ echo $? 0 13  /  34
  14. 14. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Dependency  Management Encapsulation Everything  you  need  in  a  single  package. FROM ubuntu:12.04 RUN apt-get update RUN apt-get install libapache2-mod-php5 mysql-server supervisor ADD ./conf /etc/supervisor/conf.d CMD supervisord -n $ docker build -t lampstack . $ docker run -d lampstack 14698a1366c $ lxc-attach -n 14698a1366c9 /bin/ps -- x PID TTY STAT TIME COMMAND 1 ? S 0:00 /bin/sh -c supervisord -n 7 ? S 0:00 /usr/bin/python /usr/bin/supervisord -n 8 ? S 0:00 /bin/sh /usr/sbin/apachectl -k start -X 14 ? S 0:00 /usr/bin/mysqld 38 ? R+ 0:00 /bin/ps -x Mutual  Exclusion Run  multiple  versions  of  the  same  program. $ lsb_release -r Release: 12.04 $ docker run -i -t s3cmd-v1.1.0-beta2 s3cmd --version s3cmd version 1.1.0-beta2 $ docker run -i -t s3cmd-v1.5.0-alpha3 s3cmd --version s3cmd version 1.5.0-alpha3 14  /  34
  15. 15. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Sharing Jane  Doe's  Development  Environment Dockerfile: FROM ubuntu:12.04 RUN apt-get update RUN apt-get install -y mono-complete RUN apt-get install -y openjdk-6-jdk RUN apt-get install -y python-flask RUN apt-get install -y postgresql RUN apt-get install -y rake etc. Sharing: $ docker build -t registry:443/dev . $ docker push registry:443/dev or: $ hg commit -m "updated dev environment" dev/Dockerfile $ hg push Jim  Smith's  Development  Environment Build  and  Test  Application: $ docker pull registry:443/dev $ docker run -i -t -v `pwd`:/srv -w /srv registry:443/dev rake build $ docker run -i -t -v `pwd`:/srv -w /srv registry:443/dev rake test 15  /  34
  16. 16. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Standardized Apache $ docker run -i -t -p 80 -p 443 apache Nginx $ docker run -i -t -p 80 -p 443 nginx Postgres $ docker run -i -t -p 5432 postgres ElasticSearch  +  Logstash  +  Redis  +  Kibana $ docker run -i -t -p 5544 -p 8080 -p 9200 elasticsearch What  does  this  mean? Standardized  execution  environment. Standardized  port  mapping. Standardized  volume  management. Standardized  monitoring. 16  /  34
  17. 17. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Isolation Isolate  Services $ docker run -d -p 80 -p 443 nginx f50c4854b7d8 $ lxc-attach -n f50c4854b7d8 /bin/ps -- ax PID TTY STAT TIME COMMAND 1 ? S 0:00 nginx: master process nginx 7 ? S 0:00 nginx: worker process 8 ? S 0:00 nginx: worker process 9 ? S 0:00 nginx: worker process 10 ? S 0:00 nginx: worker process 21 ? R+ 0:00 /bin/ps ax Minimize  Exposed  Surface  Area mediafly-ubuntu-12.04-base 394mb mediafly-ubuntu-12.04-base-extras 882mb mediafly-ubuntu-12.04-base-extras-mono-2.10 1,074mb mediafly-ubuntu-12.04-base-extras-mono-2.10-java6 1,369mb mediafly-ubuntu-12.04-base-flask 403mb mediafly-ubuntu-12.04-base-java6 1,172mb mediafly-ubuntu-12.04-base-nginx 410mb etc. 17  /  34
  18. 18. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Right  Sized  Deployments Only  build  what  changed $ docker history mediafly-monitor IMAGE CREATED SIZE 397b4009c926 5 seconds ago 0 B 39afabb5721c 6 seconds ago 0 B 5f470d30018a 6 seconds ago 587.5 kB fd05e65cd81d 12 days ago 0 B 0c9b69626be3 12 days ago 9.414 MB d3d6f1d5b7b0 12 days ago 0 B d29aa2f4f869 12 days ago 0 B 960ee306f74e 12 days ago 2.226 MB c10f048ae9f7 12 days ago 33.14 MB 57f2e95ecead 12 days ago 213 B 3db83c8f02f4 12 days ago 232.3 MB 9e52e7b4a0ee 4 weeks ago 149 B 02930da590e1 4 weeks ago 0 B 72e10143e54a 6 weeks ago 125.9 MB b74728ce6435 11 weeks ago 0 B 511136ea3c5a 8 months ago 0 B Only  deploy  what  changed 397b4009c926 5 seconds ago 0 B 39afabb5721c 6 seconds ago 0 B 5f470d30018a 6 seconds ago 587.5 kB 18  /  34
  19. 19. Why? Developer Friendly Iterate Testing Dependency Management Sharing Standardized Isolation Faster Deployments Infrastructural Freedom Infrastructure  Freedom What  host  can  your  service  run  on? On  host  ip-­10-­244-­202-­48: $ docker run -i -t postgres pg_dump On  host  ip-­10-­204-­151-­29: $ docker run -i -t postgres pg_dump On  host  ip-­10-­2-­1-­15: $ docker run -i -t postgres pg_dump What  host  does  your  service  run  on? Who  cares! 19  /  34
  20. 20. 20  /  34
  21. 21. How? Standardize Describe Deploy Discover Standardize Everything  is  a  Docker  Container  (except  Windows): $ docker build -t mediafly-cms . $ docker build -t mediafly-jenkins-master . $ docker build -t mediafly-jenkins-worker . $ docker build -t mediafly-monitor . $ docker build -t mediafly-postgres . $ docker build -t mediafly-rabbitmq . $ docker build -t mediafly-servicelocator . $ docker build -t mediafly-transcoder . etc. Always  Build  All  Dependent  Containers $ docker pull ubuntu:12.04 $ docker build -t mediafly-ubuntu-12.04-base $ docker build -t mediafly-ubuntu-12.04-base-flask $ docker build -t mediafly-monitor Always  EXPOSE  Ports EXPOSE 8888 Always  Specify  Default  CMD CMD gunicorn web:app --workers 8 --bind 0.0.0.0:8888 --log-level=DEBUG Prefer  One  Service/Container 21  /  34
  22. 22. How? Standardize Describe Deploy Discover Describe Create  Container  Descriptor $ cat <<EOF > appcluster01/containers/mediafly-elasticsearch.json { "tag": "mediafly-elasticsearch", "version": "r2014.03.12b01", "url": "containers/mediafly-elasticsearch:r2014.03.12b01.json", "enabled": true, "ports": [ { "host": 9200, "container": 9200 }, { "host": 9300, "container": 9300 } ], "volumes": [ { "host": "/mnt/elasticsearch", "container": "/mnt/elasticsearch" } ] } EOF Why? Provide  additional  information  about  container  such  as  where  to  deploy  it, monitoring  requirements,  volume  and  port  mappings,  etc. 22  /  34
  23. 23. How? Standardize Describe Deploy Discover Deploy Deploy  Images  to  S3 $ container-export --image mediafly-elasticsearch:r2014.03.12b01 23  /  34
  24. 24. How? Standardize Describe Deploy Discover Deploy Example  Container  Manifest $ cat mediafly-elasticsearch:r2014.03.12b01.json { "id": "e8e2c810d9af", "layers": [ { "id": "511136ea3c5a", "url": "s3://bucket/layers/511136ea3c5a.json" }, { "id": "6170bb7b0ad1", "url": "s3://bucket/layers/6170bb7b0ad1.json" }, { "id": "9cd978db300e", "url": "s3://bucket/layers/9cd978db300e.json" } etc. ] } Example  Layer  Manifest $ cat 511136ea3c5a.json { "id": "511136ea3c5a", "key": "s3://bucket/layers/511136ea3c5a.tar.xz.key.enc", "url": "s3://bucket/layers/511136ea3c5a.tar.xz.enc" } 24  /  34
  25. 25. How? Standardize Describe Deploy Discover Discover Register  All  Containers  with  Service  Locator 25  /  34
  26. 26. How? Standardize Describe Deploy Discover Discover Example  Service  Descriptor $ cat mediafly-elasticsearch.json { "mediafly-elasticsearch": { "9200": [ "ec2-12-34-56-2.compute-1.amazonaws.com:9200", "ec2-12-34-56-2.compute-1.amazonaws.com:9200", "ec2-12-34-56-3.compute-1.amazonaws.com:9200" ], "9300": [ "ec2-12-34-56-2.compute-1.amazonaws.com:9300", "ec2-12-34-56-2.compute-1.amazonaws.com:9300", "ec2-12-34-56-3.compute-1.amazonaws.com:9300" ] } } 26  /  34
  27. 27. 27  /  34
  28. 28. Lessons Layer Management Operations Support Areas  of  Friction Docker  is  not  Git Layer  Management Container  Layering Frequent  and  smaller  changes  at  the  top. Infrequent  and  larger  changes  at  the  base. 28  /  34
  29. 29. Lessons Layer Management Operations Support Areas  of  Friction Docker  is  not  Git Layer  Management Test  Caching  Behavior Make  sure  all  of  your  layers  cache  properly. Clean  Up  After  Yourself Clean  up  your  apt  packages!  You  can  easily  double  the  size  (or  worse)  of  your containers  if  you  don't  clean  up  after  yourself. Verify  the  Size  of  Your  Layers Use  the  docker history  command  to  see  the  size  of  your  layers.  Check  the  file system! Don't  Build  Layers  Manually Don't  build  containers  manually  (only  do  that  to  better  understand  how  Docker works). Favor  Building  Images  over  Pulling  from  Registry The  registry  is  great  for  sharing  and  deployment,  but  you  can  easily branch/merge/diff  Dockerfiles. 29  /  34
  30. 30. Lessons Layer Management Operations Support Areas  of  Friction Docker  is  not  Git Operations  Support Connect  to  a  running  container $ lxc-attach -n CONTAINER /bin/bash Access  a  container's  filesystem $ cd /var/lib/docker/aufs/mnt/CONTAINER Access  a  containers  log $ tail -f /var/lib/dockers/containers/CONTAINER/CONTAINER-json.log Clean  up  old  containers $ docker ps -a | grep Exit | awk '{print $1}' | xargs docker rm Clean  up  old  images $ docker images | grep IMAGENAME | awk '{print $3}' | xargs docker rmi Stop  all  running  containers $ docker ps -q | xargs docker stop -t 0 30  /  34
  31. 31. Lessons Layer Management Operations Support Areas  of  Friction Docker  is  not  Git Areas  of  Friction  (as  of  03/2014) I  fully  expect  most  of  these  issue  to  be  addressed  in  future  versions. Docker  Registry We  chose  not  to  use  the  registry  because  of  the  following: buggy,  slow,  scaling  bottleneck additional  point  of  failure increases  support  burden  and  infrastructure  costs hard  to  secure Supplementary  Services There's  no  standard  for  handling  supplementary  services  such  as  logrotate, sendmail,  and  syslog. Init  Systems Integration  with  init  systems  such  as  upstart  or  systemd  is  a  work  in  progress. Creating  Containers You  can't  create  a  container  (docker run)  without  running  it. Volume  Management Stateful  data  is  your  problem.  There  is  no  standardized  solution  for  permanent  disk storage,  backups,  and  replication. 31  /  34
  32. 32. Lessons Layer Management Operations Support Areas  of  Friction Docker  is  not  Git Docker  is  not  Git Docker  is  not  Git,  but  you  can  do  some  Git-­like  things  with  it.  Here's  an  alternate analogy  to  help  you  understand  it. What  is  Docker  like? It's  like  an  incremental  build  system  with  shared  network  storage. What's  the  souce  code? Dockerfile's  are  source  code. What  are  compiled  artifacts? Layers  are  like  binaries  compiled  from  the  source  code  (e.g.  jars). What  are  compiled  applications? Images  are  like  compiled  applications  (.e.g  like  executables). Installed  Executables Containers  are  installed  executables  (running  or  stopped). Shared  Network  Storage The  registry  is  shared  network  storage  for  binaries  (layers  and  images). 32  /  34
  33. 33. Any  Questions? 33  /  34
  34. 34. twitter.com/bryanmurphy bmurphy@mediafly.com 34  /  34

×