Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PDXPortland - Dockerize Django


Published on

  • Be the first to comment

PDXPortland - Dockerize Django

  1. 1. Dockerizing Django PDX Portland
 October 27th, 2016
  2. 2. Who are we? Michael Dougherty
 Senior Front-end Engineer
 CrowdStreet, Inc. Hannes Hapke
 Software Engineer
 Talentpair, Inc.
  3. 3. Our pre-Docker World … • Single instance world
 (e.g. celery ran on the web server) • Outdated Amazon machine image • No documentation about the setup, consultancy work • Live data monkey patching • Scaling/Recovery time > 8 hours • Clunky QA setup > bottleneck
  4. 4. Our post-Docker World … • Single instance world
 (e.g. celery ran on the web server) • Outdated Amazon machine image • No documentation about the setup, consultancy work • Live data monkey patching • Scaling/Recovery time > 8 hours • Clunky QA setup > bottleneck • One service per container, redundancy of instances • One common base image shared across all instances • Explicit, declarative server setup • Immutable infrastructure (mostly) • Scaling/Recovery time ~ 20min • As many QA instances as we want
  5. 5. What is Docker? Docker • Compose • Machine • Swarm
  6. 6. Docker containers … … wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment. * Basically a virtual env for your operating system. * from
  7. 7. Docker vs. Vagrant
  8. 8. Where is the difference? Images from VM includes OS No OS needed
  9. 9. How does Docker work?
  10. 10. • Create a Dockerfile • Build the Docker image and push it to the docker registry Plain Docker FROM ubuntu:16.04 RUN apt-get update && apt-get upgrade -y RUN pip install Django COPY requirements.txt . WORKDIR /dev $ docker build -t your_project/your-whale . $ docker images
 $ docker tag {image_hash} your_project/your-whale:latest
 $ docker push
  11. 11. Plain Docker • Run a shell in a docker container
 -i start an interactive container
 -t creates “Pseudo interface” with stdin and stdout • Run the Django server in a container
 -d run container in detached mode
 -P maps all ports to the host machine $ docker run -i -t ubuntu /bin/bash $ docker run -d -P my-container python run server
  12. 12. What if we need multiple services? Docker • Compose • Machine • Swarm
  13. 13. Docker Compose Compose is a tool for orchestrating the building, running, and intercommunication of multi-container Docker applications.
  14. 14. How does it work? 1) Define a Dockerfile for every service 2) Define a Docker compose description of the environment 3) Use docker-compose build/up to start all services version: '2' services: web: build: . ports: - "5000:5000" volumes: - .:/code redis: image: redis
  15. 15. How can I easily provision a server with the containers? Docker • Compose • Machine • Swarm
  16. 16. Docker Machine … is a great tool which creates Docker hosts anywhere.
 Yes, anywhere.
 Locally, AWS EC2, Digital Ocean, MS Azure, you name it. No Ansible, Puppet, Chef, fabric, etc. required.
  17. 17. What if I need multiple instances with multiple services? Docker • Compose • Machine • Swarm
  18. 18. Docker Swarm
  19. 19. Dockerize for real …
  20. 20. If you start from scratch … • Docker documentation includes a great Django setup • Too much work? The Django Cookie Cutter template includes a great Docker setup
 Other projects: • django-docker on github
  21. 21. If you convert a project like us …
  22. 22. Reorganize your folder structure
  23. 23. Normalize folders • Create folders for every service • docker-compose-{env}.yml go into the project root • Dockerfiles go into every service folder • scripts go into the service folders • Keep your local folder structure similar to the folder structure within the container(s) - for sanity
  24. 24. Reorganized folders Project Root
 requirements.txt Project Root
 | |-apps
 | |- …
 | |-Dockerfile
 | |
  25. 25. Build a base image
  26. 26. Base Image • Create one (or more) base Dockerfile(s) with all common packages • Service containers can use this base image - this will increase build speed • If you store the base image(s) in a separate git repo, the docker registry will build them automatically for you
  27. 27. Base Image FROM ubuntu:16.04 RUN apt-get update && apt-get upgrade -y RUN apt-get install -y vim # Install some useful editor RUN apt-get install -y build-essential git software-properties-common RUN apt-get install -y python python-dev python-setuptools build-essential RUN apt-get install -y nodejs npm RUN npm install -g n # upgrading the npm version RUN n stable ...
  28. 28. Base Image Add image of the Docker registry
  29. 29. Set up Docker compose for the different environments
  30. 30. Docker Compose • For every environment, local, QA, staging, production, define a docker-compose-{env}.yml file • The files describe the environment stack • Each service within the docker-compose file can have it’s own Dockerfile
  31. 31. Docker Composeversion: '2' volumes: postgres_data_dev: {} redisdata: {} webpack_data: {} services: postgres: image: postgres:9.5 volumes: - postgres_data_dev:/var/lib/postgresql/data restart: always environment: - POSTGRES_USER=postgres_user - POSTGRES_DB=my_fancy_db - POSTGRES_PASSWORD= webpack: image: crowdstreet/crowdstreet-whale:latest command: npm run watch environment: - NODE_PATH=/node_modules volumes: - ./webpack/frontend-src:/frontend-src - ./django:/crowdstreet-src - webpack_data:/webpack_data/ ports: - "3000:3000" restart: always
  32. 32. Docker Composedjango: build: context: . dockerfile: ./django/Dockerfile-dev command: python /crowdstreet-src/ runserver depends_on: - postgres environment: - ENV=dev - DJANGO_SETTINGS_MODULE=settings volumes: - ./django:/crowdstreet-src - ./webpack/frontend-src:/frontend-src - webpack_data:/webpack_data/ ports: - "8000:8000" - "80:8000" links: - postgres - redis - webpack - memcached redis: restart: always image: redis:latest volumes: - redisdata:/data restart: always
  33. 33. Docker Compose • Build your service stack with • Start the container stack with • Access a single container with $ docker-compose -f docker-compose-{env}.yml build $ docker-compose -f docker-compose-{env}.yml up $ docker-compose -f docker-compose-{env}.yml run django bash $ docker-compose -f docker-compose-{env}.yml 
 run container name command
  34. 34. Set up Docker machine and deploy to the world
  35. 35. Docker machine is awesome!
  36. 36. Docker Machine • With
 will provision you an AWS instance • “Activate” the instance with
 • Afterwards, any docker-compose command will be executed on the active machine • Easy to start/stop/terminate machines $ docker-machine create --driver amazonec2 
 --amazonec2-region [e.g. us-west-2] 
 --amazonec2-vpc-id [YOUR_VPC_ID vpc-xxxxxx] 
 --amazonec2-instance-type [e.g. t2.small] [INSTANCE_NAME] $ docker-machine env [INSTANCE_NAME]
  37. 37. Lessons Learned
  38. 38. Or... how to cowboy code with Docker • Sometimes you just need to manually change something • Docker provides ways to get a shell inside a running instance and copy files back and forth • Your changes will of course be lost next time you spin up a new container
  39. 39. The Disciplined Way: The Cowboy Way: $ docker-compose run django bash $ docker exec -it {container_id} bash
  40. 40. How does QA work with Docker? • No QA bottleneck anymore • No database gridlock anymore • Each feature branch gets its own instance • Once feature is tested, instance gets terminated
  41. 41. How can I access the shell/migrate? • Access the bash of the django container with
 • Continue as usual with
 Some for migrations, make_migrations, etc. • Or run it from outside of the container stack with $ docker-compose -f docker-compose-{env}.yml run django bash # ./ shell docker-compose -f … run django python migrate
  42. 42. Help, ipdb doesn’t work anymore … • Start the Django container with the service ports enabled
 • If no command is specified, then Docker will default to the command in the docker-compose.yml file $ docker-compose -f dev.yml run --service-ports django
  43. 43. How to run tests? • Start the Django container with your test command
 $ docker-compose -f … run django test
  44. 44. CI Testing is convenient • Setup for Circle CI machine: pre: - curl -sSL | bash -s -- 1.10.0 services: - docker dependencies: override: - sudo pip install docker-compose - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS - docker-compose -f docker-compose-circle.yml build - npm install -g jshint test: pre: - sudo killall postgres # not sure why, but port 5432 is already taken up sometimes! - docker-compose -f docker-compose-circle.yml up -d postgres override: - jshint ~/your_project/django/static/js/your_project* - docker-compose -f docker-compose-circle.yml run django /your_project/ test --verbosity=2
  45. 45. WTF, the files I copied into my container are missing?? • If a volume is mounted at the same directory where you copied other files, you will essentially overwrite those files
  46. 46. Sharing Docker Machine credentials • Docker machine is great, but there is no concept of sharing credentials • All credentials are simple text files, no magic • npm tool `machine-share` solved the problem • Let’s you export and import machine credentials
  47. 47. General Troubleshooting • Confirm that the correct docker-machine environment is active • Rebuild your container stack • Rebuild with the --pull and/or --no-cache options • Restart the docker daemon • Restart your docker machine with docker-machine restart [INSTANCE NAME] • Restart your docker machine VirtualBox VM • Remove and recreate your docker machine (essentially recreates your dev environment from scratch)
  48. 48. So, what does our setup look like now?
  49. 49. Dev Environment • You can use the same image as in your production builds • All services run at once, all output piped to a single log stream (which we saw earlier) • You can still have live reloading via Docker Volumes (but be careful!)
  50. 50. How does the deployment work now? • Create AWS instance with docker-machine • Activate the docker machine • Use docker-compose to build the stack • Use docker-compose up -d • Switch the load balancer
  51. 51. Summary of technologies • Learned about Docker • How to use docker to define images and containers • Learned about Docker-compose to define relationships between containers • Learned about Docker-machine to seamlessly work with containers on local/remote machines
  52. 52. Summary of benefits • Explicit, declarative server setup • Zero down time deployments • All dev services in one "window" and start with one command • Easy provisioning of multiple QA instances • Quick onboarding for new devs
  53. 53. Thank you!
  54. 54. Q&A