Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mesos & Django (and Aurora and CircleCI)

2,289 views

Published on

Slides from a talk given at Python Ireland July 2015: http://www.meetup.com/pythonireland/events/221222978/

This talk will give a brief introduction to a potential stack for building and deploying a Django app onto a Mesos cluster. It's ideal for anyone who has a little bit of experience trying to manage more than a couple of machines.

This talk will cover Mesos, Aurora, Django, CircleCI and anything else Michael can squeeze into a 30 minute talk.

Published in: Software
  • Be the first to comment

Mesos & Django (and Aurora and CircleCI)

  1. 1. Mesos & Django (and Aurora, CircleCI) Python Ireland July 2015 michael.twomey@udemy.com @micktwomey
  2. 2. Who is this talk for? • The curious • Folks who look after more than a handful of machines and services • You have a "platform team" or "devops" (what does that even mean?) • Get woken at 3am to fix things
  3. 3. Warning
  4. 4. Precious snowflake boxes • Lots of different machines (web-01, redis-03, dont-delete-web-99, memcached4) • No one remembers who or how the celery box was setup but we can't build a new one • My fabric scripts need unit tests • I really want to try that new framework but that means finding a box, I know that web server doesn't have much traffic... • Sod it, lets use Heroku!
  5. 5. No, really, use Heroku !
  6. 6. I need to manage lots of machines and those annoying "CI" developers I know I'll use Mesos!
  7. 7. Now you have two problems! No, wait, wrong analogy
  8. 8. A distributed systems kernel — mesos.apache.org
  9. 9. Wat?
  10. 10. A programmable monit/ supervisor/ upstart for all the things Wat?
  11. 11. Makes it much easier to manage lots of processes 1. Install it on all your machines 2. Run a Mesos framework (or two) 3. Profit!
  12. 12. Step 1: Deploy Mesos • Now you've got a hundred, easily deployed machines running Mesos slaves • Now what?
  13. 13. Step 2: Add a Framework What's a Framework? The brains of the operation
  14. 14. ™Apache
  15. 15. Aurora • Transactional deployments • Process management • Scaling • Mix production and development • "Cron" jobs • Did I mention transactional deployments
  16. 16. Acaversity A simple Django app
  17. 17. I mean, really simple def index(request): version = 3 html = ("<html><body>" "<h1>Hello</h1>" "<p>What do you want to learn today?</p>" "<p>Version {}</p></body></html>").format(version) return HttpResponse(html) def healthcheck(request): return JsonResponse({"ok": True})
  18. 18. Acaversity build • Django app • Build and test on CircleCI • Upload a slug to S3
  19. 19. Acaversity deployment • Deploy using Aurora • Fetches slug • Unpacks • Starts process • Health checks to victory • Registers with Zookeeper
  20. 20. Whoops
  21. 21. Circle of Life
  22. 22. acaversity.aurora fetch = Process(...) setup = Process(...) start = Process(...) django_task = SequentialTask(...) jobs = [ Service(...) ]
  23. 23. 1. Fetch the slug fetch = Process( name = 'fetch', cmdline = ('export AWS_ACCESS_KEY_ID=... ' 'export AWS_SECRET_ACCESS_KEY=... ' 'env ' '; aws s3 cp s3://... slug.tar.gz ' '&& tar -zxf slug.tar.gz' ), ) Bonus tip: Use vault for your secrets
  24. 24. 2. Run the setup script setup = Process( name = 'setup.sh', cmdline = 'bash slug/scripts/setup.sh', )
  25. 25. Start the process start = Process( name = 'start.sh', cmdline = ('bash slug/scripts/start.sh ' '--http :{{thermos.ports[http]}} ' '--http :{{thermos.ports[health]}}') )
  26. 26. 3. Define the overall task django_task = SequentialTask( processes = [fetch, setup, start], resources = Resources(cpu = 1, ram = 100*MB, disk=100*MB) )
  27. 27. Side note: resources Resources(cpu = 1, ram = 100*MB, disk=100*MB) • Asks for a Mesos slave which can give a logical core, 100MB of RAM and 100MB of disk. • Runs in container and restricts to these resources • e.g. kills your process if it starts spewing data to disk • CPU scheduler restricts run time
  28. 28. 4. Create a Service update_config = UpdateConfig(watch_secs=45, batch_size=2) health_check_config = HealthCheckConfig( initial_interval_secs=15, interval_secs=10, expected_response='{"ok": true}', ) announcer_config = Announcer() jobs = [ Service(cluster = 'devcluster', environment = 'devel', role = 'www-data', name = 'acaversity', instances = 4, task = django_task, update_config = update_config, health_check_config = health_check_config, announce = announcer_config, constraints = { 'host': 'limit:10', })]
  29. 29. Bonus: What would the Docker version look like? jobs = [ Service(cluster = 'devcluster', environment = 'devel', role = 'www-data', name = 'acaversity-docker', task = django_docker, container = Container( docker = Docker( image = 'acaversity:1.2' )))]
  30. 30. Deploying aurora update start devcluster/www-data/devel/acaversity acaversity.aurora
  31. 31. Scaling up instances = 10 aurora update start devcluster/www-data/devel/acaversity acaversity.aurora Question: What does it do with the old intances?
  32. 32. Cron Jobs jobs = [ Job( cluster = 'devcluster', role = 'www-data', environment = 'test', name = 'cron_hello_world', cron_schedule = '*/5 * * * *', task = SimpleTask( 'cron_hello_world', 'echo "Hello world from cron"'), ), ]
  33. 33. Time to anger the demo gods
  34. 34. Thank You! PS: Going to Ely's PPS: Udemy is hiring PPPS: Pick up one chair each and take it back to the middle

×