Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Iguazú: A Long-Running Job Scheduler using Docker and Mesos

21,506 views

Published on

These slides describe how Coursera created its next generation of long-running job scheduler by integrating Docker and Mesos.

Published in: Engineering

Iguazú: A Long-Running Job Scheduler using Docker and Mesos

  1. 1. IGUAZÚ A Job Scheduler Using Mesos and Docker Colleen Lee Software Engineer
  2. 2. "COURSERA" IN 2011
  3. 3. "COURSERA" IN 2011
  4. 4. ! ! ! ! "COURSERA" IN 2011 ! • Work to be done: gradebook exports, regrading quizzes, sending batch emails, encoding videos, etc.
  5. 5. CASCADE
  6. 6. Cascade: Lifecycle of a Job
  7. 7. HOW DOES CASCADE WORK? Client
  8. 8. HOW DOES CASCADE Job data Client SQS Database Job information WORK?
  9. 9. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade Database
  10. 10. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade New job Database Poll
  11. 11. HOW DOES CASCADE Client WORK? SQS Worker Worker Running... Cascade Status information Database
  12. 12. HOW DOES CASCADE Client SQS Worker Worker Cascade No new jobs! Database Poll Zzz... WORK?
  13. 13. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade Status? Job status Database
  14. 14. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade Job is done! Database
  15. 15. LACK OF ISOLATION Worker 2 Worker 1 Worker 2 Cascade Worker 1 CPUs Memory Worker 2 Worker 1
  16. 16. LACK OF ISOLATION Worker 1 Worker Worker 2 Worker 1 Worker 2 Cascade 2 Worker 1 CPUs Memory
  17. 17. FRAGILE DEPLOYMENT FLOW
  18. 18. FRAGILE DEPLOYMENT FLOW System code vs. Job code
  19. 19. FRAGILE DEPLOYMENT FLOW System code vs. Job code Rarely updated Updated all the time
  20. 20. FRAGILE DEPLOYMENT FLOW System code vs. Job code Rarely updated Updated all the time Unique system no deploy tooling
  21. 21. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code???
  22. 22. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code!
  23. 23. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code! Worker (new code) Poll for job... Run job... Poll for job... Run job...
  24. 24. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code! Worker (new code) Poll for job... Run job... Poll for job... Run job...
  25. 25. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code! Worker (new code) Poll for job... Run job... Poll for job... Run job... Worker Worker Worker Cascade Worker Worker Worker Cascade Worker Worker Worker Cascade
  26. 26. POOR DEVELOPMENT STORY
  27. 27. POOR DEVELOPMENT STORY Client SQS Worker Worker Cascade Database
  28. 28. POOR DEVELOPMENT STORY Dev 1 Client SQS Worker Cascade DB
  29. 29. POOR DEVELOPMENT STORY Dev 1 Client SQS Worker Cascade DB Client Worker Dev 2 Cascade DB
  30. 30. POOR DEVELOPMENT STORY Job data Poll Dev 1 Client SQS Worker Cascade DB New job Client Worker Dev 2 Cascade DB Where'd my job go? What is this job?
  31. 31. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story
  32. 32. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language
  33. 33. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language Cascade-Scala? Cascade-Python?
  34. 34. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language Cascade-Scala? Cascade-Python? Duplicating work: BAD Static partitioning: BAD
  35. 35. 2014: CASCADE V2??
  36. 36. 2014: CASCADE V2??
  37. 37. 2014: CASCADE V2?? DATABASE MIGRATIONS!! SCHEDULED JOBS!! PROGRAMMING ASSIGNMENTS!
  38. 38. 2014: CASCADE V2??
  39. 39. IGUAZÚ!
  40. 40. IGUAZÚ!
  41. 41. • Resource isolation: cgroups ! ! ! ! !
  42. 42. • Resource isolation: cgroups • Master(s) has/have "soft state" • Coordination AND robustness ! ! !
  43. 43. • Resource isolation: cgroups • Master(s) has/have "soft state" • Coordination AND robustness • Implementation: • Scheduler: accepts and manages resources • Executor: process launched on slaves to run tasks
  44. 44. • Lightweight, but provides abstraction of a VM
  45. 45. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting!
  46. 46. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting!
  47. 47. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting! • Private Docker registry: convenience, versioning
  48. 48. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting! • Private Docker registry: convenience, versioning • Usage: specify the image and specify a command
  49. 49. Iguazú: Lifecycle of a Job
  50. 50. HOW DOES IGUAZÚ WORK? Client
  51. 51. HOW DOES IGUAZÚ WORK? Client SQS Iguazú Job data Production mode Job id Database Job information
  52. 52. HOW DOES IGUAZÚ WORK? Client In-memory Iguazú Job data Development mode Job id Database Job information
  53. 53. HOW DOES IGUAZÚ WORK? Client SQS Poll Manager Iguazú Database
  54. 54. HOW DOES IGUAZÚ WORK? Client SQS Job data Manager Scheduler Iguazú Database
  55. 55. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Slave
  56. 56. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor Slave
  57. 57. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor registry Check for Slave new image
  58. 58. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor registry Status updates Slave
  59. 59. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database TASSKla_vFeINISHED Executor registry Status updates Slave
  60. 60. HOW DOES IGUAZÚ WORK? All done! TASSKla_vFeINISHED Master(s) Client SQS Manager Scheduler Iguazú Database Executor registry Status updates Slave
  61. 61. HOW DOES IGUAZÚ WORK? All done! TASSKla_vFeINISHED Master(s) Client SQS Manager Scheduler Iguazú Database Executor registry Status updates Slave OK!
  62. 62. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor registry Job is done! Slave Status? Job status
  63. 63. REMEMBER CASCADE'S PROBLEMS? • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language
  64. 64. REMEMBER CASCADE'S PROBLEMS? Mesos: cgroups! • Isolation! • Fragile deployment flow • Poor development story • Tied exclusively to one language
  65. 65. REMEMBER CASCADE'S PROBLEMS? • Isolation! • Easy deployment flow • Poor development story • Tied exclusively to one language Private Docker repo
  66. 66. REMEMBER CASCADE'S PROBLEMS? • Isolation! • Easy deployment flow • Consistent development story • Tied exclusively to one language Iguazú: proper abstractions
  67. 67. REMEMBER CASCADE'S PROBLEMS? • Isolation! • Easy deployment flow • Consistent development story • Any language No restrictions
  68. 68. OTHER BENEFITS
  69. 69. OTHER BENEFITS • Ease of transition ! ! ! ! Mesos: job management Docker: job packaging
  70. 70. OTHER BENEFITS • Ease of transition • Performance ! ! ! Mesos: long-running! PHP: blaaargh
  71. 71. OTHER BENEFITS • Ease of transition • Performance • Flexibility ! ! Use Docker, run Scala code, etc.
  72. 72. OTHER BENEFITS • Ease of transition • Performance • Flexibility • Fine-grained control over scheduling ! Autoscaling!
  73. 73. OTHER BENEFITS • Ease of transition • Performance • Flexibility • Fine-grained control over scheduling • Designed to work on a heterogeneous pool of resources Security :)
  74. 74. THANKS! QUESTIONS? ! We are hiring! See http://coursera.org/jobs ! @firejade0 clee@coursera.org

×