Scala, ECS, Docker: Delayed Execution @Coursera
•
0 likes•467 views
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1stYuc2. Brennan Saeta covers aspects of Coursera’s architecture that enables them to rapidly build sophisticated features for their learning platform. Saeta discusses also their experience running containers in production, what works, what doesn’t, and why. He briefly touches upon container threat models, and how to architect a defense-in-depth strategy to mitigate both known and unknown vulnerabilities. Filmed at qconlondon.com. Brennan Saeta is a Lead Infrastructure Engineer, leading the ‘Cour’ (core) group responsible for the development environment, core libraries, and the common infrastructure powering Coursera.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (17)
Similar to Scala, ECS, Docker: Delayed Execution @Coursera
Similar to Scala, ECS, Docker: Delayed Execution @Coursera (20)
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
Scala, ECS, Docker: Delayed Execution @Coursera
- 1. ECS & Docker: Secure Async Execution @ Brennan Saeta
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ scala-ecs-docker-coursera
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
- 5. The Beginnings — 2012 10 courses 1 million learners worldwide 4 partners
- 6. Education at Scale 1,800 courses 18 million learners worldwide 140 partners
- 7. Outline • Evolution of Coursera’s nearline execution systems • Next-generation execution framework: Iguazú • Iguazú application deep dive: GrID — evaluating programming assignments
- 8. Key Takeaways • What is nearline execution, and why it is useful • Best practices for running containers in production in the cloud • Hardening techniques for securely operating container infrastructure at scale
- 9. A history of nearline execution
- 12. Early days - Requirements • Video re-encoding for distribution • Grade computation for 100,000+ learners • Pedagogical data exports for courses
- 16. Upgrading to Scala Re-architecting delayed execution for our 2nd generation learning platform.
- 17. Upgrading to the JVM • Leverage mature Scala & JVM ecosystems for code sharing • JVM much more reliable (no memory leaks) • New job model: scheduled recurring jobs. • Named: Saturn
- 18. Saturn Architecture Service A Service B Service C C* Online Serving Scala/micro-service architecture C*
- 19. Saturn Architecture Service A Service B Service C C* Online Serving Scala/micro-service architecture Saturn C*
- 20. Saturn Architecture Service A Service B Service C C* Saturn C* ZK Ensemble
- 21. Saturn Architecture Saturn Leader ZK Ensemble Service A Service B Service C C*C*
- 22. Problems with Saturn • Single master meant naïve implementation ran all jobs in same JVM • Huge CPU contention @ top of the hour • OOM Exceptions & GC issues
- 23. Enter: Docker Containers allow for resource isolation! CC-by-2.0 https://www.flickr.com/photos/photohome_uk/1494590209
- 24. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ☑️ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 25. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 26. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 27. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 28. Supported Features Platform Saturn Docker Amazon ECS ??? Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 29. Solution: Iguazú Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0
- 30. Solution: Iguazú • Framework & service for asynchronous execution • Optimized Scala developer experience for Coursera • Unified scheduler supports: • Immediate execution (nearline) • Scheduled recurring execution (cron-like) • Deferred execution (run once @ time X) Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0
- 31. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers SQS ECS API Devs Users
- 32. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers SQS Queue ECS API Devs Users
- 33. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers ECS API Devs Users SQS Queue
- 34. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers ECS API Devs Users ZK Ensemble SQS Queue
- 35. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers ECS API Devs Users ZK Ensemble SQS Queue
- 37. Autoscaling ⇄ Iguazú ⇆ ECS Iguazu ECS APIAutoscaling EC2 Worker EC2 Worker Shutdown Lifecycle Notification Poll Worker Job Status All finished Proceed Term- inate EC2 Worker
- 38. Failure in Nearline Systems • Most jobs are non-idempotent • Iguazú: At most once execution • Time-bounded delay • Future: At least once execution • With caveats
- 39. Iguazú adoption by the numbers ~100 jobs in production >1000 runs per day >100 different job schedules
- 40. Iguazú Applications Nearline Jobs • Pedagogical Instructor Data Exports • System Integrations • Course Migrations Scheduled Recurring Jobs • Course Reminders • System Integrations • Payment reconciliation • Course translations • Housekeeping • Build artifact archival • A/B Experiments
- 41. While containers may help you on your journey, they are not themselves a destination.CC-by-2.0 https://www.flickr.com/photos/usoceangov/5369581593
- 42. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 43. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 44. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 45. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 46. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 47. Testing an Iguazu job
- 48. The Hollywood Principle applies to distributed systems. CC-by-2.0 https://www.flickr.com/photos/raindog808/354080327
- 49. Deploying a new Iguazu Job • Developer • merge into master… done • Jenkins Build Steps • Compile & package job JAR • Prepare Docker image • Pushes image into registry • Register updated job with Amazon ECS API
- 50. Invoking an Iguazú Job // invoking a job with one function call // from another service via REST framework RPC val invocationId = iguazuJobInvocationClient .create(IguazuJobInvocationRequest( jobName = "exportQuizGrades", parameters = quizParams))
- 51. A clean environment increases reliability.CC-by-2.0 https://www.flickr.com/photos/raindog808/354080327
- 52. Evaluating Programming Assignments An application of Iguazú
- 55. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 56. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 57. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 58. Solution: GrID Patrick Hoesly (https://www.flickr.com/photos/zooboing/5665221326/) CC-BY-2.0 • Service + framework for grading programming assignments • Builds on Iguazú • Named for Tron’s “digital frontier” • Backronym: Grading Inside Docker
- 59. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS APIs Grading MachinesVPC Firewalls Coursera Production Account Coursera GrID Grading Account
- 60. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS APIs Grading MachinesVPC Firewalls Coursera Production Account Coursera GrID Grading Account
- 61. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS API Grading MachinesVPC Firewalls Production Acct GrID Grading Account
- 62. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS API Grading Machines VPC Firewalls Production Acct GrID Grading Account
- 63. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 65. The Security Challenge Compiling and running untrusted, arbitrary code on our cluster in near real time. Would you like to compile and run C code from random people on the Internet on your servers?
- 66. FROM redis FROM ubuntu:latest FROM jane’s-image
- 67. Security Assumptions • Run arbitrary binaries • Instructor grading scripts may have vulnerabilities • ∴ Grading code is untrusted • Unknown vulnerabilities in Docker and Linux name-spacing and/or container implementation
- 68. Security Goals Prevent submitted code from: • impacting the evaluation of other submissions. • disrupting the grading environment (e.g., DoS) • affecting the rest of the Coursera learning platform
- 69. Grading assignment submissions CC-by-2.0 https://www.flickr.com/photos/dherholz/4367511580/
- 71. CPU CPU CPU CPU RAM Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 72. CPU CPU CPU CPU RAM Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 73. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 74. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 75. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk — blkio limits & btrfs quotas
- 76. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk — blkio limits & btrfs quotas
- 77. Attacks: Kernel Resource Exhaustion • Open file limits per container (nofile) • nproc Process limits • Limit kernel memory per cgroup • Limit execution time
- 79. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel — cgroups, ulimits Disk — blkio limits & btrfs quotas Network
- 80. Attacks: Network attacks Attacks: • Bitcoin mining • DoS attacks on other systems • Access Amazon S3 and other AWS APIs Defense: • Deny network access
- 81. Docker Network Modes NetworkDisabled too restrictive • Some graders require local loopback • Feature also deprecated --net=none + deny net_admin + audit network • Isolation via Docker creating an independent network stack for each container github.com/coursera/amazon-ecs-agent
- 85. Defense in Depth • Mandatory Access Control (App Armor) • Allows auditing or denying access to a variety of subsystems • Drop capabilities from bounding set • No need for NET_BIND_SERVICE, CAP_FOWNER, MKNOD • Deny root within container
- 86. Deny Root Escalations • We modify instructor grader images before allowing them to be run • Clears setuid • Inserts C wrapper to drop privileges from root and redirect stdin/stdout/stderr • Run cleaning job on another Iguazú cluster • Run Docker in Docker! • Docker 1.10 adds User Namespaces
- 87. If all else fails… • Utilizes VPC security measures to further restrict network access • No public internet access • Security group to restrict inbound/outbound access • Network flow logs for auditing • Separate AWS account • Run in an Auto Scaling group • Regularly terminate all grading EC2 instances
- 88. Other Security Measures • Utilize AWS CloudTrail for audit logs • Third-party security monitoring (Threat Stack) • No one should log in, so any TTY is an alert • Penetration testing by third-party red team (Synack)
- 89. Lessons Learned - GrID • Building a platform for code execution is hard! • Carefully monitor disk usage • Run the latest kernels • Latest security patches • btrfs wedging on older kernels • Default Ubuntu 14.04 kernel not new enough!
- 90. Reliable deploy tooling pays for itself.
- 91. Thank you! Brennan Saeta github/saeta @bsaeta saeta@coursera.org Frank Chen github/frankchn @frankchn frankchn@coursera.org GrID lead Iguazú Lead
- 92. Questions? Brennan Saeta github/saeta @bsaeta saeta@coursera.org Frank Chen github/frankchn @frankchn frankchn@coursera.org GrID lead Iguazú Lead
- 93. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/scala- ecs-docker-coursera