1.
Scaling Open edX
with Kubernetes
DevOpsDays Boston
9.15.2015

2.
Who we are
Nate Aune
Morgan Robertson

3.
What we’ll cover
● Background -- Open edX
● Introducing Kubernetes
● Kubernetes concepts
● Scaling + resiliency
● Open edX on Kubernetes

4.
Open edX background
● edX: non-profit founded by MIT and Harvard
● 500+ courses, 5M students learning on edX.org
● edX released Open edX in June 2013
● Stanford, MongoDB, Salesforce, Google, Microsoft,
McKinsey, Johnson & Johnson, Smithsonian

5.
Open edX - a catalyst for innovation
212 Contributors
One of the fastest
growing open
source projects on
Github

6.
Technical components
LMS/CMS (Django/Python)
Forum (Sinatra/Ruby)
User DB (MySQL)
Course DB (Mongo)
Tasks (Celery/RabbitMQ)
Caching (Memcache)
Proxy (Nginx)
Search (ElasticSearch)
Mapreduce (Hadoop)

7.
Hosting infrastructure
S3 for serving:
● static assets
● grade downloads
● certificate downloads
● videos (for mobile)
● Load balancer
● Application server(s)
● Database server(s)
● Search server
● Utility server (tasks)
● Caching server
● Hadoop cluster

8.
Typical scalable deployment of Open edX on AWS

9.
Introducing Kubernetes
● Scheduling + orchestration layer for
containerized applications
● Abstracts your infrastructure
● Open source project by Google
● Production-ready as of July 2015

10.
Kubernetes architecture

11.
Kubernetes vs. the Docker triad
Kubernetes Swarm Compose Machine
Scheduling ✔ ✔
Service discovery ✔ ✅
Container scaling ✔ ✔
Machine provisioning ✅ ✔
Health checking ✔
Secret management ✔
Production-ready ✔

12.
Kubernetes core concepts
● Pods
● Services
● Replication controllers

13.
Pods
● Group of containers + volumes scheduled together
● Smallest deployable unit
● Containers share certain resources including network
stack

14.
Services

15.
Services
● Endpoint for a set of pods
● IP address, port, and label selectors
● Use round-robin routing to direct traffic to backend
pods

16.
Services + Pods

17.
Replication Controllers
● Manage pod lifecycles for a number of replicas
● Provide scaling + fault tolerance
● Use label selectors

18.
Pods + Services + Replication Controllers

19.
Scaling with Kubernetes
● Replication controllers scale pods
● Services provide a single endpoint for a group of
pods
● The Kubernetes master schedules pods across nodes

20.
Resiliency with Kubernetes
● Replication controllers ensure a number of pods are
running
● Services provide load balancing
● Health checks allow bad pods to be ignored/removed

21.
Open edX on Kubernetes
● Goals:
○ Multi-tenant
○ Scalable + resilient

22.
The challenge

23.
Architecture

24.
Monitoring with Sysdig

25.
Sysdig drill-down

26.
Lessons learned
● Containers should be stateless
● Put initialization tasks into separate pods that run
once
● Services can be used to abstract non-containerized
components

27.
Conclusion
● We’re still learning, but..
○ Kubernetes is a promising technology for
providing both scalability and resiliency

28.
More info
Open edX - http://open.edx.org
Kubernetes - http://kubernetes.io
Google Container Engine - http://cloud.
google.com/container-engine

29.
Thank you for your time!
Questions?
Slides: http://bit.ly/open-edx-kubernetes
nate@appsembler.com
morgan@appsembler.com
@appsembler