The document discusses Comcast's journey towards continuous delivery. It describes how Comcast has transitioned from a process where deployments were done by many humans over much time, to one where a single human and many machines can deploy changes much faster. It introduces "Gumby", a tool developed by Comcast to automate deployments across various cloud platforms like vSphere, Openstack, and EC2 using technologies like Puppet, Git, and Cloud-Init. Gumby started as an experiment using Play and Akka but was rewritten using Spray and Akka to scale deployments. It is now used to deploy around 60% of Comcast's X1 backend infrastructure.

1. Are we there yet?
Continuous Delivery @Comcast
Brendan O’Bra, Principal Engineer (brendan_o’bra@cable.comcast.com)
Twitter: @brendanobra
LinkedIn: https://www.linkedin.com/in/brendanobra
September 17th, 2015
Scala Bay Meetup

2. We are more than just cable TV…
Internet Phone On Demand
Home
Security
Business
Services
(Internet)
Next Gen
TV (IP
Video)
Internet TV
Apps (On
X1)
TV Content
Production
(NBC)
Movies
(Universal)

3. Almost everything at “National Scale”
• 10’s of Datacenters across US
• Traffic can be “surge-ish”
• “Carrier Grade” Customer Expectations
• Software Upgrades
• Millions of Devices
• Cloud heterogeneity is important
• 1000’s of VMs running at any time, across
all clouds
Continuous Delivery @Comcast
3

4. In the past… Many humans, some machines MUCH time
Continouos Delivery @Comcast
4

5. Now: One Human, many machines much LESS time
Continouos Delivery @Comcast
5

6. Let’s go FAST
 Market was clearly changing, with competition delivering some cool stuff.
 Time to market became a stronger driver
 We did some light reading:
6

7. And then drank gallons of Continuous Delivery
7

8. And then… Epiphany!
8
Immutable
Servers!

9. We arrived at a system that consisted of:
9
Puppet
App
Config
Hiera
App
Automation
Openstack
VMWare
Route53
Git
Cloud-init

10. Which can also be viewed like this…
10

11. Feedback from initial attempts at Continuous Delivery
• It needs a GUI
• Why is it so hard?
• What are all those moving parts there? WTF?!
• write the GUI with whatever you want!
11

12. 12
Gumby!
Your elastic friend in cloud

13. V1 : Play + Akka in Scala
• We made an app, in Play! GUI was in Angular, REST layer -> Biz Layer in
Play/Akka (one dude want to learn Akka/Play/Scala, other dude wanted to
learn AngularJS). Gumby was born!
• It was a great experiment, it worked in production, and actually solved a real
problem(s)
13

14. Gumby: Spray + Akka to scale the Comcast Cloud
• Then.. The folks that that make X1 heard about this “gumby” thing and told us
they wanted to use it.. That’s great, but it was only a science experiment….
• Time to rewrite in Spray, and Go Big
• Spray allowed up to really clean things up, and because it was so fast, we
could DOS any cloud api we encontered ;)
14

15. Gumby: Current Architecture (Monolithic)
15
Workflow
(Akka)
VSphere
Openstack
EC2
REST(Spray)Client

16. Fun Facts about Gumby
• Can deploy ~400 vms per dc / hour
• Currently about 21000 lines of scala
• Typically does 2-3 full national deploys/week
• Deploys 60% of national X1 backend footprint
(4000 VMs)
• Will deploy 100% of national X1 backend footprint
by Q1 2016
• Is transitioning to microservices
• Can deploy itself
• David Bolene: “One Big Side Effect”
16