RightScale at PBS<br />Drew Engelson <email@example.com><br />Chief Architect, PBS Interactive<br />
PBS is…<br />PBS is a national federation of independently owned and operated public television stations and producers<br />Each with their own management and development resources<br />1500+ highly trafficked websites:<br />http://www.pbs.org/<br />http://www.pbs.org/nova/ <br />http://pbskids.org/<br />http://pbskids.org/sesame/<br />http://video.pbs.org/<br />Enterprise services/APIs<br />
Before the cloud<br />Hosting provider for 3rd party developers<br />Shared hosting platform and database cluster<br />Numerous widely varying technologies supported<br />No application isolation<br />Application deployment<br />Difficult QA and deployment (Dev > Staging > Live)<br />Traditional IT<br />Fixed bandwidth<br />Inflexible infrastructure, poorly scalable = degraded uptime<br />Aging operational platform (near EOL)<br />Strategic applications coming down the pipeline<br />Lack of data for forecasting<br />Need financial visibility per application<br />
Our challenge<br />On a limited, shared infrastructure<br />Support diverse technologies<br />Grant developers full access<br />Keep the system running<br />Something’s gotta give!<br />x<br />
Thanks, RightScale<br />Amazon EC2, S3, EBS, ELB, SQS, …<br />180 (+/-) server instances and growing<br />All managed via RightScale Enterprise sub-accounts<br />Standardize baseline configuration<br />Streamlined set of standard server templates<br />LAMP Stacks (EBS, S3), App server, DB server, Cache<br />CentOS 5.2 RightImage<br />Standard + custom RightScripts<br />Application specific configurations<br />Designed for failure, fully repeatable<br />
PBS hosting now<br />Diverse technologies<br />CentOS 5.2 Linux (AWS small)<br />Apache, MySQL, Python, PHP<br />Memcached (use it!)<br />Devs install addl software packages<br />Complete server isolation<br />Automated backups, “Easy” recovery<br />S3 and/or EBS<br />SSH access with sudo<br />Proxy through www.pbs.org<br />Staging and Live<br />✓<br />Developer access<br />System is Up<br />✓<br />✓<br />
In the cloud today<br />
PBS Video (video.pbs.org)<br />Admin<br />EC2/EBS<br />Cache<br />EC2<br />Proxy<br />EC2<br />Proxy<br />EC2<br />Cache ELB<br />Cache<br />EC2<br />App Server ELB<br />DB Master EC2/EBS<br />EBS Snapshots<br />App<br />EC2<br />App<br />EC2<br />App<br />#2<br />App<br />#2<br />App<br />#3<br />App<br />#3<br />App<br />#N<br />App<br />#N<br />DB Slave EC2/EBS<br />US East 1a<br />US East 1d<br />Database ELB<br />DB Slave EC2/EBS<br />S3 backups<br />
Additional uses<br />Software vetting/selection<br />Spin up new servers to test/vet software packages<br />No need to requisition expensive hardware<br />Hosting operational services<br />Documentation wikis (Confluence)<br />Issue tracker (JIRA)<br />Source control management (SVN, Git, Gitorious)<br />Log monitoring (Splunk)<br />
Moving forward<br />Hybrid cloud for development<br />Eucalyptus<br />Build local, deploy to cloud<br />Utilize queuing services (SQS) for:<br />Video transcoding<br />Log processing<br />Amazon Relation Database Service<br />More Ubuntu<br />CentOS packages outdated… lots of upgrading needed<br />
Who?-National membership organizationDecentralized control and governanceAlmost 200 member stations who produce show and sitesAt least one of which is in the audience… Hi Brian!What?-Hosting and management of show-related websitesDevelopment of cross-brand we applicationsPBS.org / PBSKids.orgVideo portals: PBS Video, PBS Kids Go VideoTargeted audience: PBS Parents, PBS TeachersSpecialized services: PBS TeacherLine, PBS Kids IslandDevelopment of shared web services for systemTV Schedules APIs and widgetsSocial media tools: Comments, Bookmarks, DiscussionsContent APIsSingle sign-onAnalyticsComplexity of systemTypical TV network: ~50 programsPBS actively runs ~700 programsCategories we’re using today:60% Scalable website40% Test & developmentDabbling in grid computingGoal is 100% automation of traditional IT functionsLet me tell you why and how…
Our physical data center is cutting edge….If you want to host static HTML with the occasional CGI script here or there.This is what we did quite successfully for many years.Times changed, we did not. Today, our server infrastructure looks much like it did 10 years ago!
So here we’ve found ourselves….With limited, shared infrastructureSupport diverse server technologiesGrant more control to developers-Everyone of these items increases the risk of failureWe are looking at a recipe for disaster.Something’s got to give... And it did.Inevitably we began experiencing degraded performance and availability.
What did we do?AWS EC2 seemed to offer an easily accessible solutionWe considered a number of different management approaches including rolling out own solution.One look at RightScale and we were hooked. We immediately began to slice off small pieces of our infrastructure and move them to AWS via RightScale.Most new applications were built there as well.[Describe slide]
Productizing our cloud hosting product.Evolving following agile methodologies.Every application get’s their own EC2 mini-environment.[Describe slides]Still have a lot of work to do… documentation!
Cross-brand applications: PBS Video, PBS Engage (social media experiments)Targeted audience: PBS TeachersSpecilized services: PBS TeacherLineEnterprise services: Single sign-on, Video APIAn increasing number of show sites. About 29 today.Some receiving extremely high traffic.
Scalable website success story…We were delivered a not so scalable web application.Spend months optimizing? Or release immediately?Auto-scaling arrays bought us the time we needed to launch immediately, and incrementally optimize the application.At times we were running 128+ application servers… Now down to 12 (last I looked).So easy to change or upgrade infrastructure.BTW, we also an duplicate set up for QACoders, who primarily target the App server layer, use identical EC2 servers for development environment.
Cheap way to test/vet new software.
Of our 180+ servers, over half are development or QA. These cost just as much as our live servers. Ideally we’d use RightScale to deploy Dev servers into our locally hosted private cloud where we don’t need to worry about scalability and bandwidth.Currently prototyping integration with EucalyptusLooking to improve other business services…. Our video transcoding service probably can’t compete with a grid based solution. We’re trying to prove that. Some of what I’ve seen today just blows me away.