Sept 17, 2009Migrating Jive to the Cloud:Practical Tips and TricksMatt TuckerCTO, Jive Software

Jive OverviewFounded in 2001 – Series A with Sequoia in Fall ’07Growing revenue 100% Y/Y and generating cashIndustry analyst recognition as leader in our spaceOnly vendor to bridge external and internal communitiesMore than 2,500 customersOperations in 5 countries, new office in Palo AltoOver 150 employees (currently hiring +20)

Jive’s Cloud EvolutionSummer of 2008All deployments were installed software or ASP managed hostingCustomers lacked easy, cheap way to start with Social Business SoftwareEvaluated and selected Amazon EC2 environmentOctober 2008 – February 2009Dedicated, skunk-works team devoted to migrating Jive’s offering to the cloudFirst instances up and running in January 2009 – 3 months total developmentSummer 2009Over 250 Customers on Jive ExpressAll customer sandbox sites migrated to the EC2 cloudCosts to run a Jive Express environment are 1/10 the cost of ASPIn process of launching additional products via the cloud

Advantages for Jive’s Cloud OfferingASP takes too long and costs too much6 weeks to procure and install new serversApproximately $20K per installation, all up-front and no linear rampNo ability to turn off or manage capacityEasier to manageMinutes, not days or months to get up and runningInstances spin up and down automaticallyFail-over happens with admin tools in the backgroundVastly lower operational expense due to automationEasy deployment for customersTools to manage and track adoptionCustomized wizards guide through typical use casesCustomers can migrate from EC2 to ASP or their own SW deployments

The Cloud & EnterpriseEnterprise Readiness IssuesNo SAS70 Type II certification for AWSNeed to improve SLA for high end customers. On Jive side as well as AWSEnterprise security reviews have not caught up with the cloud yet. Standard evaluation criteria still focuses on things like hardware vs. virtualization, data center tours, etcRamificationsCloud is generally for smaller or “starter” Jive implementations; low percentage of revenue but gets us into larger dealsWe’ve built an easy migration path to on-prem or ASPAt least 2 years away from widespread enterprise cloud readiness, but trend is happening

PaaS or IaaS?

Key Technical ChallengesBring multi-tenant cost efficiency to a single-tenant appJive is a “fat” application. How do we fit in a small EC2 instance? Cut down app startup time from 10 mins to 2 mins, use small Java heapAny customizations break easy/automated upgradingBuilt new simplified admin console and did other simplifications via product overlayMust eliminate per-instance manual laborInvested in radical level of automation that maintains the environment with very little manual intervention

Cut down app startup time from 10 mins to 2 mins, use small Java heap

Built new simplified admin console and did other simplifications via product overlay

Invested in radical level of automation that maintains the environment with very little manual intervention

Architecture OverviewS3EC2 InstancesEBSXMPPController ServiceRedirect ServiceProvisioning SiteSQS

Trick: Scripting Java InstallBasic tip: fully script the creation of your AMI!Ran into problem that install of Java can’t be automated# Install Sun JDK (messing with whiptail to avoid license prompt)mv /usr/bin/whiptail /usr/bin/whiptail.origcat > /usr/bin/whiptail <<EOMexit 0EOMchmod +x /usr/bin/whiptailapt-get install -y sun-java6-jdkrm /usr/bin/whiptailmv /usr/bin/whiptail.orig /usr/bin/whiptailexport JAVA_HOME=/usr/lib/jvm/java-6-sunrm /usr/bin/javaln -s $JAVA_HOME/bin/java /usr/bin/java

Trick: HibernationFurther cut costs by automatically turning off instances that don’t get active useTrick is to use DNS redirect so that they can be turned back on within minutes via self-serviceRedirect ServiceStale EC2 InstanceRedirect DNSSet TTL to 60sHibernateNew EC2 InstanceRedirect ServiceRedirect DNSSet Normal TTLRe-Awaken

Trick: UpgradesThe trick: use elastic compute to do things you hadn’t imagined previouslyIn ASP environment upgrades are run in-place and manually; requires multiple hours of scheduled downtime in case something goes wrongAt EC2 we upgrade “alongside” rather than in-placeUpgrades at EC2 are fully automated and performed en-masseHave achieved low 2% failure rate (fix generally only requires minor intervention)

Trick: UpgradesHow upgrades are done:Make an instance read-only by putting up an upgrade messageTake an EBS snapshot of instance dataCreate a NEW instance with NEW EBS volume from snapshotRun upgrade on new instance using scriptsRun tests to ensure upgrade workedChange elastic IP from old to new instanceDelete old EC2 instance and EBS volumeIf any step fails, remove maintenance message on existing instance and log error message. Failed attempts only cost $0.10

Trick: XMPPSQS is fantastic for asynchronous message processing; we use it to deliver things like hourly stats. But doesn’t solve all problemsUse XMPP for real-time controller to instance communicationEnables multi-step synchronous actions like creating a downloadable data backupSimpler and faster development than complicated web services

Tip: Reserved InstancesLower costs by >= 30% -- purchase reserved instancesUpdated provisioning code to ensure that we always use an availability zone that has reserved instances first

Tip: Retry AWS CallsWe’ve found that 2-5% of AWS web services calls failWork-around by adding re-try logic to critical code paths; retry of major functional actions has been easier than re-try of individual AWS calls (i.e., retry everything that goes into creating new instance and include robust cleanup code)Added reporting to track all “orphaned” resources for edge cases where cleanup isn’t perfect

Tip: Use UserdataPossible to pass in dynamic data to instance when booting as userdataUserdata has small size limit so we securely download full startup script from S3 then execute it$ export INSTANCESTARTUP_VERSION=instanceStartup-1.0.1.sh$ /usr/local/jive/bin/s3-curl/s3curl.pl --id $AWS_ACCESS_ID --key $AWS_SECRET_KEY -- -f --retry 5 --connect-timeout 10 -y 10 http://xxx.s3.amazonaws.com/$INSTANCESTARTUP_VERSION > $INSTANCESTARTUP_VERSION$ chmod +x $INSTANCESTARTUP_VERSION$ ./$INSTANCESTARTUP_VERSION

Tip: Handling EmailSending email from EC2 doesn’t work: reverse DNS won’t resolve it needs, big providers simply mark all of EC2 as SPAMSolution: relay mail to external server at trusted IP address. We use same infrastructure that ASP environment does. Large amount of email being sent = high sender scoreAlso, check out Sendgrid (http://www.sendgrid.com)

Demo: Backend Tools

Q&AMatt Tuckermatt@jivesoftware.com@matttucker