Dibi Conference 2012

Just Build It! Scott Rutherford, DIBI 2012

A Quick Question

• Axomic (OpenAsset)• Caron Software• Caron Records / Publishing• Yabb / Cominded• ThatsUseful (BuddyMap, Boovu, et al.)• RecommendBox• Njoyed• SlimTimer

Market leading customer feedback &support solution21 strong in San Francisco & Raleigh90,000+ organizations have usedus$2M+ raised over 2 fundingroundsTens of millions ofregistered users182 people/sec see ourproduct

Failing Up

Happen’s To The Best• Facebook Beacon (Feature)• Google Wave (Product)• Twitter - Scaling (Technology)• Coke - New Coke! (Marketing)

“were not that dumb, and were not thatsmart!” - Don Keough

Shit Happens!

You Can’t Plan For Everything!Things are going to go wrong. It’s the way we react at these moments that deﬁnes to a great degree how successful we will be. Don’t spend time planning for everyeventuality. Just be prepared to move quickly and decisively when situations arise. Don’t build failure in though!!!

Just Build It!

ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code

ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code >100K Installs > 750K Visitors

Version 1.0• Rails 1.2.3 / HopToad / NewRelic / CloudFront• Engine Yard • Shared MySQL database • 2 slices (and staging)• Admin section based on ActiveScaffold• No billing ability until the end of 2008 (then using Freemium OSS)

Building Fast• Created our own problems• Insufﬁcient test coverage and no CI

Building Fast• Created our own problems• Insufﬁcient test coverage and no CI • deploy production:rollback• Watch those AJAX calls

Building Fast• Created our own problems• Insufﬁcient test coverage and no CI • deploy production:rollback• Watch those AJAX calls• Be careful when testing.....

Stunning Timing

Stunning Timing April 2009 Raised $800k

Version 2.0

Features, Features, Features • Reporting / Administration Tools • Multiple Languages (now crowd sourced) • Search that worked (Sphinx) • Single Sign On • API • Billing (constant PITA)

Growing....• 15 - 20% month• 6 web boxes• 2 databases (master / slave)• Staging• Admin• Ofﬁce!

Problems• Shared database started to creak• Tables growing in size - queries not scaling• Overused MySQL - queue, stats• Not enough caching• Dark corners - data exports, email notiﬁcations• Still insufﬁcient test coverage and no CI• Us...... (production deploy:rollback)

“Zee Germans”

Into The Cloud....• Moved servers to SoftLayer• Passenger and Ruby Enterprise• Signed up 4 million new users in 2 hours• 6 slices expanded to 24• Stability resumed.......

Still Shaky• Redis / Resque for the queue• Redis for stats• More memcached• Make as many things background jobs as possible• Things are under control• Or so we thought.....

Meltdown• 10 days of almost stability• Flew to SXSW• MySQL melted (still on 5.0)• Percona rescued us from the dreaded MYSQL AUTO INCREMENT BUG (Moved us to 5.1)• Finally have a scalable redundant system, right?

DNS can fail?• Using ZoneEdit (free)• All nameservers went down, all hell broke loose• Contacted by DynECT over Twitter!• Gave us a free account to get going again. Moved to them• Stopped thinking about DNS!

Retention / Metrics 91% :( 10 months

What now?• Ongoing monitoring - daily slice stored in Mongo• Surveys at exit• Addition of HelpDesk & Knowledge Base• Integrations• API, Facebook, iOS

“Zee Germans”

And back to Earth with a bump• Softlayer had a few major outages on their cloud infrastructure• Slices are still shared resources• You can’t get someone to just reboot the box• Lesson: cloud is great to scale with, not so good for stability

So where are we now? • 20 systems in production (2 cloud instances) • Web - 6x dual cpu/quad core 12G RAM nginx/passenger, memcache • Resque - 3x single cpu/quad core 8gb RAM • Redis - Dual CPU/quad core 12g RAM • Mongo - 2x dual cpu/hex core 48G RAM 12x 15k RPM SAS drives RAID10 • MySQL - 2x dual cpu/hex core 72G RAM 24x 15k RPM SAS drives RAID10 • 97% month / month retention

What have we learned?

What have we learned? if anything.....

Performance Realization > 200ms stuffed

More Gems

Mouth / UGC Styles

The right tool!

Databases• Redis - Queue, Stats / Admin• Mongo - Stats / Mouth, PageViews, Financials, Retention• MySQL - Main data store• Sphinx - Search• Memcached - Caching

Libraries• jQuery• Backbone• D3• SVG• MooTools Server• SASS• Compass

Software• DSpam• Chef• Ganglia• Nagios• Capistrano• Passenger• Nginx / HAProxy

External Services• Mailgun • Akamai• Embedly • Akismet• Pusher App • Dynect• Airbrake • SoftLayer• S3 • GitHub

Wish We’d Had

K.I.S.S 200 moving parts to change a lightbulb.....

Localize

Unsinkable?

Stealth mode is for..

Listen, no really... "If I had asked people what they wanted, they would have said faster horses." Attributed to Henry Ford

Our Creations• Resque Sliders - https://github.com/ kmullin/resque-sliders• Serve - http://get-serve.com/• Mouth - https://github.com/cypriss/ mouth• UGC Styles - https://github.com/ joshuarudd/ugc-styles• ActiveScaffold - https://github.com/ activescaffold/active_scaffold

Just Build It!The best time to start is NOW!!

Scott Rutherford @scottrutherfordscott@uservoice.com

http://www.scoop.it	392
http://lanyrd.com	51
http://dannybloomfield.tumblr.com	4
https://si0.twimg.com	2
http://us-w1.rockmelt.com	2
https://twimg0-a.akamaihd.net	1
http://twitter.com	1

Dibi Conference 2012

by Scott Rutherford on Apr 25, 2012

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 453

Statistics

Dibi Conference 2012 Presentation Transcript