Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dibi Conference 2012

2,967 views

Published on

My talk from the DIBI conference in Newcastle in April 2012

Published in: Technology
  • Be the first to comment

Dibi Conference 2012

  1. Just Build It! Scott Rutherford, DIBI 2012
  2. A Quick Question
  3. CERN
  4. • Axomic (OpenAsset)• Caron Software• Caron Records / Publishing• Yabb / Cominded• ThatsUseful (BuddyMap, Boovu, et al.)• RecommendBox• Njoyed• SlimTimer
  5. Market leading customer feedback &support solution21 strong in San Francisco & Raleigh90,000+ organizations have usedus$2M+ raised over 2 fundingroundsTens of millions ofregistered users182 people/sec see ourproduct
  6. Failing Up
  7. Happen’s To The Best• Facebook Beacon (Feature)• Google Wave (Product)• Twitter - Scaling (Technology)• Coke - New Coke! (Marketing)
  8. “were not that dumb, and were not thatsmart!” - Don Keough
  9. Shit Happens!
  10. You Can’t Plan For Everything!Things are going to go wrong. It’s the way we react at these moments that defines to a great degree how successful we will be. Don’t spend time planning for everyeventuality. Just be prepared to move quickly and decisively when situations arise. Don’t build failure in though!!!
  11. Just Build It!
  12. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code
  13. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code
  14. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code
  15. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code >100K Installs > 750K Visitors
  16. Version 1.0• Rails 1.2.3 / HopToad / NewRelic / CloudFront• Engine Yard • Shared MySQL database • 2 slices (and staging)• Admin section based on ActiveScaffold• No billing ability until the end of 2008 (then using Freemium OSS)
  17. Building Fast• Created our own problems• Insufficient test coverage and no CI
  18. Building Fast• Created our own problems• Insufficient test coverage and no CI • deploy production:rollback• Watch those AJAX calls
  19. Building Fast• Created our own problems• Insufficient test coverage and no CI • deploy production:rollback• Watch those AJAX calls• Be careful when testing.....
  20. Building Fast• Created our own problems• Insufficient test coverage and no CI • deploy production:rollback• Watch those AJAX calls• Be careful when testing.....
  21. Stunning Timing
  22. Stunning Timing April 2009 Raised $800k
  23. Version 2.0
  24. Features, Features, Features • Reporting / Administration Tools • Multiple Languages (now crowd sourced) • Search that worked (Sphinx) • Single Sign On • API • Billing (constant PITA)
  25. Growing....• 15 - 20% month• 6 web boxes• 2 databases (master / slave)• Staging• Admin• Office!
  26. Problems• Shared database started to creak• Tables growing in size - queries not scaling• Overused MySQL - queue, stats• Not enough caching• Dark corners - data exports, email notifications• Still insufficient test coverage and no CI• Us...... (production deploy:rollback)
  27. “Zee Germans”
  28. Into The Cloud....• Moved servers to SoftLayer• Passenger and Ruby Enterprise• Signed up 4 million new users in 2 hours• 6 slices expanded to 24• Stability resumed.......
  29. Still Shaky• Redis / Resque for the queue• Redis for stats• More memcached• Make as many things background jobs as possible• Things are under control• Or so we thought.....
  30. Meltdown• 10 days of almost stability• Flew to SXSW• MySQL melted (still on 5.0)• Percona rescued us from the dreaded MYSQL AUTO INCREMENT BUG (Moved us to 5.1)• Finally have a scalable redundant system, right?
  31. DNS can fail?• Using ZoneEdit (free)• All nameservers went down, all hell broke loose• Contacted by DynECT over Twitter!• Gave us a free account to get going again. Moved to them• Stopped thinking about DNS!
  32. Retention / Metrics 91% :( 10 months
  33. What now?• Ongoing monitoring - daily slice stored in Mongo• Surveys at exit• Addition of HelpDesk & Knowledge Base• Integrations• API, Facebook, iOS
  34. “Zee Germans”
  35. And back to Earth with a bump• Softlayer had a few major outages on their cloud infrastructure• Slices are still shared resources• You can’t get someone to just reboot the box• Lesson: cloud is great to scale with, not so good for stability
  36. So where are we now? • 20 systems in production (2 cloud instances) • Web - 6x dual cpu/quad core 12G RAM nginx/passenger, memcache • Resque - 3x single cpu/quad core 8gb RAM • Redis - Dual CPU/quad core 12g RAM • Mongo - 2x dual cpu/hex core 48G RAM 12x 15k RPM SAS drives RAID10 • MySQL - 2x dual cpu/hex core 72G RAM 24x 15k RPM SAS drives RAID10 • 97% month / month retention
  37. What have we learned?
  38. What have we learned? if anything.....
  39. Performance Realization > 200ms stuffed
  40. Gems
  41. More Gems
  42. Mouth / UGC Styles
  43. The right tool!
  44. Databases• Redis - Queue, Stats / Admin• Mongo - Stats / Mouth, PageViews, Financials, Retention• MySQL - Main data store• Sphinx - Search• Memcached - Caching
  45. Libraries• jQuery• Backbone• D3• SVG• MooTools Server• SASS• Compass
  46. Software• DSpam• Chef• Ganglia• Nagios• Capistrano• Passenger• Nginx / HAProxy
  47. External Services• Mailgun • Akamai• Embedly • Akismet• Pusher App • Dynect• Airbrake • SoftLayer• S3 • GitHub
  48. Wish We’d Had
  49. K.I.S.S 200 moving parts to change a lightbulb.....
  50. Localize
  51. Unsinkable?
  52. Stealth mode is for..
  53. Listen, no really... "If I had asked people what they wanted, they would have said faster horses." Attributed to Henry Ford
  54. Our Creations• Resque Sliders - https://github.com/ kmullin/resque-sliders• Serve - http://get-serve.com/• Mouth - https://github.com/cypriss/ mouth• UGC Styles - https://github.com/ joshuarudd/ugc-styles• ActiveScaffold - https://github.com/ activescaffold/active_scaffold
  55. Just Build It!The best time to start is NOW!!
  56. Scott Rutherford @scottrutherfordscott@uservoice.com

×