Dev Ops @ Envato ... or how we found out there’s a name and a supportgroup for all the stuff we already did to let a team ...
John Barton @johnbartonjohn@envato.com
Envato?http://envato.com
Stock Marketplaces         & Tutorial Network
Marketplaces
Like iStockPhoto but for  other creative niches           or  eBay for digital goods
Tuts+ Network
Big Blog Network for   education in creativefields.... and linkbait (ducks)
Big traffic and interesting init’s own right, but that’s for    @ryanfaceryan to talk            about
Why?
2 things recently made us think we might be doingsomething a bit different
Neal Ford @ YOW! “Rails can Scale”
200k+ daily requests         vs 2m+ daily requests
Dev Ops Melbourne“Continuous Deployment”
1 Deployment per week or         fortnight            vs  5 Deployments per Day
The Marketplace
August 2006
One Marketplace
FlashDen
Rails 0.13bNo UsersNo Traffic
March 2011
9 Marketplaces
ActiveDen (nee FlashDen)       AudioJungle      ThemeForest        VideoHive      GraphicRiver      CodeCanyon        3D O...
Rails 2.3.11       687,830 Users20 Million Requests Weekly
One Codebase and OneProduction Environment
... and well gosh darn it we    just deploy to that site whenever we darned well           feel like it
As much a product of timeand circumstances as good decisions and hard work
The “Golden Age”
3 Developers30 minute feedback cycledeploy, discuss on forums,       deploy again
Decision 1:Preserve everything goodabout the the startup days  for as long as we can
Commodity Hosting
Notice I’m not saying“Cloud Computing”
You canttrust “thecloud” but  you cantrust “the cloud to  be “the  cloud”.
Decision 2:  Conservative Platformchoice so we don’t have to     sweat the details
Corporate AntiPatterns
We’ve all been doing dev long enough to see thisstuff screwed up over and         over again
Decision 3:Don’t do all that stuff
So what do we actually do?
“Culture of respect &trust, good attitude toward          failure...”Ted Dzubia
“How about "culture of     stop fucking up"?” Ted Dzubiahttp://teddziuba.com/2011/03/devops-scam.html
Ultimate Responsibility
"The fault, dearBrutus, is not in our    QA or Ops, But in ourselves."
Test Driven Development           vs.        QA Team
Test Driven Infrastructure           vs.        Ops Team
Both as a team and asindividuals we own ourwork from when we are    asked to do it...
... until is is demonstrablyerror free and performant         in production
Everyone is in the (paid) on       call roster
Everyone takes a turn atLevel 2 Customer Support
Want those jobs to be       easier?   Stop fucking up.
Process
LEAN / TPS Principles...without the process
You cannot write code anyfaster than you can deploy      it to production
Long running projects?
A. B. C.Always Be Cmerging  (the c is silent)
Dark LaunchFeature FlagsPrivate Beta
Community
We do trip up running this           fast
But through years of openness with our usersvia our forums and owning     up to our mistakes
... we’ve ended up with a (relatively) sympathetic        community
Time Zones both help and         hurt
Traffic peaks during US day  means that if things gowrong we’re usually asleep
But it makes it very easy todeploy during our business          hours
Dev vs. Ops
Our Solution:Don’t have Ops
Outsource commodity       platform bits: virtualisation/cloud, haverackspace take care of db/         mailserver
Ensure the dev team hasthe skills to take care of the              rest
Keep the stack as Vanilla as         possible
Virtualised servers in our      own sandbox.Cloud Flexibility - Cloud   Shit-ness = WIN
Automate Configuration    Management
Don’t modify live boxes,tear down and build again      from scratch
Take advantage of an  individuals talents, but   don’t rely upon themie. don’t accidentally create an ops guy
Performance & Scaling
Not as big a deal as everyone thinks
Shared-nothing loadbalanced app servers + outof request queue workers   not rocket surgery
Measure, deploy, measureagain and then tweak or        rollback    New Relic FTW
There is no code faster    than no code.
Caveats
We rely on our growthfollowing a steady exponent   for this approach to be             viable
We made a conscious tradeoff between deliverytime and production snafus
We’ve traded off platform innovation for product       innovation   (no MongoDB, etc)
Auditors aren’t big fans ofone team with access to everything, but you can mitigate these problems
Question?
Dev Ops @ Envato
Dev Ops @ Envato
Dev Ops @ Envato
Dev Ops @ Envato
Upcoming SlideShare
Loading in...5
×

Dev Ops @ Envato

3,565

Published on

My presentation from the Melbourne Dev Ops meetup on 22nd March 2011 on how we structure our team, deployments, and infrastructure.

Published in: Technology
2 Comments
5 Likes
Statistics
Notes
No Downloads
Views
Total Views
3,565
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
2
Likes
5
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Dev Ops @ Envato

    1. 1. Dev Ops @ Envato ... or how we found out there’s a name and a supportgroup for all the stuff we already did to let a team of 8deploy heaps of times a day to a Ruby on Rails app that has scaled up to around 20 million requests a week without an ops team.
    2. 2. John Barton @johnbartonjohn@envato.com
    3. 3. Envato?http://envato.com
    4. 4. Stock Marketplaces & Tutorial Network
    5. 5. Marketplaces
    6. 6. Like iStockPhoto but for other creative niches or eBay for digital goods
    7. 7. Tuts+ Network
    8. 8. Big Blog Network for education in creativefields.... and linkbait (ducks)
    9. 9. Big traffic and interesting init’s own right, but that’s for @ryanfaceryan to talk about
    10. 10. Why?
    11. 11. 2 things recently made us think we might be doingsomething a bit different
    12. 12. Neal Ford @ YOW! “Rails can Scale”
    13. 13. 200k+ daily requests vs 2m+ daily requests
    14. 14. Dev Ops Melbourne“Continuous Deployment”
    15. 15. 1 Deployment per week or fortnight vs 5 Deployments per Day
    16. 16. The Marketplace
    17. 17. August 2006
    18. 18. One Marketplace
    19. 19. FlashDen
    20. 20. Rails 0.13bNo UsersNo Traffic
    21. 21. March 2011
    22. 22. 9 Marketplaces
    23. 23. ActiveDen (nee FlashDen) AudioJungle ThemeForest VideoHive GraphicRiver CodeCanyon 3D Ocean Tuts+ Marketplace PhotoDune
    24. 24. Rails 2.3.11 687,830 Users20 Million Requests Weekly
    25. 25. One Codebase and OneProduction Environment
    26. 26. ... and well gosh darn it we just deploy to that site whenever we darned well feel like it
    27. 27. As much a product of timeand circumstances as good decisions and hard work
    28. 28. The “Golden Age”
    29. 29. 3 Developers30 minute feedback cycledeploy, discuss on forums, deploy again
    30. 30. Decision 1:Preserve everything goodabout the the startup days for as long as we can
    31. 31. Commodity Hosting
    32. 32. Notice I’m not saying“Cloud Computing”
    33. 33. You canttrust “thecloud” but you cantrust “the cloud to be “the cloud”.
    34. 34. Decision 2: Conservative Platformchoice so we don’t have to sweat the details
    35. 35. Corporate AntiPatterns
    36. 36. We’ve all been doing dev long enough to see thisstuff screwed up over and over again
    37. 37. Decision 3:Don’t do all that stuff
    38. 38. So what do we actually do?
    39. 39. “Culture of respect &trust, good attitude toward failure...”Ted Dzubia
    40. 40. “How about "culture of stop fucking up"?” Ted Dzubiahttp://teddziuba.com/2011/03/devops-scam.html
    41. 41. Ultimate Responsibility
    42. 42. "The fault, dearBrutus, is not in our QA or Ops, But in ourselves."
    43. 43. Test Driven Development vs. QA Team
    44. 44. Test Driven Infrastructure vs. Ops Team
    45. 45. Both as a team and asindividuals we own ourwork from when we are asked to do it...
    46. 46. ... until is is demonstrablyerror free and performant in production
    47. 47. Everyone is in the (paid) on call roster
    48. 48. Everyone takes a turn atLevel 2 Customer Support
    49. 49. Want those jobs to be easier? Stop fucking up.
    50. 50. Process
    51. 51. LEAN / TPS Principles...without the process
    52. 52. You cannot write code anyfaster than you can deploy it to production
    53. 53. Long running projects?
    54. 54. A. B. C.Always Be Cmerging (the c is silent)
    55. 55. Dark LaunchFeature FlagsPrivate Beta
    56. 56. Community
    57. 57. We do trip up running this fast
    58. 58. But through years of openness with our usersvia our forums and owning up to our mistakes
    59. 59. ... we’ve ended up with a (relatively) sympathetic community
    60. 60. Time Zones both help and hurt
    61. 61. Traffic peaks during US day means that if things gowrong we’re usually asleep
    62. 62. But it makes it very easy todeploy during our business hours
    63. 63. Dev vs. Ops
    64. 64. Our Solution:Don’t have Ops
    65. 65. Outsource commodity platform bits: virtualisation/cloud, haverackspace take care of db/ mailserver
    66. 66. Ensure the dev team hasthe skills to take care of the rest
    67. 67. Keep the stack as Vanilla as possible
    68. 68. Virtualised servers in our own sandbox.Cloud Flexibility - Cloud Shit-ness = WIN
    69. 69. Automate Configuration Management
    70. 70. Don’t modify live boxes,tear down and build again from scratch
    71. 71. Take advantage of an individuals talents, but don’t rely upon themie. don’t accidentally create an ops guy
    72. 72. Performance & Scaling
    73. 73. Not as big a deal as everyone thinks
    74. 74. Shared-nothing loadbalanced app servers + outof request queue workers not rocket surgery
    75. 75. Measure, deploy, measureagain and then tweak or rollback New Relic FTW
    76. 76. There is no code faster than no code.
    77. 77. Caveats
    78. 78. We rely on our growthfollowing a steady exponent for this approach to be viable
    79. 79. We made a conscious tradeoff between deliverytime and production snafus
    80. 80. We’ve traded off platform innovation for product innovation (no MongoDB, etc)
    81. 81. Auditors aren’t big fans ofone team with access to everything, but you can mitigate these problems
    82. 82. Question?

    ×