My presentation from the Melbourne Dev Ops meetup on 22nd March 2011 on how we structure our team, deployments, and infrastructure.

1. Dev Ops @ Envato
... or how we found out there’s a name and a support
group for all the stuff we already did to let a team of 8
deploy heaps of times a day to a Ruby on Rails app that
has scaled up to around 20 million requests a week
without an ops team.

2. John Barton
@johnbarton
john@envato.com

3. Envato?
http://envato.com

4. Stock Marketplaces
&
Tutorial Network

5. Marketplaces

6. Like iStockPhoto but for
other creative niches
or
eBay for digital goods

7. Tuts+ Network

8. Big Blog Network for
education in creative
fields.... and linkbait (ducks)

9. Big traffic and interesting in
it’s own right, but that’s for
@ryanfaceryan to talk
about

10. Why?

11. 2 things recently made us
think we might be doing
something a bit different

12. Neal Ford @ YOW!
“Rails can Scale”

13. 200k+ daily requests
vs
2m+ daily requests

14. Dev Ops Melbourne
“Continuous Deployment”

15. 1 Deployment per week or
fortnight
vs
5 Deployments per Day

16. The Marketplace

17. August 2006

18. One Marketplace

19. FlashDen

20. Rails 0.13b
No Users
No Traffic

21. March 2011

22. 9 Marketplaces

23. ActiveDen (nee FlashDen)
AudioJungle
ThemeForest
VideoHive
GraphicRiver
CodeCanyon
3D Ocean
Tuts+ Marketplace
PhotoDune

24. Rails 2.3.11
687,830 Users
20 Million Requests Weekly

25. One Codebase and One
Production Environment

26. ... and well gosh darn it we
just deploy to that site
whenever we darned well
feel like it

27. As much a product of time
and circumstances as good
decisions and hard work

28. The “Golden Age”

30. 3 Developers
30 minute feedback cycle
deploy, discuss on forums,
deploy again

31. Decision 1:
Preserve everything good
about the the startup days
for as long as we can

32. Commodity Hosting

33. Notice I’m not saying
“Cloud Computing”

36. You can't
trust “the
cloud” but
you can
trust “the
cloud to
be “the
cloud”.

37. Decision 2:
Conservative Platform
choice so we don’t have to
sweat the details

38. Corporate AntiPatterns

39. We’ve all been doing dev
long enough to see this
stuff screwed up over and
over again

40. Decision 3:
Don’t do all that stuff

41. So what do we actually do?

42. “Culture of respect &
trust, good attitude toward
failure...”
Ted Dzubia

43. “How about "culture of
stop fucking up"?”
Ted Dzubia
http://teddziuba.com/2011/03/devops-scam.html

44. Ultimate Responsibility

45. "The fault, dear
Brutus, is not in our
QA or Ops,
But in ourselves."

46. Test Driven Development
vs.
QA Team

47. Test Driven Infrastructure
vs.
Ops Team

48. Both as a team and as
individuals we own our
work from when we are
asked to do it...

49. ... until is is demonstrably
error free and performant
in production

50. Everyone is in the (paid) on
call roster

51. Everyone takes a turn at
Level 2 Customer Support

52. Want those jobs to be
easier?
Stop fucking up.

53. Process

54. LEAN / TPS Principles
...without the process

56. You cannot write code any
faster than you can deploy
it to production

57. Long running projects?

58. A. B. C.
Always Be Cmerging
(the c is silent)

59. Dark Launch
Feature Flags
Private Beta

60. Community

61. We do trip up running this
fast

62. But through years of
openness with our users
via our forums and owning
up to our mistakes

63. ... we’ve ended up with a
(relatively) sympathetic
community

64. Time Zones both help and
hurt

65. Traffic peaks during US day
means that if things go
wrong we’re usually asleep

66. But it makes it very easy to
deploy during our business
hours

67. Dev vs. Ops

68. Our Solution:
Don’t have Ops

69. Outsource commodity
platform bits:
virtualisation/cloud, have
rackspace take care of db/
mailserver

70. Ensure the dev team has
the skills to take care of the
rest

71. Keep the stack as Vanilla as
possible

72. Virtualised servers in our
own sandbox.
Cloud Flexibility - Cloud
Shit-ness = WIN

73. Automate Configuration
Management

74. Don’t modify live boxes,
tear down and build again
from scratch

75. Take advantage of an
individual's talents, but
don’t rely upon them
ie. don’t accidentally create an ops guy

76. Performance & Scaling

77. Not as big a deal as
everyone thinks

78. Shared-nothing load
balanced app servers + out
of request queue workers
not rocket surgery

79. Measure, deploy, measure
again and then tweak or
rollback
New Relic FTW

80. There is no code faster
than no code.

81. Caveats

82. We rely on our growth
following a steady exponent
for this approach to be
viable

83. We made a conscious
tradeoff between delivery
time and production snafus

84. We’ve traded off platform
innovation for product
innovation
(no MongoDB, etc)

85. Auditors aren’t big fans of
one team with access to
everything, but you can
mitigate these problems

86. Question?