The document summarizes the speaker's experience with developing and improving the architecture for a Ruby on Rails application hosted on a customer's VMware infrastructure. It describes initial flaws with logging and database issues. It then discusses efforts to improve the architecture using tools like Chef but facing challenges, and moving to a model of customizing VMs using Capistrano scripts. Key lessons learned include the importance of monitoring performance, being aware of virtualization impacts, and hiring strong technical staff.

1. DevOps Down Under 2011 Sprinkling DevOps Magic in Other People's Environments Robert Postill

2. How's This Gonna Go Down?

3. How's This Gonna Go Down? Everybody's got a story and this is our's

4. Our first architecture

5. Learning from failure

6. A brief aside

7. Getting better

8. Tough messages

9. Where to from here

10. C3's Story There was a dream

11. It makes the excel go into the data warehouse

12. And it's done badly

13. So we built a prototype

14. Then we made a sale

15. A little bit of how it works

16. Priorities of our first architecture Works!

17. Restarts when the machine restarts

18. Remotely deploy updates

19. Not a lot of state on the VM

20. Our first architecture

21. Our first architecture

22. Lesson: Most customers will accept a small selection of services if you give them a report from that service

23. create_deployment.sh Poor man's capistrano

24. A shell script that: Fetched the latest from github

25. Exported it to a datestamped directory

26. Made a set of symlinks point to the right places

27. Restarted the app

28. Flaws We knew practically nothing about what was happening on the box

29. The logs... THE LOGS FIX THOSE FREAKING LOGS!!!

30. And the worst flaw of all... We started to get calls that started with: “ Integrity’s down, what's the score?” Then we'd have a look...

31. And it would be the database

32. Lesson: Things you don't own go badly wrong and the first people to know are the end users

33. A lot of sad face

34. So we revved the architecture

35. Then more stuff happened... We continued to get calls that started with: “ Integrity’s down, what's the score?” Then we'd have a look...

36. And it would be the VM, mounted disks read-only

37. Lesson: Virtual Machines are prone to at least a couple of novel modes of failure

38. Which started to lead to the inevitable

39. So the next problem... Us New Relic gives you slow transaction reports

40. In ruby select, collect and friends are ways of making in memory decisions over collections of things

41. Which works on test set sizes of ten or so

42. But doesn't on large volumes of things, like say a couple of million objects