Rodrigo Campos presented on the challenges faced and benefits of adopting a DevOps culture at Walmart's Latin American e-commerce division. The key challenges included deploying a new platform before Black Friday, expanding infrastructure for traffic spikes, developing an agile mindset, and rebuilding trust between teams. Adopting DevOps practices like improved communication, transparency and collaboration helped address these challenges. It resulted in the successful deployment of the new platform with 100% uptime on Black Friday and increased deployment frequency, success rates and adoption of agile methodologies.

1. Real World DevOps
Rodrigo Campos (@xinu)
Velocity NY 2014

2. Hi I’m Rodrigo!
• Proudly crimping cables since 1992
• System Administrator
• Operations Director at walmart.com in Latin
America

3. I’m not going to talk about
• Silver Bullets
• Unicorns
• Tools and Utilities

4. Instead I’ll focus on
• The challenges we faced
• How DevOps has helped us
• Bumps and bruises along the way

5. Our challenges

6. Our challenges
• Deploy a whole new platform before Black Friday

7. Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand
web traffic spikes

8. Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand
web traffic spikes
• Develop a company-wide agile mindset

9. Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand
web traffic spikes
• Develop a company-wide agile mindset
• Rebuild trust between tech and business

10. Different companies, same
problems

11. Different companies, same
problems
• Silos not only between Developers and
Operations, but within departments as well

12. Different companies, same
problems
• Silos not only between Developers and
Operations, but within departments as well
• BOFH attitude started to kick-in

13. Different companies, same
problems
• Silos not only between Developers and
Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments

14. Different companies, same
problems
• Silos not only between Developers and
Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops

15. Different companies, same
problems
• Silos not only between Developers and
Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops
• Lots of bureaucracy and red-tape

16. Some phrases (roughly
translated)

17. Some phrases (roughly
translated)
• “I'll go to the 9th floor to help you kick everyone’s
a**”

18. Some phrases (roughly
translated)
• “I'll go to the 9th floor to help you kick everyone’s
a**”
• “I don’t care if it’s stable as long as it passed QA
testing”

19. Some phrases (roughly
translated)
• “I'll go to the 9th floor to help you kick everyone’s
a**”
• “I don’t care if it’s stable as long as it passed QA
testing”
• “This is operations’ job”

20. Some phrases (roughly
translated)
• “I'll go to the 9th floor to help you kick everyone’s
a**”
• “I don’t care if it’s stable as long as it passed QA
testing”
• “This is operations’ job”
• “Consider this a favor”

21. How DevOps was seen

22. So what is DevOps anyway?

23. So what is DevOps anyway?
• Being loosely defined can actually be a
good thing!

24. So what is DevOps anyway?
• Being loosely defined can actually be a
good thing!
• For us:
“DevOps is a culture that fosters
communication, transparency and
collaboration"

25. So what is DevOps anyway?

26. So what is DevOps anyway?
• Communication: I’ll stand up and go to your
desk if I need to talk to you, you’re more
than welcome to do the same!

27. So what is DevOps anyway?
• Communication: I’ll stand up and go to your
desk if I need to talk to you, you’re more
than welcome to do the same!
• Transparency: I want to know your
problems, I’ll make sure you know mine!

28. So what is DevOps anyway?
• Communication: I’ll stand up and go to your
desk if I need to talk to you, you’re more
than welcome to do the same!
• Transparency: I want to know your
problems, I’ll make sure you know mine!
• Collaboration: We’re in the same boat and I
really mean it!

29. We needed to change
people's mindset
Operations
Developers
Us Them
InfoSec
Works on my Machine

30. We needed to change
people's mindset
Source: http://youtu.be/Fx8OBeNmaWw

31. We needed to change
people's mindset

32. We needed to change
people's mindset
• Presentations to technology, business and
human capital teams

33. We needed to change
people's mindset
• Presentations to technology, business and
human capital teams
• Reinforcing an inclusive and welcoming
culture was essential

34. We needed to change
people's mindset
• Presentations to technology, business and
human capital teams
• Reinforcing an inclusive and welcoming
culture was essential
• Everyone felt prepared and willing to
change

35. Typical Scenario
Business
Developers
Operations
Infrastructure
{ }
• Things go over the wall
• No alignment between
tech and business
• One-way
communications
• IT as a necessary evil
(aka cost center)

36. We were already a little bit
different…
Business
Developers
Operations
Infrastructure
{ }
DevOps Team

37. 1st step
Business
Developers
Operations
Infrastructure
{ }
Production
Engineering
DevOps

38. 2nd step
Business
Infrastructure
Production
{ }
Developers Management
Production Engineering + { }
DevOps

39. InfoSec

40. InfoSec
• First and foremost: end the dossier culture

41. InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration +
transparency = safer systems

42. InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration +
transparency = safer systems
• Integrate security tests into continuous
integration

43. InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration +
transparency = safer systems
• Integrate security tests into continuous
integration
• Rugged DevOps: http://slidesha.re/1r87Vs7

44. InfoSec
• No more FUD
• What is riskier to the business?
• Lack of speed is a risk too!

45. Compliance
• ITIL: “You keep using that word, I don’t think
it means what you think it means”
• People often blame ITIL as an excuse for
badly designed processes
• Surprisingly ITIL compliance was not our
biggest problem

46. Compliance
• Audits are still our biggest challenge
• In our case, we have a sponsor (the CTO)
• Someone (who happens to be me) will be
held accountable if something goes wrong

47. The bad apples
• We’ve adopted the “No asshole rule"
• There will be obstructive people that won’t
embrace change
• If you can't get rid of them, simply let them
fade into oblivion

48. KPIs

49. KPIs
• Deployment Frequency
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

50. KPIs
• Deployment Frequency
• Speed of Deployment
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

51. KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

52. KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

53. KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
• Culture
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

54. Bumps & Bruises

55. Bumps & Bruises
• DO:

56. Bumps & Bruises
• DO:
• Understand beforehand what you want to
achieve

57. Bumps & Bruises
• DO:
• Understand beforehand what you want to
achieve
• Use measurable KPIs

58. Bumps & Bruises
• DO:
• Understand beforehand what you want to
achieve
• Use measurable KPIs
• Explain to business areas what you’re doing

59. Bumps & Bruises
• DO:
• Understand beforehand what you want to
achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP

60. Bumps & Bruises
• DO:
• Understand beforehand what you want to
achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP
• Work hard on a sponsor

61. Bumps & Bruises

62. Bumps & Bruises
• DON'T:

63. Bumps & Bruises
• DON'T:
• Create a “Ninja Superstar team”

64. Bumps & Bruises
• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication

65. Bumps & Bruises
• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes

66. Bumps & Bruises
• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes
• Underestimate the managers' ability to
influence behavior

67. Keep in mind
• In the beginning it will be a lonely fight
• Some people will label you as the
“hippie-anarchist”

68. Results

69. Results
• New eCommerce platform successfully deployed

70. Results
• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have
100% uptime during Black Friday

71. Results
• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have
100% uptime during Black Friday
• Increase in deployment frequency and success
rate

72. Results
• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have
100% uptime during Black Friday
• Increase in deployment frequency and success
rate
• Business is incorporating agile methodologies

73. Any questions?
• Special thanks to:
https://www.flickr.com/photos/51321237@N07/
https://www.flickr.com/photos/kh-67/
https://www.flickr.com/photos/jrparis/
https://www.flickr.com/photos/andrewdubock/
https://www.flickr.com/photos/rachelyoung/
!
https://bitly.com/bundles/o_4p62vc3lid/5