Talk given at DevOps Meetup in Sydney

1. DevOps and
Performance
Why, How &
Best Practices
@grabnerandi
http://apmblog.compuware.com

3. What you may have heard about Austrians

4. And just very recently @ Euro Song Contest

5. How we would like the world to see us 

6. What we are also proud of 

7. What you should check out …

9. The stuff we did
when we were a Start Up
and we All were
Devs, Testers and Ops

11. YOU ARE NOT ALONE: Popularity on Google

12. Who is doing it? How many successful
deployments can they do?
300 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds

13. More on Amazons Story
75% fewer outages since 2006
90% fewer outage minutes
~0.001% of deployments cause a problem
Instantaneous automatic rollback
Deploying every 11.6s

18. Testing is Important – and gives Confidence

19. But are we ready for “The Real” world?

20. Measure Performance during the game
Ball Possession: 40 : 60
Fouls: 0 : 0
Score: 0 : 0
Minute 1 - 5

21. Measure Performance during the game
Minute 6 - 35
Ball Possession: 80 : 20
Fouls: 2 : 12
Score: 0 : 0

22. Deep Dive Analysis

23. Options “To Fix” the situation

24. Not always a happy ending 
Minute 90
Ball Possession: 80 : 20
Fouls: 4 : 25
Score: 3 : 0

25. FRUSTRATED FANS!!
25

26. How does that
relate to
Software?

27. From Deploy to …
Deploy Promotion/Event Problems Ops Playbook War Room
Timeline

28. The “War Room” – back then
'Houston, we have a problem‘
NASA Mission Control Center, Apollo 13, 1970

29. The “War Room” – NOW
Facebook – December 2012

30. 3 Situations on
WHY this happens,
HOW to avoid it

31. Image taken from https://www.scriptrock.com/blog/devops-whats-hype-about/

32. #Disconnected
Teams

33. “Teamwork” between Dev and Ops
SEV1 Problem in Production
Need access to log files
Where are they? Can’t get them
Need to increase log level
Can’t do! Can’t change config
files in prod!

34. “Solution”: Implement a Custom “On Demand”
Remote Logger

35. Implementation and Rollout
Implemented
Custom Logger
Worked well in
Load Testing

36. What happened?
~ 1Mio Lock Exceptions in 30
mins

37. Root Cause: A special WebSphere Setting!
Log Service provides a synchronized
log file across ALL JVMs
Log Service provides a
synchronized log file across
ALL JVMs

38. Metrics: # Log Messages,
# Exceptions
Share: Same Server
Settings
DevOps: Agree on Data
for Troubleshooting

40. #No “Agile”
Deployment

41. Adonair
Load Spike resulted in Unavailability

42. Alternative: “GoDaddy goes DevOps”
1h before
SuperBowl KickOff
1h after
Game ended

43. Behind the Scenes

44. Metrics: Availability
Page Size, # Objects
# Hosts, # Connections
DevOps: “Feature”
Switches

46. #Push
without a Plan

47. Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:
230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~
20MB

48. m.store.com redirects to www.store.com
ALL CSS and JS files are
redirected to the www domain
This is a lot of time “wasted”
especially on high latency mobile
connections

49. Critical Pages not Optimized!
Browse, Search
and Product Info
performs well
… because they don’t follow
best practices: 87 Requests, 28
Redirects, …
Critical Pages such as
Shopping Cart are very
slow …

50. Metrics: Load Time,
# Resources (Images, …),
# HTTP 3xx, 4xx, 5xx
Dev: Build for Mobile
Test: Test on Mobile
Ops: Monitor Mobile

53. # of Requests / User
# of Log Messages
# of Exceptions
# Objects Allocated
# Objects In Cache
Cache Hit Ratio
# of Images
# of SQLs
# SQLs per RequestAvailability
# HTTP 3xx, 4xx
Page Size

54. 54

55. Commit Stage
• Compile
• Execute Unit Test
• Code Analysis
• Build installers
Automated
Acceptance
Testing
Automated
Capacity
Testing
Manual testing
• Key showcases
• Exploratory testing Release
Unit & Integration Tests
Functional Tests
Performance Tests
Production
Monitoring
Functional Tests

57. If we do all that

58. Which gives you more time for the real important
things in life …

59. Want MORE of these and more details?
http://apmblog.compuware.com

60. Recommended Book
https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/

61. FREE Products & More Info
• dynaTrace Enterprise
– Full End-to-End Visibility in your Java, .NET, PHP Apps
– Sign up for a 15 Days Free Trial on http://compuwareapm.com
• dynaTrace AJAX Edition
– Browser Diagnostics for IE + FF
– Download @ http://ajax.dynatrace.com
• Our Blog: http://apmblog.compuware.com