Downloaded 116 times
Uploaded byAndreas Grabner
DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
DevOps aims to improve collaboration between development and operations teams. When these teams are disconnected, it can lead to severe production issues from a lack of shared understanding and tools. Organizations should strive for agile deployments by implementing continuous integration and deployment pipelines. Pushing changes to production without proper planning, such as failing to optimize critical pages, can also cause problems. Establishing metrics and monitoring across the entire software lifecycle helps align teams and catch issues early.
More Related Content
Similar to DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
More from Andreas Grabner
DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
- 1. DevOps and Performance Why, How& Best Practices @grabnerandi http://apmblog.compuware.com
- 3. What you mayhave heard about Austrians
- 4. And just veryrecently @ Euro Song Contest
- 5. How we wouldlike the world to see us
- 6. What we arealso proud of
- 7. What you shouldcheck out …
- 9. The stuff wedid when we were a Start Up and we All were Devs, Testers and Ops
- 11. YOU ARE NOTALONE: Popularity on Google
- 12. Who is doingit? How many successful deployments can they do? 300 Deployments / Year 50-60 Deployments / Day 10+ Deployments / Day Every 11.6 seconds
- 13. More on AmazonsStory 75% fewer outages since 2006 90% fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automatic rollback Deploying every 11.6s
- 18. Testing is Important– and gives Confidence
- 19. But are weready for “The Real” world?
- 20. Measure Performance duringthe game Ball Possession: 40 : 60 Fouls: 0 : 0 Score: 0 : 0 Minute 1 - 5
- 21. Measure Performance duringthe game Minute 6 - 35 Ball Possession: 80 : 20 Fouls: 2 : 12 Score: 0 : 0
- 22.
- 23. Options “To Fix”the situation
- 24. Not always ahappy ending Minute 90 Ball Possession: 80 : 20 Fouls: 4 : 25 Score: 3 : 0
- 25.
- 26. How does that relateto Software?
- 27. From Deploy to… Deploy Promotion/Event Problems Ops Playbook War Room Timeline
- 28. The “War Room”– back then 'Houston, we have a problem‘ NASA Mission Control Center, Apollo 13, 1970
- 29. The “War Room”– NOW Facebook – December 2012
- 30. 3 Situations on WHYthis happens, HOW to avoid it
- 31. Image taken fromhttps://www.scriptrock.com/blog/devops-whats-hype-about/
- 32.
- 33. “Teamwork” between Devand Ops SEV1 Problem in Production Need access to log files Where are they? Can’t get them Need to increase log level Can’t do! Can’t change config files in prod!
- 34. “Solution”: Implement aCustom “On Demand” Remote Logger
- 35. Implementation and Rollout Implemented CustomLogger Worked well in Load Testing
- 36. What happened? ~ 1MioLock Exceptions in 30 mins
- 37. Root Cause: Aspecial WebSphere Setting! Log Service provides a synchronized log file across ALL JVMs Log Service provides a synchronized log file across ALL JVMs
- 38. Metrics: # LogMessages, # Exceptions Share: Same Server Settings DevOps: Agree on Data for Troubleshooting
- 40.
- 41. Adonair Load Spike resultedin Unavailability
- 42. Alternative: “GoDaddy goesDevOps” 1h before SuperBowl KickOff 1h after Game ended
- 43.
- 44. Metrics: Availability Page Size,# Objects # Hosts, # Connections DevOps: “Feature” Switches
- 46.
- 47. Mobile Landing Pageof Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
- 48. m.store.com redirects towww.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections
- 49. Critical Pages notOptimized! Browse, Search and Product Info performs well … because they don’t follow best practices: 87 Requests, 28 Redirects, … Critical Pages such as Shopping Cart are very slow …
- 50. Metrics: Load Time, #Resources (Images, …), # HTTP 3xx, 4xx, 5xx Dev: Build for Mobile Test: Test on Mobile Ops: Monitor Mobile
- 53. # of Requests/ User # of Log Messages # of Exceptions # Objects Allocated # Objects In Cache Cache Hit Ratio # of Images # of SQLs # SQLs per RequestAvailability # HTTP 3xx, 4xx Page Size
- 54.
- 55. Commit Stage • Compile •Execute Unit Test • Code Analysis • Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing • Key showcases • Exploratory testing Release Unit & Integration Tests Functional Tests Performance Tests Production Monitoring Functional Tests
- 57. If we doall that
- 58. Which gives youmore time for the real important things in life …
- 59. Want MORE ofthese and more details? http://apmblog.compuware.com
- 60.
- 61. FREE Products &More Info • dynaTrace Enterprise – Full End-to-End Visibility in your Java, .NET, PHP Apps – Sign up for a 15 Days Free Trial on http://compuwareapm.com • dynaTrace AJAX Edition – Browser Diagnostics for IE + FF – Download @ http://ajax.dynatrace.com • Our Blog: http://apmblog.compuware.com
Editor's Notes
- #15 MEASURE!!
- #16 Who knows what that is? It’s the Fifa World Cup Trophy
- #17 Teams are currently competing in the qualifications to compete in Brazil 2014
- #18 This is “my” austrian national team soccer team. Their GOAL is to qualify for Brazil 2014. After the many failed attempts in the past we hired a new coach who’s goal is to form a new team that PERFORMs good enough to qualify
- #19 In order to get there the team competed in many test games. Which gaves them a lot of confidence because they played against teams that were “easier” to beat. At the end of these tests we even started in the qualification with some wins against teams that we were expecting to win So – at the end of these “test and easy qualification games” we thought: “ALL GOOD – THE ROAD IS OPEN FOR 2014 – NOT ONLY WILL WE QUALIFY BUT WE ALSO BELIEVE WE HAVE SUCH A STRONG TEAM THAT WILL ALSO DO WELL AT THE WORLDCUP”
- #20 Then reality kicked in when we had our first “real competitor” – it was the first qualification against a team whos quality level is at a level that we have to expect at the world cup. The competing team was Germany – and – based on these images you can see how the game went
- #21 The coach is responsible to watch the game and see how things are going. Like in other sports – soccer has a couple of Key Performance Indicators such as Ball Possession, Fouls and the actual score The first 5 minutes actually didn’t look too bad
- #22 After the first 5 minutes the game changes – with germany taking over the game in their typical way. The KPIs make this very clear The coach is responsible to react based on these values and how the game wents
- #23 The coach should use more data for detailed analysis on what is going wrong in the game
- #24 One of his options is to substitute players – or even change tactics Does this succeed based on the KPIs that we have seen before?
- #25 Well – not always. Just replacing players – putting some in that are faster in chasing the ball doesn’t always help
- #28 Story New Build Deployed on Thursday Evening Everything runs smooth on Friday Daytime An Ad Campaign hits the Air Friday Night The site crashes under load -> ALERTS GO OFF Restarting Server -> SERVER DOESN’T START Adding more Servers-> PROBLEM REMAINS Calling in the “App Experts” and Pizza Delivery!
- #32 Well – I guess there is just not more to say about this. The attitude between these teams doesn’t help in solving issues any faster