DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney

  • 4,932 views
Uploaded on

Talk given at DevOps Meetup in Sydney

Talk given at DevOps Meetup in Sydney

More in: Technology , Software
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,932
On Slideshare
4,779
From Embeds
153
Number of Embeds
9

Actions

Shares
Downloads
78
Comments
0
Likes
105

Embeds 153

http://cloud-by.exoplatform.com 101
https://twitter.com 36
http://wp.planview.local 4
http://www.scoop.it 4
http://ans.localhost 3
https://www.linkedin.com 2
http://plus.url.google.com 1
http://www.slideee.com 1
http://translate.googleusercontent.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • MEASURE!!
  • Who knows what that is? <br /> <br /> It’s the Fifa World Cup Trophy <br />
  • Teams are currently competing in the qualifications to compete in Brazil 2014 <br />
  • This is “my” austrian national team soccer team. <br /> <br /> Their GOAL is to qualify for Brazil 2014. After the many failed attempts in the past we hired a new coach who’s goal is to form a new team that PERFORMs good enough to qualify
  • In order to get there the team competed in many test games. Which gaves them a lot of confidence because they played against teams that were “easier” to beat. <br /> <br /> At the end of these tests we even started in the qualification with some wins against teams that we were expecting to win <br /> <br /> So – at the end of these “test and easy qualification games” we thought: “ALL GOOD – THE ROAD IS OPEN FOR 2014 – NOT ONLY WILL WE QUALIFY BUT WE ALSO BELIEVE WE HAVE SUCH A STRONG TEAM THAT WILL ALSO DO WELL AT THE WORLDCUP”
  • Then reality kicked in when we had our first “real competitor” – it was the first qualification against a team whos quality level is at a level that we have to expect at the world cup. <br /> The competing team was Germany – and – based on these images you can see how the game went
  • The coach is responsible to watch the game and see how things are going. <br /> Like in other sports – soccer has a couple of Key Performance Indicators such as Ball Possession, Fouls and the actual score <br /> <br /> The first 5 minutes actually didn’t look too bad
  • After the first 5 minutes the game changes – with germany taking over the game in their typical way. The KPIs make this very clear <br /> <br /> The coach is responsible to react based on these values and how the game wents
  • The coach should use more data for detailed analysis on what is going wrong in the game
  • One of his options is to substitute players – or even change tactics <br /> <br /> Does this succeed based on the KPIs that we have seen before?
  • Well – not always. Just replacing players – putting some in that are faster in chasing the ball doesn’t always help
  • Story <br /> New Build Deployed on Thursday Evening <br /> Everything runs smooth on Friday Daytime <br /> An Ad Campaign hits the Air Friday Night <br /> The site crashes under load -> ALERTS GO OFF <br /> Restarting Server -> SERVER DOESN’T START <br /> Adding more Servers-> PROBLEM REMAINS <br /> Calling in the “App Experts” and Pizza Delivery! <br />
  • Well – I guess there is just not more to say about this. The attitude between these teams doesn’t help in solving issues any faster

Transcript

  • 1. DevOps and Performance Why, How & Best Practices @grabnerandi http://apmblog.compuware.com
  • 2. What you may have heard about Austrians
  • 3. And just very recently @ Euro Song Contest
  • 4. How we would like the world to see us 
  • 5. What we are also proud of 
  • 6. What you should check out …
  • 7. The stuff we did when we were a Start Up and we All were Devs, Testers and Ops
  • 8. YOU ARE NOT ALONE: Popularity on Google
  • 9. Who is doing it? How many successful deployments can they do? 300 Deployments / Year 50-60 Deployments / Day 10+ Deployments / Day Every 11.6 seconds
  • 10. More on Amazons Story 75% fewer outages since 2006 90% fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automatic rollback Deploying every 11.6s
  • 11. Testing is Important – and gives Confidence
  • 12. But are we ready for “The Real” world?
  • 13. Measure Performance during the game Ball Possession: 40 : 60 Fouls: 0 : 0 Score: 0 : 0 Minute 1 - 5
  • 14. Measure Performance during the game Minute 6 - 35 Ball Possession: 80 : 20 Fouls: 2 : 12 Score: 0 : 0
  • 15. Deep Dive Analysis
  • 16. Options “To Fix” the situation
  • 17. Not always a happy ending  Minute 90 Ball Possession: 80 : 20 Fouls: 4 : 25 Score: 3 : 0
  • 18. FRUSTRATED FANS!! 25
  • 19. How does that relate to Software?
  • 20. From Deploy to … Deploy Promotion/Event Problems Ops Playbook War Room Timeline
  • 21. The “War Room” – back then 'Houston, we have a problem‘ NASA Mission Control Center, Apollo 13, 1970
  • 22. The “War Room” – NOW Facebook – December 2012
  • 23. 3 Situations on WHY this happens, HOW to avoid it
  • 24. Image taken from https://www.scriptrock.com/blog/devops-whats-hype-about/
  • 25. #Disconnected Teams
  • 26. “Teamwork” between Dev and Ops SEV1 Problem in Production Need access to log files Where are they? Can’t get them Need to increase log level Can’t do! Can’t change config files in prod!
  • 27. “Solution”: Implement a Custom “On Demand” Remote Logger
  • 28. Implementation and Rollout Implemented Custom Logger Worked well in Load Testing
  • 29. What happened? ~ 1Mio Lock Exceptions in 30 mins
  • 30. Root Cause: A special WebSphere Setting! Log Service provides a synchronized log file across ALL JVMs Log Service provides a synchronized log file across ALL JVMs
  • 31. Metrics: # Log Messages, # Exceptions Share: Same Server Settings DevOps: Agree on Data for Troubleshooting
  • 32. #No “Agile” Deployment
  • 33. Adonair Load Spike resulted in Unavailability
  • 34. Alternative: “GoDaddy goes DevOps” 1h before SuperBowl KickOff 1h after Game ended
  • 35. Behind the Scenes
  • 36. Metrics: Availability Page Size, # Objects # Hosts, # Connections DevOps: “Feature” Switches
  • 37. #Push without a Plan
  • 38. Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
  • 39. m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections
  • 40. Critical Pages not Optimized! Browse, Search and Product Info performs well … because they don’t follow best practices: 87 Requests, 28 Redirects, … Critical Pages such as Shopping Cart are very slow …
  • 41. Metrics: Load Time, # Resources (Images, …), # HTTP 3xx, 4xx, 5xx Dev: Build for Mobile Test: Test on Mobile Ops: Monitor Mobile
  • 42. # of Requests / User # of Log Messages # of Exceptions # Objects Allocated # Objects In Cache Cache Hit Ratio # of Images # of SQLs # SQLs per RequestAvailability # HTTP 3xx, 4xx Page Size
  • 43. 54
  • 44. Commit Stage • Compile • Execute Unit Test • Code Analysis • Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing • Key showcases • Exploratory testing Release Unit & Integration Tests Functional Tests Performance Tests Production Monitoring Functional Tests
  • 45. If we do all that
  • 46. Which gives you more time for the real important things in life …
  • 47. Want MORE of these and more details? http://apmblog.compuware.com
  • 48. Recommended Book https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/
  • 49. FREE Products & More Info • dynaTrace Enterprise – Full End-to-End Visibility in your Java, .NET, PHP Apps – Sign up for a 15 Days Free Trial on http://compuwareapm.com • dynaTrace AJAX Edition – Browser Diagnostics for IE + FF – Download @ http://ajax.dynatrace.com • Our Blog: http://apmblog.compuware.com