Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Challenges Engineering Applications For High Performance


Published on

This presentation talks about the common pitfalls from a systems performance perspective and what development teams could do to engineer performance proactively into the system design, build and test process. It also throws light on the practices of SPE (Systems Performance Engineering) across the Development Life Cycle.

Published in: Technology
  • Be the first to comment

Challenges Engineering Applications For High Performance

  1. 1. Challenges Engineering Applications for High Performance Practical Performance Analyst
  2. 2. • Started mucking around with Open Source 15 years ago • Fell in love with Gnu/Linux and more importantly the Free Software / Open Source movement • Travelled across the country advocating the use of Linux and Free Software • Went to work for Red Hat. Sold & deployed Free Software • Worked at the Performance R&D Labs at TCS and then moved to Consulting • Moved to Accenture Technology Consulting and now part of Tech Arch, Performance Engineering • Founded Practical Performance Analyst ( • Revived CMGA (Computer Management Group, Australia) A few words about me
  3. 3. How Do You Define Performance Throughput & Congestion across the last mile Poor End User Experience Client Device Performance Congestion on the Internet Infrastructure Performance Application Performance Backbone Network Throughput & Performance Data Center Network Performance
  4. 4. • All of us have cribbed about poor performing solutions at one point or the other. • Unfortunately, poor performance seems to be the norm and we’ve all accepted that it’s just how things are. Lets look at some examples: • Web Store: The web is abound with case studies about online retailers loosing revenue as users turn to competitors offerings. Users today aren’t very tolerant of poor performing websites and with easy access to google finding an alternate offering is no more than a few clicks away. There’s tons of cheap tools available to monitor system these days but engineering systems to performance is a completely different kettle of fish. • Ticketing / Help Desk Application: You tried reaching out to your internal help desk for support and it seemed like eternity before you got through. But even after you got through to the help desk, the wait continued because the help desk technician just couldn’t get to the data they needed quickly to help you out. The tech team looking into the issue needed insight into the performance bottleneck but the tools they had at their disposal only gave them a siloed view of performance which wasn’t good enough to get to the root cause quickly. • Batch Job Performance : You’ve had to go back and confess to business that the batch jobs your teams had designed weren’t performing the way you intended them to perform. Un-fortunately, the developers of the batch job hadn’t taken into account the volume of data that the system would be need to tested for. Your development environments happen to be a fraction of your production environments and so was the data store for performance testing. How Does Poor Performance Manifest Itself
  5. 5. What is SPE • Definition – Systems Performance Engineering (SPE) is a systematic and quantitative approach for the cost-effective development of software systems to meet stringent Non Functional Requirements i.e. Performance, Capacity, Scalability, Availability, Reliability, etc. • Systems Performance Engineering is focused on optimal selection of application architecture, design, build and implementation choices with the objective of meeting Non Functional Requirements. • Systems Performance Engineering can also be defined functionally as the set of tasks or activities that need to be performed across the Software Development Life Cycle (SDLC) to meet the documented Non Functional Requirements. • Software Performance Engineering is often viewed as the art of building systems that meeting Non Functional requirements within the allocated time frame and budget constraints.
  6. 6. Where Do Programs Generally Go Wrong • Lack of Non Functional Requirements - 99% of programs I’ve worked on have no or very poorly define Non Functional Requirements. Ask 5 different people on the team their interpretation of on Non Functional and you will end up getting 5 different answers. As a result – • Architects have no guidance on what are the right design choices • Developers don’t have unit performance or tier performance targets • Infrastructure designers have no idea of what specs they need to be building for • Inability to regression test nightly builds against unit performance or tier performance targets • Performance testers have no idea of what and how the various application workloads should be evaluated • Lack of investment in right capability – Performance isn’t just about response times or server utilization. Engineering performance goes far beyond all of that and it happens to be a way of thinking. Programs rarely invest in the right capability. The fact is most program leads lack an understanding of what is required to build performance into the solution. Functional capability is easy to nail down, performance is rather complicated since it traverses so many domains across the delivery chain. • Lack of investment in appropriate tools – Tooling is a very complicated subject and performance tools happen to be expensive. Most projects think that Nagios (performance monitoring) and Jmeter (performance testing) is good enough to get them over the line. • Lack of investment in the required infrastructure – To validate performance you need appropriately sized infrastructure. Testing on infrastructure that is <50% of production and drawing conclusions based on assumptions of linear scalability is quite frequent and a perfect recipe for disaster. • Lack of funding to design, build, validate and manage performance across the delivery cycle – Performance starts at the requirements gathering stage and continues onto architecture, design, build, test and then capacity planning. Each of these functions requires the right capability across the board. Projects seldom invest in the right capability across the board.
  7. 7. SPE Across the SDLC Software Development Life Cycle Functional Requirements Gathering Architecture & Design Build Application System Test, System Integrated Test & UAT Deploy Into Production Performance Engineering Life Cycle Non Functional Requirements Gathering Design for Performance & Performance Modelling Unit Performance Test & Code Optimization Performance Test Monitoring & Capacity Management
  8. 8. Performance By Design • Requirements Gathering – • Document Non Functional Requirements • Work with Business & IT to get NFR’s signed off • Decompose NFR targets across various application tiers • Identify & budget for relevant tools for monitoring, diagnostics, performance testing and capacity planning • Identify & budget relevant capability required to integrate performance across the SDLC i.e. Performance at Design, Performance at Build, Performance Testing, Diagnostics and Tuning, Monitoring & Capacity Management • On programs that can afford it, have an owner for Performance across the program • Design – • Work with Architecture teams on adopting the relevant design patterns • PoC design / implementation options and identify potential constraints • Document Overall Performance Engineering strategy • Document Overall Performance Testing Strategy • Document Overall Capacity Management & Monitoring Strategy Performance Requirements Analysis Performance Modelling & Capacity Planning Build & Optimization Performance Testing Performance Monitoring Capacity Management
  9. 9. Performance By Design • Build – • Translate the NFR’s to tier wise performance targets • Work with development teams and establish process to track regression of code performance • Enforce unit performance testing of code and work with developers to track tier wise performance targets • Work with development teams and assist with usage of profiling & diagnostics tools. Use diagnostics tools to identify potential bottlenecks and optimize code • Test – • Build stubs and begin early performance testing • Document workload models and get them signed off • Execute end to end performance testing • Use Diagnostics tools to identify bottlenecks • Analyst results, tune, optimize and re-run • Go Live - • Implement system, application, transaction and end user monitoring • Collect relevant business/transaction performance metrics • Forecast & model performance impacts using performance metrics Performance Requirements Analysis Performance Modelling & Capacity Planning Build & Optimization Performance Testing Performance Monitoring Capacity Management
  10. 10. SPE Roles Across The SDLC Phase Role Performed Requirements Gathering Determine Business Volumes & Growth plans Determine Non Functional Requirements Design Validate Application Architecture Validate Infrastructure Architecture Validate Infrastructure Design Determine Infrastructure Capacity Requirements Build & Optimization Set performance targets for developers Validate outcome of Unit Performance Tests Recommend optimization to Code Recommend optimization to Application Design Recommend optimization to Infrastructure Design SIT & UAT Begin validating Application Performance for functionality that’s available Begin Tier Performance Tests for functionality that’s available Test, tune & optimize Performance for functionality that has been released SVT Execute End to End Performance Tests Validate Performance End to End Identify applicable bottlenecks Use Diagnostics tools to identify bottlenecks, tune & Optimize application Ensure application meets its Non Functional Requirements Evolve Performance Monitoring Requirements Pre-Go live Setup Performance Monitors for Network, OS, Application, Business Txn Set alerting for various Performance metrics Setup capture Performance metrics for purposes of Capacity Management Go-Live & Post Go Live Monitor application for potential breaches in SLA’s Identify hotspots using low overhead transactional tracing & diagnostics tools Model application performance and predict capacity impacts for growing business workload
  11. 11. Devops, Continous Integration, Agile, etc. • Devops, Agile, Continous Integration are paradigms that are have begun to influence development practices around the world. Each of these paradigms have their own strengths and weaknesses. • Whatever you development paradigms you may adopt for development of your systems you will need to integrate Proactive Performance Management or SPE principles into it • Un-fortunately most of the bad habits (mentioned in earlier slides) with regards to poor SPE practices are carried into the new paradigms as well. It’s quite rare that we find programs adopting proactive SPE practices into their Agile development processes. • Agile, Devops, CI are paradigms have a lot of good to offer but don’t forget to integrate Proactive Performance Management practices and more importantly SPE practices into your implementation of those processes.
  12. 12. What Am I Doing To Change The World • Practical Performance Analyst - • What is PPA - Practical Performance is a community portal that I started work on in 2012. Practical Performance Analyst has grown significantly over the last couple of years with a growing readership base. Our aim is for Practical Performance Analyst to be a open platform for sharing and collaboration on standards, practices and guidelines around Systems Performance Engineering (SPE). Practical Performance Analyst has a growing list of international authors who are experts in their own domains of Performance. Practical Performance Analyst aims to be a hub for all things SPE (Systems Performance Engineering). Drop in and find out. • Accessing PPA – • Contributing & Supporting PPA – • Computer Management Group, Australia – • What is CMGA - Computer Management Group Australia is a professional body that focuses on bringing together professionals from across Australia who have interests in Systems Performance Engineering (SPE). CMG is an international organization of professionals ( and CMGA is the Australian Chapter of CMG International. Spent the last 6 months reviving CMGA and getting CMGA going again here in Australia. • Accessing CMGA – • Joining CMGA – Joining CMGA is quite easy. Hop over to the CMG website and join the group using the Meetup link on the right hand side corner. We use the meetup group to keep in touch and co-ordinate all national events.
  13. 13. What Can You Do To Help • Learning & Sharing – If you are interested in learning more about Proactive Performance Management & Systems Performance Engineering head off to Practical Performance Analyst ( • There’s tons of resources out there on all topics related to SPE. • You’ll also find access to books and tools on all aspects of SPE • Grab your daily dose of SPE news from around the world at PPA • Contributing & Helping Build the Community – If you are interested in helping building the SPE Community and contribute your knowledge from a Design, Architecture, Build, Optimization, APM, Performance Testing, Capacity Management, Modelling, etc. perspective please drop me an email at We are keen to expand out list of authors and grow our community of readers. • Networking & Lobbying – Come and join us at Computer Management Group Australia. CMG Australia is a professional organization with a focus on evolving standards, fostering innovation and most importantly advocating adoption of SPE across programs. CMG Australia has various initiatives focused on SPE across industry, academia and research. Come join us and help build appreciation for good development practices.
  14. 14. Thank You Practical Performance Analyst