Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BBCRM Performance: From Cobblestones to Autobahnen


Published on

bbcon 2013

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

BBCRM Performance: From Cobblestones to Autobahnen

  1. 1. 10/7/2013 #bbcon 1 BBCRM Performance From Cobblestones to Autobahnen PRESENTED BY LEN WYATT
  2. 2. 10/7/2013 #bbcon 3 Cobblestones? Autobahnen? Why such a silly subtitle?
  4. 4. 10/7/2013 #bbcon 5 WHAT IS NEEDED TO BUILD AN AUTOBAHN
  5. 5. 10/7/2013 #bbcon 6 WHAT IS NEEDED TO USE A COBBLESTONE ROAD
  6. 6. 10/7/2013 #bbcon 7 WHAT IS NEEDED TO USE AN AUTOBAHN
  7. 7. 10/7/2013 #bbcon 8 • To schnell fahren auf der Autobahn, there is a lot behind the scenes! • The same is true for complex multi-user systems like BBCRM • So when something goes wrong, why is software so much harder to troubleshoot? LAYERS OF INFRATRUCTURE, KNOWLEDGE AND SUPPORT
  8. 8. 10/7/2013 #bbcon 9 • Browser • Operating System • Client computer • CPU, Memory, Disk • Local LAN • Internet • Web server • CPU, Memory, Disk • Data center LAN • Database server • CPU, Memory, Disk • Database schema • Database contents • Stored procedures • Customizations LAYERS OF ABSTRACTION What the user did What everybody else is doing What is stored in the systems How the systems have been configured How the systems have been maintained Developers learn to use these abstractions to hide the behavior of other layers Performance issues can occur in any layer. Everything is subject to examination.
  9. 9. 10/7/2013 #bbcon 10 • Because abstraction is so easy in software, we do it a lot! • All the tools, infrastructure and support are built on a digital background that we can’t physically see – we have to hold all the concepts in mind • Evolution has not prepared us for this • We do not have hundreds of years of shared experience LAYERS OF ABSTRACTION
  10. 10. 10/7/2013 #bbcon 11 • “It’s taking a long time” does not point to the problem • This is not the user’s fault - The layers of abstraction hide the issue • We have to ask a lot of questions, and use tools to collect information WHAT THE USER CAN TELL US
  11. 11. 10/7/2013 #bbcon 12 ENTER THE ENTERPRISE PERFORMANCE TEAM
  12. 12. 10/7/2013 #bbcon 13 ENTERPRISE PERFORMANCE TEAM WORK AREAS • Tactical improvements. Some customers have performance issues today, and we need to address key ones. Tactical work focuses on carefully chosen areas that are common to multiple customers, and seeks solutions. Recommendations are made to product teams. • Strategic improvements. Long term, we must find ways to improve the product design and engineering processes to focus on performance early and continually. • Practices. Deliver information to help customers as well as our own customer-facing teams (SDO, Prof. Svcs, Support, Sales) do their jobs better.
  13. 13. 10/7/2013 #bbcon 14 THIS IS A NEW APPROACH • Yep, we have had some performance problems with BBCRM • No, we haven’t always handled them well • It feels like we have been playing Whack-a-Mole • Now we have… • A focused team • Working on root causes • Being scientific • Bringing new thinking about performance to the organization
  14. 14. 10/7/2013 #bbcon 15 Performance work across the organization… Product performance deliverables: • Ad-Hoc Query Improvements • Revenue Batch Improvements • Security Model Improvements • Blackbaud Direct Marketing segmentation on the DW Driven by development teams • Planned, scheduled work BBCRM PERFORMANCE INITIATIVES Research-based approaches: • Tactical improvements • Strategic improvements • Practices information • All based on representative workloads Driven by the Enterprise Performance Team • Exploratory & opportunistic work
  15. 15. 10/7/2013 #bbcon 16 ONE TACTICAL IMPROVEMENT • Individual Revenue Summary View Data Form • Found issue by observing Slow Web Requests in SDM • Issued patch for 2.93 (later versions were already improved) • The impact on AHA alone: • Before (6/1/2013 – 6/30/2013) • 4,889 SWRs • Average 60 seconds per SWR (55 seconds over threshold) • After (7/1/2013 – 8/5/2013) • 113 SWRs • Average of 7 seconds per SWR (2 seconds over threshold) • Nearly ½ of a full time person no longer spent waiting!
  16. 16. 10/7/2013 #bbcon 17 • Postings on the BBCRM blog - EARLY PRACTICES RESULTS
  17. 17. 10/7/2013 #bbcon 18 • Who has seen those blog postings? • What is the most effective way to get information out to you? • Is it different depending on the type of information? - User tips - Database administrator recommendations - Configuration & deployment guidance QUESTIONS!
  19. 19. 10/7/2013 #bbcon 20 • We study and learn from customer experience - Review and analyze Work Items related to performance - Study available instrumentation • SDM data for hosted customers • We rarely work directly on a specific customer case - Because we are looking for the root causes that affect many customers • We examine the workings of a system in operation - To see what’s really happening, and not speculate • We work with engineering teams to implement changes - Code ownership stays where it belongs - Developers and QA people learn from the process SCIENCE STARTS WITH OBSERVATION
  20. 20. 10/7/2013 #bbcon 21 • To study a working system, a testbed is required - Does it help if we make this change? - Are there side-effects from changing this? - Is alternative A better than B? - Is this configuration optimal? - What if we have more (of fewer) CPUs? Memory? Disks? - Could we support more users? - What is the source of database growth? - How does index fragmentation change performance? - Would database compression help performance? - … and on and on… • Using the testbed requires a workload - Models the activities the system is supporting - Models the data in the system - Allows measurement of response times TESTBED
  21. 21. 10/7/2013 #bbcon 22 • Both the state of the SUT and the applied work must be meaningful in order to get a meaningful result - Mostly this means the database contents and the user actions • Representative workloads will allow us to do science instead of guessing MEANINGFUL WORKLOADS ARE VITAL FOR PERFORMANCE WORK Database Reflects the rich complexity of real customer databases Web Tier User Actions Reflect what users do and action rates Test Driver System Under Test
  22. 22. 10/7/2013 #bbcon 23 • Workloads model user actions - What do users do? - What parameters are filled in? - How much time between actions? - How many users? • Workloads model the database contents - How many constituents? Addresses? Contributions? Reports? Tributes? - The patterns in the data are important… • Do some constituents make many contributions and others only a few? • Are there some for whom we have lots of address changes recorded? • What is the ratio of general contributions vs. directed ones? • Creating a workload like this is expensive; we can’t do it too often! - So it has to be representative of many customers WORKLOADS
  23. 23. 10/7/2013 #bbcon 24 • To create lab workloads that are highly representative of real customer workloads, we are modeling from systems run by Performance Partners - If we do our job well, the lab workloads will be almost doppelgangers of the originals • Performance Partners… - Are BBCRM customers - Chosen because they have issues representative of multiple others - Agree to work with us - Let us work with a (masked) copy of their database - Let us monitor user activity for an extended time (a month) - Agree to let us instrument systems from time to time, if needed - Want to try updates when applicable - Have willingness and capability to test, measure and try some things PERFORMANCE PARTNERS
  24. 24. 10/7/2013 #bbcon 25 • Heifer International • The University of Iowa Foundation • They are different… - Usage models - Sizes of databases - Patterns of performance issues - Hosted by Blackbaud vs. self-hosted • They are similar to others in their categories PERFORMANCE PARTNERS
  25. 25. 10/7/2013 #bbcon 26 • What system configurations do our customers use? - For hosted customers, we know - For self-hosted customers, we have only spotty information • How can we know if our testbed is representative of self-hosted sites? • Help us help you… - We have sent a survey through the TAMs - It asks about physical equipment - Accompanied by a script that collects statistics on the database • If you haven’t seen this, ask your TAM - Have them ask me if they don’t know  TESTBED, REDUX
  26. 26. 10/7/2013 #bbcon 27 THE SELF-HELP SECTION
  27. 27. 10/7/2013 #bbcon 28 • End Users - Watch for tips like we have posted on the BBCRM blog • We have posted about Constituent Search and Querying by Merging Selections - Try not to run large batch operations or slow reports during prime time • System administrators - Make sure maintenance tasks are performed regularly • Update statistics • Rebuild indexes - Eliminate unnecessary data • Excessively old audit data • Unused export tables - Know your baseline performance • Monitor PerfMon counters MAKING THINGS BETTER YOURSELF
  28. 28. 10/7/2013 #bbcon 29 • Developers - Familiarize yourselves with SQL query tuning techniques - Follow known best practices • Learn them at talks tomorrow: “Blackbaud CRM Boot Camp” Parts 1 & 2 • Support staff - Be familiar with what Blackbaud support will need to know about performance issues • You can start to ask the same questions MAKING THINGS BETTER YOURSELF
  29. 29. 10/7/2013 #bbcon 30 HELP BB SUPPORT HELP YOU!
  30. 30. 10/7/2013 #bbcon 31 • Was any of that helping-yourself material a surprise? • Which of those things are you doing today? • What tools do you use for routine maintenance tasks? DISCUSSION
  31. 31. 10/7/2013 #bbcon 32 FIN • We’re taking a whole new approach to performance • This is going to take time • The benefits are going to be big • We are improving BBCRM • We are changing the organization
  32. 32. 10/7/2013 #bbcon 33 WHEW! THAT WAS A LOT OF MATERIAL.