Understanding System Performance


Published on

Learn about when the best time for a system tune-up of your data warehouse is by Jim Blair.

Published in: Technology
  • Be the first to comment

Understanding System Performance

  1. 1. Understanding System Performance – When is it Time for a System Tune-up? Jim Blair Data Warehouse Consultant / Marketing Specialist – Teradata
  2. 2. Agenda <ul><li>Overall System Performance </li></ul><ul><ul><li>Overview of DBQL and Viewpoint </li></ul></ul><ul><ul><li>Baselines and Benchmarks </li></ul></ul><ul><ul><ul><li>Metrics and reports </li></ul></ul></ul><ul><ul><li>Real-time alerts </li></ul></ul><ul><ul><li>Growth Patterns </li></ul></ul><ul><li>Time for a tune up </li></ul><ul><ul><li>Best Practices in Benchmarks </li></ul></ul><ul><ul><li>Query Tuning </li></ul></ul><ul><ul><li>Load considerations </li></ul></ul><ul><ul><li>Compression </li></ul></ul><ul><ul><li>Other Considerations </li></ul></ul>
  3. 3. High Level Performance <ul><li>Inefficient or bad queries? </li></ul><ul><li>Updates and data loading? </li></ul><ul><ul><li>Locking tables and rows for too long </li></ul></ul><ul><li>Too many concurrent tasks? </li></ul><ul><ul><li>Big consumption workloads </li></ul></ul><ul><ul><li>Jobs that should run later </li></ul></ul>
  4. 4. Heatmap from DBQL <ul><li>Total CPU usage – Is Your System Running HOT ? </li></ul>HOT
  5. 5. Heartbeat Graph
  6. 6. What is Teradata Viewpoint? <ul><li>System administration portal </li></ul><ul><ul><li>Portlets for status displays </li></ul></ul><ul><ul><li>Targets business and technical users </li></ul></ul><ul><ul><li>Rewind for system analysis </li></ul></ul><ul><ul><li>Manage multiple systems </li></ul></ul><ul><li>Appliance </li></ul><ul><ul><li>Server + OS + software </li></ul></ul><ul><ul><li>Completely supported by Teradata </li></ul></ul><ul><li>Future platform for Teradata management products </li></ul><ul><ul><li>All management Teradata Tools and Utilities </li></ul></ul><ul><li>NOT an Enterprise Portal </li></ul><ul><ul><li>Does not compete with WebSphere, BEA, SAP, etc. </li></ul></ul>
  7. 7. Viewpoint Portlets in Action <ul><li>Filtered Queries </li></ul><ul><li>List View of all sessions on a Teradata system </li></ul><ul><li>View sessions by different categories </li></ul><ul><ul><li>Active, parsing, delayed, etc. </li></ul></ul><ul><li>Set thresholds to spot problem queries </li></ul><ul><li>Drill down into a session, and access explain plan and SQL text </li></ul><ul><li>Allows for DBAs to easily manage and track sessions and queries filtered by state </li></ul>
  8. 8. Benchmarks and Baselines <ul><li>Create a ‘real life’ benchmark </li></ul><ul><ul><li>Body of 4 hrs real workload </li></ul></ul><ul><ul><li>Do not use static tables </li></ul></ul><ul><ul><li>Predictable results </li></ul></ul><ul><ul><li>Include all types of queries </li></ul></ul><ul><ul><ul><li>capture 25-40 different queries and extrapolate in 4 hrs of work </li></ul></ul></ul><ul><li>Run the benchmark with consistency </li></ul><ul><ul><li>Same # of concurrent queries each time </li></ul></ul><ul><ul><li>Same queries each time </li></ul></ul><ul><ul><li>Access production data </li></ul></ul><ul><ul><li>Capture metrics on each query </li></ul></ul>
  9. 9. <ul><li>Establish a Baseline </li></ul><ul><ul><li>Run benchmark in lowest period of system usage </li></ul></ul><ul><ul><li>Quiese the system if possible </li></ul></ul><ul><ul><li>Capture metrics on all queries </li></ul></ul><ul><li>When to run the Benchmark process </li></ul><ul><ul><li>Quarterly </li></ul></ul><ul><ul><li>Before/After every Upgrade </li></ul></ul><ul><ul><ul><li>HW, SW, </li></ul></ul></ul><ul><ul><ul><li>major implementations </li></ul></ul></ul><ul><ul><li>On Demand </li></ul></ul><ul><li>Analyze results </li></ul><ul><ul><li>What’s changed since last run? </li></ul></ul>Baselines and Benchmarks Major Upgrade
  10. 10. Real Time Alerts <ul><li>Canary Queries </li></ul><ul><ul><li>One in each work load </li></ul></ul><ul><ul><li>Set threshold for alerts </li></ul></ul><ul><li>DBA Alerts </li></ul><ul><ul><li>High level of spool </li></ul></ul><ul><ul><li>Skewing </li></ul></ul><ul><ul><li>Number of users </li></ul></ul>
  11. 11. <ul><li>All production Raw data </li></ul><ul><li>Benchmark tables </li></ul><ul><li>Don’t forget Overhead! </li></ul><ul><ul><li>Spool, DBC, DBA workspace </li></ul></ul>Track Growth Patterns
  12. 12. Data Growth <ul><li>Trends Over Time </li></ul>
  13. 13. Performance Tuning <ul><li>Time to Get Under the Hood </li></ul>Your Sr. DBA
  14. 14. When to Run a Benchmark <ul><li>General performance trending </li></ul><ul><li>To compare when migrating to or from Teradata </li></ul><ul><li>Regressions </li></ul><ul><li>Before and after Patches and Major or Minor Hardware or Software upgrades </li></ul><ul><li>Test before and after certain project implementations </li></ul><ul><li>May want to test before and after TASM implementations or modifications </li></ul>
  15. 15. Makings of a Successful Benchmark <ul><li>Consistent data being accessed </li></ul><ul><li>Consistent indexing, views, security, priorities </li></ul><ul><li>Queries should represent a true mixed workload </li></ul><ul><li>Maintains consistent concurrency levels </li></ul><ul><li>Correct results captured every time </li></ul><ul><li>Queries should be run in same order every time </li></ul><ul><li>Same number of queries completed every time </li></ul><ul><li>Meaningful reports </li></ul>
  16. 16. Building the Benchmark Process <ul><li>Process should be 100% automated </li></ul><ul><li>Capture Explains before and after </li></ul><ul><li>Data and queries represent a true workload </li></ul><ul><li>Process should ensure that all indices, statistics, join indices, etc. are current as expected </li></ul><ul><li>Use DBQL to capture queries to represent workload </li></ul>
  17. 17. Building the Benchmark Process <ul><li>Capture short, medium and long running queries </li></ul><ul><li>Process should allow for consistent number of concurrent queries </li></ul><ul><li>Design queries to return a count and not return huge sets of rows – Network could skew timings </li></ul><ul><li>Process should report on results when finished </li></ul>
  18. 18. Things to Check For <ul><li>Make sure response times for each query and overall process are not abnormally high </li></ul><ul><li>Checking overall or individual response times is NOT enough! </li></ul><ul><li>Make sure results are also accurate </li></ul><ul><li>Examine explain plans to see if dramatically changed </li></ul><ul><ul><li>Note: You probably will not care about this if query is running much better. </li></ul></ul><ul><li>One query can throw off the entire benchmark </li></ul>
  19. 19. Result Report Example <ul><li>Investigate! </li></ul><ul><li>Different results normally indicate a problem, but it could be that the benchmark spanned two dates </li></ul>
  20. 20. Response Time Report Example <ul><li>Look for Consistency and Compare to Past </li></ul>QUERY NUM AVG TIME MIN TIME MAX TIME TIMES RUN 12 1:41:09 1:25:17 1:48:49 4 26 1:24:57 1:05:16 1:37:53 20 17 1:11:02 1:04:50 1:16:54 4 14 0:53:57 0:36:20 1:01:54 4 22 0:29:16 0:23:36 0:35:28 8 24 0:18:13 0:17:45 0:18:38 4 6 0:18:33 0:14:43 0:24:43 24
  21. 21. Final Thoughts on Benchmarks <ul><li>Take special note when changes are made to views, indices, TASM, etc. </li></ul><ul><li>These changes may unexpectedly improve or even impair your benchmark </li></ul><ul><li>Keep the benchmark current </li></ul>
  22. 22. Workload Management Number of Incoming Queries Reject Filter Delay Throttle Reject Throttle Exception Reject Outside Of SLA Queries After Filters and Throttles Exception Reclass
  23. 23. Query Tuning <ul><li>Can save a tremendous load on your system </li></ul><ul><li>Need process to tune query, but then inform and train users as well </li></ul><ul><li>Identify offending queries and report </li></ul><ul><ul><li>Viewpoint-Query Replay </li></ul></ul><ul><ul><li>Document all findings, strategies, time saved, CPU and IO saved, etc. </li></ul></ul>
  24. 24. Query Tuning <ul><li>Look for common mistakes such as missing aliases, product joins, poor primary index selection on “Create Tables” </li></ul><ul><li>Make sure necessary statistics are collected </li></ul><ul><li>diagnostic helpstats on for session; </li></ul><ul><li>Try tricks like GROUP BY instead of DISTINCT if it applies </li></ul>
  25. 25. Query Tuning <ul><li>Look for ways to impact multiple queries with one tuning effort or enhancement </li></ul>
  26. 26. Load Techniques <ul><li>Goal – Avoid contention between loads, users, and backups </li></ul><ul><li>Establish DDL naming conventions and document </li></ul><ul><li>Automate process to validate all DDL </li></ul><ul><li>Get Developers to collaborate with DBAs </li></ul><ul><li>Educate Developers on database technology and features (i.e. Locking, Backups, Utilities) </li></ul><ul><li>Educate DBAs on ETL complexities </li></ul>
  27. 27. Load Techniques <ul><li>Start using TPT </li></ul><ul><li>Establish Load standards or conventions and enforce them at the ETL level </li></ul><ul><li>Examples </li></ul><ul><ul><li>Trickle Batch </li></ul></ul><ul><ul><li>Mini Batch </li></ul></ul><ul><ul><li>TPUMP </li></ul></ul><ul><ul><li>Dynamic stage table creation </li></ul></ul><ul><ul><li>Dual Database design </li></ul></ul>
  28. 28. Compression <ul><li>Facts </li></ul><ul><ul><li>Compress 255 values + Nulls </li></ul></ul><ul><ul><li>Cannot compress PI columns </li></ul></ul><ul><ul><li>Can improve performance </li></ul></ul><ul><li>May even pay to compress small tables so they can remain memory resident </li></ul><ul><li>Changes coming soon </li></ul><ul><ul><li>Compression on variable length data types </li></ul></ul><ul><ul><li>Algorithmic compression </li></ul></ul>
  29. 29. Other Ideas – Don’t reinvent the wheel! <ul><li>Materialize view definitions through Join Indexes or automation </li></ul><ul><li>Utilize new features like Multilevel Partitioning </li></ul><ul><li>Archive data </li></ul><ul><li>Horizontal partitioning </li></ul><ul><li>Upgrade – Query Rewrite and Optimizer improvements </li></ul><ul><li>Investigate Hot/Cold Data </li></ul>
  30. 30. Questions? Thank You!