Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Elegant database tuning

1,228 views

Published on

Database tuning is an ongoing exercise. This presentation (from Postgres Conf 2018 NJ) shows you how to continually tune your database. What metrics do you need to track and how can you feed them back into your application, data lifecycle and database config.

Published in: Technology

Elegant database tuning

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Postgres Conf US - April, 2018
  2. 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Tuning • Application • Data Lifecycle • Database configuration
  3. 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tuning Methodology 1. Run unit tests 2. Track database statistics 3. Baseline Config 4. Run Load 5. Track database statistics 6. Update config 7. GOTO 4
  4. 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What should we track? • Activity • Database Utilization • Table Utilization • Index Utilization • Locking
  5. 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How do we track it? • PG Catalog • Schema in every postgres db (pg_catalog) • Appended to search_path • “Data Dictionary”
  6. 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog SELECT pn.nspname, pc.relkind, count(1) FROM pg_namespace pn, pg_class pc WHERE pn.oid = pc.relnamespace AND pn.nspname = 'pg_catalog' GROUP BY pn.nspname, pc.relkind ORDER BY 1 DESC; nspname | relkind | count ------------+---------+------- pg_catalog | i | 115 pg_catalog | r | 62 pg_catalog | v | 59 i = indexes r = tables v = views
  7. 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog functions select count(1) FROM ( select pn.nspname, pp.proname FROM pg_namespace pn, pg_proc pp WHERE pn.oid = pp.pronamespace AND pn.nspname = 'pg_catalog' ) foo; count ----- 2882
  8. 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog functions SELECT pn.nspname, pp.proname FROM pg_namespace pn, pg_proc pp WHERE pn.oid = pp.pronamespace AND pn.nspname = 'pg_catalog’; nspname | proname ------------+-------------- pg_catalog | boolin pg_catalog | boolout pg_catalog | byteain pg_catalog | byteaout pg_catalog | charin pg_catalog | charout pg_catalog | namein pg_catalog | nameout ...
  9. 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog • Structural • Performance • Informational
  10. 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog – translation! • Relation = table, index, sequence, etc… • Attribute = column • Namespace = schema
  11. 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog - oid • Object Identifier datatype • 4 byte unsigned int • Hidden column on catalog tables • Primary / Foreign keys throughout pg_catalog schema
  12. 12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog - oid postgres=# select * from pg_namespace; nspname | nspowner | nspacl --------------------+----------+---------------------------- pg_toast | 10 | pg_temp_1 | 10 | pg_toast_temp_1 | 10 | pg_catalog | 10 | {meads=UC/meads,=U/meads} public | 10 | {meads=UC/meads,=UC/meads} information_schema | 10 | {meads=UC/meads,=U/meads} (6 rows)
  13. 13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog - oid postgres=# select oid, * from pg_namespace; oid | nspname | nspowner | nspacl -------+--------------------+----------+---------------------------- 99 | pg_toast | 10 | 11816 | pg_temp_1 | 10 | 11817 | pg_toast_temp_1 | 10 | 11 | pg_catalog | 10 | {meads=UC/meads,=U/meads} 2200 | public | 10 | {meads=UC/meads,=UC/meads} 12349 | information_schema | 10 | {meads=UC/meads,=U/meads} (6 rows)
  14. 14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog – List all tables in a schema SELECT pn.nspname, pc.relkind, count(1) FROM pg_namespace pn, pg_class pc WHERE pn.oid = pc.relnamespace AND pn.nspname = 'pg_catalog' GROUP BY pn.nspname, pc.relkind ORDER BY 1 DESC;
  15. 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What’s this have to do with tuning??
  16. 16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tuning Methodology 1. Run unit tests 2. Track database statistics 3. Baseline Config 4. Run Load 5. Track database statistics 6. Update config 7. GOTO 4
  17. 17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What should we track? • Activity • Database Utilization • Table Utilization • Index Utilization • Locking
  18. 18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Where is the data? • pg_stat_activity • pg_stat_database • pg_stat_all_tables / pg_stat_user_tables • pg_stat_all_indexes / pg_stat_user_indexes • pg_locks
  19. 19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Activity • pg_stat_activity • Current activity in the database • Snapshot of that instant • Starting point for tuning • State • Idle • Active • Idle in transaction select datname, usename, client_addr, wait_event, now() - state_change as runtime, state, query from pg_stat_activity ;
  20. 20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Activity - Tuning • Application • Connections • Concurrency • Data Lifecycle • Query runtimes • WAL
  21. 21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Activity - Tuning • Server configuration • max_connections • shared_buffers • work_mem
  22. 22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Statistics • pg_stat_database • Utilization stats for each DB in an instance • Global view of all DBs select datname, numbackends, xact_commit, tup_returned, tup_fetched, tup_inserted, tup_updated, tup_deleted, pg_database_size(datname), pg_size_pretty(pg_database_size(datname)) FROM pg_stat_database;
  23. 23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table Statistics • pg_stat_all_tables / pg_stat_user_tables • Cumulative view of table-level stats • Basis for workload profile • pg_stat_all_indexes / pg_stat_user_indexes • Cumulative view of index-level stats • Index utilization stats select schemaname, relname, n_tup_ins, n_tup_upd, n_tup_del, idx_scan, seq_scan FROM pg_stat_user_tables;
  24. 24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Where are my hot-spots? • What’s the most modified (INS / UPD / DEL)? • What’s the most read ? • What’s the largest?
  25. 25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table Stats -Tuning • Application • Partitioning • SQL • Stored Procedures
  26. 26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table Stats - Tuning • Data Lifecycle • Datamodel • Indexing • Workflow (Truncate vs Delete) • Database configuration • Vacuum • Shared_buffers
  27. 27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Locking • pg_locks • Status of all locks in a given database • Look for conflicting locks WHERE granted = ‘f’
  28. 28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Locking - Tuning • Application • Workflow • Data Lifecycle • Data Model • Normalization • ETL • Maintenance
  29. 29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tracking vs Storing • Cloud Watch • Performance Insights • POWA • PGWatch2 • Snapshot scripts • Grafana • ELK Max CPU
  30. 30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Learn more.. aws.amazon.com/rds aws.amazon.com/rds/aurora
  31. 31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Scott Mead meads@amazon.com

×