1. pg proctab
Accessing System Stats in PostgreSQL
Mark Wong markwkm@postgresql.org
Gabrielle Roth gorthx@gmail.com
PGWest Seattle (JDCon) 2009
Oct 16-18, 2009

2. Slides available on slideshare
http://www.slideshare.net/markwkm

3. Soooo . . .
You can query the PostgreSQL system catalog tables (e.g.
pg stat activity, pg stat all tables,
pg stat all indexes) to find out which queries are taking a long
time, which indexes are being scanned an unreasonable number of
times, etc.

4. Example:
portal=# SELECT datname, procpid, usename, current_query
FROM pg_stat_activity;
datname | procpid | usename | current_query
---------+---------+----------+-------------------------------------------------
portal | 5412 | markwkm | <IDLE>
portal | 5437 | postgres | SELECT datname, procpid, usename, current_query
: FROM pg_stat_activity;
(2 rows)

5. What if you want to know about the OS?
pg proctab provides a collection of four C stored functions:
◮ pg cputime
◮ pg loadavg
◮ pg memusage
◮ pg proctab

6. What can do you with pg proctab?
◮ Query operating system process table
◮ Query operating system statistics
◮ Processor time
◮ Load averages
◮ Memory usage
◮ Without escaping out to a shell!
◮ ...plus generate reports about timeslices

7. pg cputime() Example
SELECT *
FROM pg_cputime();
user | nice | system | idle | iowait
--------+--------+--------+------------+--------
681317 | 109924 | 395481 | 1466101128 | 462661
(1 row)

8. pg cputime() Column Description
From Linux kernel source code at
Documentation/filesystems/proc.txt:
user: normal processes executing in user mode
nice: niced processes executing in user mode
system: processes executing in kernel mode
idle: processes twiddling thumbs
iowait: waiting for I/O to complete

9. pg loadavg() Example
SELECT *
FROM pg_loadavg();
load1 | load5 | load15 | last_pid
-------+-------+--------+----------
0.99 | 0.78 | 0.67 | 27719
(1 row)

10. pg loadavg() Column Description
load1: load average of last minute
load5: load average of last 5 minutes
load15: load average of last 15 minutes
last pid: last pid running

11. pg memusage() Example
SELECT *
FROM pg_memusage();
memused | memfree | memshared | membuffers | memcached | swapused | swapfree | swapcached
---------+---------+-----------+------------+-----------+----------+----------+------------
3809140 | 224084 | 0 | 60656 | 2389700 | 76 | 8385844 | 0
(1 row)

12. pg memusage() Column Description
Paraphrased from Linux kernel source code at
Documentation/filesystems/proc.txt:
memused: Total physical RAM used
memfree: Total physical RAM not used
memshared: Not used, always 0. (I don’t remember why. . . )
membuffers: Temporary storage for raw disk blocks
memcached: In-memory cache for files read from disk
swapused: Total swap space used
swapfree: Memory evicted from RAM that is now temporary on
disk
swapcached: Memory that was swapped out, now swapped in but
still in swap

13. pg proctab() Example 1
SELECT datname, procpid, usesysid, usename, uid, username
FROM pg_stat_activity, pg_proctab()
WHERE procpid = pid;
datname | procpid | usesysid | usename | uid | username
---------+---------+----------+----------+-----+----------
markwkm | 27801 | 10 | markwkm | 500 | markwkm
dbt3 | 27787 | 16770 | postgres | 500 | markwkm
(2 rows)

14. pg proctab() Example 2
SELECT datname, procpid, processor, state, fullcomm
FROM pg_stat_activity, pg_proctab()
WHERE procpid = pid;
datname | procpid | processor | state | fullcomm
---------+---------+-----------+-------+------------------------------------------
markwkm | 27801 | 0 | R | postgres: markwkm markwkm [local] SELECT
dbt3 | 29325 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29327 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29333 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29328 | 2 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29329 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29324 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29331 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 27787 | 1 | S | postgres: postgres dbt3 [local] idle
(9 rows)

15. pg proctab() Partial Column Description
Everything from the operating system such as /proc/<pid>/stat,
/proc/<pid>/io and /proc/<pid>/cmdline as well as data
from PostgreSQL system catalog such as pg stat activity table
are available but we’ll only cover some of the fields here:
Informative:
◮ pid
◮ comm - filename of the executable
◮ fullcomm (/proc/<pid>/cmdline)
◮ uid
◮ username
Processor:
◮ utime - user mode jiffies
◮ stime - kernel mode jiffies
...

16. pg proctab() Partial Column Description (cont.)
Memory:
◮ vsize - virtual memory size
◮ rss - resident set memory size
I/O:
◮ syscr - number of read I/O operations
◮ syscw - number of write I/O operations
◮ reads - number of bytes which this process really did cause to
be fetched from the storage layer
◮ writes - number of bytes which this process really did cause to
be sent from the storage layer
◮ cwrites - number of bytes which this process caused to not
happen, by truncating pagecache

17. __ __ /
/ ~~~/ . o O | Let’s try something |
,----( oo ) | more useful. |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

18. __ __ /
/ ~~~/ . o O | Measuring performance |
,----( oo ) | of a query. |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

19. (You can find the following examples in the pg proctab contrib
directory.)

20. Create snapshot tables.
(Only need to do this once.)
i create-ps_procstat-tables.sql

21. Identify yourself.
SELECT *
FROM pg_backend_pid();
pg_backend_pid
----------------
4590
(1 row)

22. Take a snapshot before running the query
i ps_procstat-snap.sql
BEGIN
ps_snap_stats
---------------
1
(1 row)
COMMIT

23. Execute the query
Don’t focus too much on the actual query, the idea is that is you
want to collect statistics for a single query:
SELECT nation,
o_year,
Sum(amount) AS sum_profit
FROM (SELECT n_name AS nation,
Extract(YEAR FROM o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount
FROM part,
supplier,
lineitem,
partsupp,
orders,
nation
WHERE s_suppkey = l_suppkey
AND ps_suppkey = l_suppkey
AND ps_partkey = l_partkey
AND p_partkey = l_partkey
AND o_orderkey = l_orderkey
AND s_nationkey = n_nationkey
AND p_name LIKE ’%white%’) AS profit
GROUP BY nation,
o_year
ORDER BY nation,
o_year DESC;

24. Take a snapshot after running the query
i ps_procstat-snap.sql
BEGIN
ps_snap_stats
---------------
2
(1 row)
COMMIT

25. Calculate Processor Utilization
$ ./ps-processor-utilization.sh [pid] [before] [after]
$ ./ps-processor-utilization.sh 4590 1 2
Processor Utilization = 1.00 %
What’s going on (partially):
SELECT stime, utime, stime + utime AS total,
extract(epoch FROM time)
FROM ps_snaps a, ps_procstat b
WHERE pid = ${PID}
AND a.snap = b.snap
AND a.snap = ${SNAP1}
TIMEDIFF=‘echo "scale = 2; (${TIME2} - ${TIME1}) * ${HZ}" | bc -l‘
U=‘echo "scale = 2; (${TOTAL2} - ${TOTAL1}) / ${TIMEDIFF} * 100" | bc -l‘

26. Calculate Disk Utilization
$ ./ps-io-utilization.sh 4590 1 2
Reads = 276981
Writes = 63803
Reads (Bytes) = 2164604928
Writes (Bytes) = 508166144
Cancelled (Bytes) = 36880384
SELECT syscr, syscw, reads, writes, cwrites
FROM ps_snaps a, ps_procstat b
WHERE pid = ${PID}
AND a.snap = b.snap
AND a.snap = ${SNAP1}
TIMEDIFF=‘echo "scale = 2; (${TIME2} - ${TIME1}) * ${HZ}" | bc -l‘
U=‘echo "scale = 2; (${TOTAL2} - ${TOTAL1}) / ${TIMEDIFF} * 100" | bc -l‘

27. __ __ /
/ ~~~/ . o O | Creating Custom |
,----( oo ) | Reports! |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

28. __ __ /
/ ~~~/ . o O | Warning! Too much data |
,----( oo ) | to fit on screen! |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

29. Creating Reports: Section 1
Database : dbt3
Snapshot Start : 2009-04-18 00:43:56.716034-07
Snapshot End : 2009-04-18 00:45:17.031167-07
-------------------
Database Statistics
-------------------
Commits : 0
Rollbacks : 2
Blocks Read : 213295
Blocks Hit : 1679509

30. Creating Reports: Section 2
================
Table Statistics
================
------------------------------------------ -------- ------------ -------- ------------- --------- --------
Schema.Relation Seq Scan Seq Tup Read Idx Scan Idx Tup Fetch N Tup Ins N Tup Up
------------------------------------------ -------- ------------ -------- ------------- --------- --------
information_schema.sql_features 0 0 0 0 0
information_schema.sql_implementation_info 0 0 0 0 0
information_schema.sql_languages 0 0 0 0 0
information_schema.sql_packages 0 0 0 0 0
information_schema.sql_parts 0 0 0 0 0
information_schema.sql_sizing 0 0 0 0 0
information_schema.sql_sizing_profiles 0 0 0 0 0
pg_catalog.pg_aggregate 0 0 2 2 0
pg_catalog.pg_am 1 1 0 0 0
pg_catalog.pg_amop 0 0 19 46 0
pg_catalog.pg_amproc 0 0 11 11 0
pg_catalog.pg_attrdef 0 0 1 2 0
pg_catalog.pg_attribute 0 0 137 331 0
pg_catalog.pg_auth_members 0 0 0 0 0
pg_catalog.pg_authid 3 2 0 0 0
pg_catalog.pg_autovacuum 0 0 0 0 0
pg_catalog.pg_cast 0 0 160 51 0
pg_catalog.pg_class 3 747 101 88 0
pg_catalog.pg_constraint 0 0 0 0 0
pg_catalog.pg_conversion 0 0 0 0 0
pg_catalog.pg_database 5 12 0 0 0
pg_catalog.pg_depend 0 0 0 0 0
pg_catalog.pg_description 0 0 0 0 0
pg_catalog.pg_index 2 200 39 50 0
...

31. Creating Reports: Section 2 - Falling off the right side...
◮ N Tup Upd
◮ N Tup Del
◮ Last Vacuum
◮ Last Autovacuum
◮ Last Analyze
◮ Last Autoanalyze

32. Creating Reports: Section 3
================
Index Statistics
================
------------------------------------------------------------ -------- ------------ -------------
Schema.Relation.Index Idx Scan Idx Tup Read Idx Tup Fetch
------------------------------------------------------------ -------- ------------ -------------
pg_catalog.pg_aggregate.pg_aggregate_fnoid_index 2 2 2
pg_catalog.pg_am.pg_am_name_index 0 0 0
pg_catalog.pg_am.pg_am_oid_index 0 0 0
pg_catalog.pg_amop.pg_amop_opc_strat_index 12 36 36
pg_catalog.pg_amop.pg_amop_opr_opc_index 7 10 10
pg_catalog.pg_amproc.pg_amproc_opc_proc_index 11 11 11
pg_catalog.pg_attrdef.pg_attrdef_adrelid_adnum_index 1 2 2
pg_catalog.pg_attrdef.pg_attrdef_oid_index 0 0 0
pg_catalog.pg_attribute.pg_attribute_relid_attnam_index 0 0 0
pg_catalog.pg_attribute.pg_attribute_relid_attnum_index 137 331 331
pg_catalog.pg_auth_members.pg_auth_members_member_role_index 0 0 0
pg_catalog.pg_auth_members.pg_auth_members_role_member_index 0 0 0
pg_catalog.pg_authid.pg_authid_oid_index 0 0 0
pg_catalog.pg_authid.pg_authid_rolname_index 0 0 0
pg_catalog.pg_autovacuum.pg_autovacuum_vacrelid_index 0 0 0
pg_catalog.pg_cast.pg_cast_oid_index 0 0 0
pg_catalog.pg_cast.pg_cast_source_target_index 160 51 51
pg_catalog.pg_class.pg_class_oid_index 71 71 71
pg_catalog.pg_class.pg_class_relname_nsp_index 30 17 17
pg_catalog.pg_constraint.pg_constraint_conname_nsp_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_conrelid_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_contypid_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_oid_index 0 0 0
pg_catalog.pg_conversion.pg_conversion_default_index 0 0 0
...

33. What else can we do with pg proctab?
Enable pg top to monitor remote databases by providing access to
the database system’s operating system process table.

34. pg top

35. __ __
/ ~~~/ . o O ( Thank you! )
,----( oo )
/ __ __/
/| ( |(
^ /___ / |
|__| |__|-"

36. . . . the fine print . . .
◮ Linux-only
◮ Developed on 8.3; still works on 8.4
◮ Download it from:
http://git.postgresql.org/gitweb?p=pg_proctab.git
◮ Change it:
git clone
git://git.postgresql.org/git/pg_proctab.git
◮ Patches welcome! We’ll be in the (Sn—H)ackers’ Lounge!

37. Acknowledgements
Haley Jane Wakenshaw
__ __
/ ~~~/
,----( oo )
/ __ __/
/| ( |(
^ /___ / |
|__| |__|-"

38. License
This work is licensed under a Creative Commons Attribution 3.0
Unported License. To view a copy of this license, (a) visit
http://creativecommons.org/licenses/by/3.0/us/; or, (b)
send a letter to Creative Commons, 171 2nd Street, Suite 300, San
Francisco, California, 94105, USA.