The document discusses using pg_proctab, a PostgreSQL extension that provides functions to query operating system process and statistics tables from within PostgreSQL. It demonstrates how to use pg_proctab to monitor CPU and memory usage, I/O, and other process-level metrics for queries. The document also shows how to generate custom reports on database activity and performance by taking snapshots before and after queries and analyzing the differences.

1. pg proctab
Accessing System Stats in PostgreSQL
Mark Wong markwkm@postgresql.org
Gabrielle Roth gorthx@gmail.com
PGWest Seattle (JDCon) 2009
Oct 16-18, 2009

2. Slides available on slideshare
http://www.slideshare.net/markwkm

3. Soooo . . .
You can query the PostgreSQL system catalog tables (e.g.
pg stat activity, pg stat all tables,
pg stat all indexes) to find out which queries are taking a long
time, which indexes are being scanned an unreasonable number of
times, etc.

4. Example:
portal=# SELECT datname, procpid, usename, current_query
FROM pg_stat_activity;
datname | procpid | usename | current_query
---------+---------+----------+-------------------------------------------------
portal | 5412 | markwkm | <IDLE>
portal | 5437 | postgres | SELECT datname, procpid, usename, current_query
: FROM pg_stat_activity;
(2 rows)

5. What if you want to know about the OS?
pg proctab provides a collection of four C stored functions:
◮ pg cputime
◮ pg loadavg
◮ pg memusage
◮ pg proctab

6. What can do you with pg proctab?
◮ Query operating system process table
◮ Query operating system statistics
◮ Processor time
◮ Load averages
◮ Memory usage
◮ Without escaping out to a shell!
◮ ...plus generate reports about timeslices

7. pg cputime() Example
SELECT *
FROM pg_cputime();
user | nice | system | idle | iowait
--------+--------+--------+------------+--------
681317 | 109924 | 395481 | 1466101128 | 462661
(1 row)

8. pg cputime() Column Description
From Linux kernel source code at
Documentation/filesystems/proc.txt:
user: normal processes executing in user mode
nice: niced processes executing in user mode
system: processes executing in kernel mode
idle: processes twiddling thumbs
iowait: waiting for I/O to complete

9. pg loadavg() Example
SELECT *
FROM pg_loadavg();
load1 | load5 | load15 | last_pid
-------+-------+--------+----------
0.99 | 0.78 | 0.67 | 27719
(1 row)

10. pg loadavg() Column Description
load1: load average of last minute
load5: load average of last 5 minutes
load15: load average of last 15 minutes
last pid: last pid running

11. pg memusage() Example
SELECT *
FROM pg_memusage();
memused | memfree | memshared | membuffers | memcached | swapused | swapfree | swapcached
---------+---------+-----------+------------+-----------+----------+----------+------------
3809140 | 224084 | 0 | 60656 | 2389700 | 76 | 8385844 | 0
(1 row)

12. pg memusage() Column Description
Paraphrased from Linux kernel source code at
Documentation/filesystems/proc.txt:
memused: Total physical RAM used
memfree: Total physical RAM not used
memshared: Not used, always 0. (I don’t remember why. . . )
membuffers: Temporary storage for raw disk blocks
memcached: In-memory cache for files read from disk
swapused: Total swap space used
swapfree: Memory evicted from RAM that is now temporary on
disk
swapcached: Memory that was swapped out, now swapped in but
still in swap

13. pg proctab() Example 1
SELECT datname, procpid, usesysid, usename, uid, username
FROM pg_stat_activity, pg_proctab()
WHERE procpid = pid;
datname | procpid | usesysid | usename | uid | username
---------+---------+----------+----------+-----+----------
markwkm | 27801 | 10 | markwkm | 500 | markwkm
dbt3 | 27787 | 16770 | postgres | 500 | markwkm
(2 rows)

14. pg proctab() Example 2
SELECT datname, procpid, processor, state, fullcomm
FROM pg_stat_activity, pg_proctab()
WHERE procpid = pid;
datname | procpid | processor | state | fullcomm
---------+---------+-----------+-------+------------------------------------------
markwkm | 27801 | 0 | R | postgres: markwkm markwkm [local] SELECT
dbt3 | 29325 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29327 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29333 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29328 | 2 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29329 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29324 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29331 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 27787 | 1 | S | postgres: postgres dbt3 [local] idle
(9 rows)

15. pg proctab() Partial Column Description
Everything from the operating system such as /proc/<pid>/stat,
/proc/<pid>/io and /proc/<pid>/cmdline as well as data
from PostgreSQL system catalog such as pg stat activity table
are available but we’ll only cover some of the fields here:
Informative:
◮ pid
◮ comm - filename of the executable
◮ fullcomm (/proc/<pid>/cmdline)
◮ uid
◮ username
Processor:
◮ utime - user mode jiffies
◮ stime - kernel mode jiffies
...

16. pg proctab() Partial Column Description (cont.)
Memory:
◮ vsize - virtual memory size
◮ rss - resident set memory size
I/O:
◮ syscr - number of read I/O operations
◮ syscw - number of write I/O operations
◮ reads - number of bytes which this process really did cause to
be fetched from the storage layer
◮ writes - number of bytes which this process really did cause to
be sent from the storage layer
◮ cwrites - number of bytes which this process caused to not
happen, by truncating pagecache

17. __ __ /
/ ~~~/ . o O | Let’s try something |
,----( oo ) | more useful. |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

18. __ __ /
/ ~~~/ . o O | Measuring performance |
,----( oo ) | of a query. |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

19. (You can find the following examples in the pg proctab contrib
directory.)

20. Create snapshot tables.
(Only need to do this once.)
i create-ps_procstat-tables.sql

21. Identify yourself.
SELECT *
FROM pg_backend_pid();
pg_backend_pid
----------------
4590
(1 row)

22. Take a snapshot before running the query
i ps_procstat-snap.sql
BEGIN
ps_snap_stats
---------------
1
(1 row)
COMMIT

23. Execute the query
Don’t focus too much on the actual query, the idea is that is you
want to collect statistics for a single query:
SELECT nation,
o_year,
Sum(amount) AS sum_profit
FROM (SELECT n_name AS nation,
Extract(YEAR FROM o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount
FROM part,
supplier,
lineitem,
partsupp,
orders,
nation
WHERE s_suppkey = l_suppkey
AND ps_suppkey = l_suppkey
AND ps_partkey = l_partkey
AND p_partkey = l_partkey
AND o_orderkey = l_orderkey
AND s_nationkey = n_nationkey
AND p_name LIKE ’%white%’) AS profit
GROUP BY nation,
o_year
ORDER BY nation,
o_year DESC;

24. Take a snapshot after running the query
i ps_procstat-snap.sql
BEGIN
ps_snap_stats
---------------
2
(1 row)
COMMIT

25. Calculate Processor Utilization
$ ./ps-processor-utilization.sh [pid] [before] [after]
$ ./ps-processor-utilization.sh 4590 1 2
Processor Utilization = 1.00 %
What’s going on (partially):
SELECT stime, utime, stime + utime AS total,
extract(epoch FROM time)
FROM ps_snaps a, ps_procstat b
WHERE pid = ${PID}
AND a.snap = b.snap
AND a.snap = ${SNAP1}
TIMEDIFF=‘echo "scale = 2; (${TIME2} - ${TIME1}) * ${HZ}" | bc -l‘
U=‘echo "scale = 2; (${TOTAL2} - ${TOTAL1}) / ${TIMEDIFF} * 100" | bc -l‘

26. Calculate Disk Utilization
$ ./ps-io-utilization.sh 4590 1 2
Reads = 276981
Writes = 63803
Reads (Bytes) = 2164604928
Writes (Bytes) = 508166144
Cancelled (Bytes) = 36880384
SELECT syscr, syscw, reads, writes, cwrites
FROM ps_snaps a, ps_procstat b
WHERE pid = ${PID}
AND a.snap = b.snap
AND a.snap = ${SNAP1}
TIMEDIFF=‘echo "scale = 2; (${TIME2} - ${TIME1}) * ${HZ}" | bc -l‘
U=‘echo "scale = 2; (${TOTAL2} - ${TOTAL1}) / ${TIMEDIFF} * 100" | bc -l‘

27. __ __ /
/ ~~~/ . o O | Creating Custom |
,----( oo ) | Reports! |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

28. __ __ /
/ ~~~/ . o O | Warning! Too much data |
,----( oo ) | to fit on screen! |
/ __ __/ /
/| ( |(
^ /___ / |
|__| |__|-"

29. Creating Reports: Section 1
Database : dbt3
Snapshot Start : 2009-04-18 00:43:56.716034-07
Snapshot End : 2009-04-18 00:45:17.031167-07
-------------------
Database Statistics
-------------------
Commits : 0
Rollbacks : 2
Blocks Read : 213295
Blocks Hit : 1679509

30. Creating Reports: Section 2
================
Table Statistics
================
------------------------------------------ -------- ------------ -------- ------------- --------- --------
Schema.Relation Seq Scan Seq Tup Read Idx Scan Idx Tup Fetch N Tup Ins N Tup Up
------------------------------------------ -------- ------------ -------- ------------- --------- --------
information_schema.sql_features 0 0 0 0 0
information_schema.sql_implementation_info 0 0 0 0 0
information_schema.sql_languages 0 0 0 0 0
information_schema.sql_packages 0 0 0 0 0
information_schema.sql_parts 0 0 0 0 0
information_schema.sql_sizing 0 0 0 0 0
information_schema.sql_sizing_profiles 0 0 0 0 0
pg_catalog.pg_aggregate 0 0 2 2 0
pg_catalog.pg_am 1 1 0 0 0
pg_catalog.pg_amop 0 0 19 46 0
pg_catalog.pg_amproc 0 0 11 11 0
pg_catalog.pg_attrdef 0 0 1 2 0
pg_catalog.pg_attribute 0 0 137 331 0
pg_catalog.pg_auth_members 0 0 0 0 0
pg_catalog.pg_authid 3 2 0 0 0
pg_catalog.pg_autovacuum 0 0 0 0 0
pg_catalog.pg_cast 0 0 160 51 0
pg_catalog.pg_class 3 747 101 88 0
pg_catalog.pg_constraint 0 0 0 0 0
pg_catalog.pg_conversion 0 0 0 0 0
pg_catalog.pg_database 5 12 0 0 0
pg_catalog.pg_depend 0 0 0 0 0
pg_catalog.pg_description 0 0 0 0 0
pg_catalog.pg_index 2 200 39 50 0
...

31. Creating Reports: Section 2 - Falling off the right side...
◮ N Tup Upd
◮ N Tup Del
◮ Last Vacuum
◮ Last Autovacuum
◮ Last Analyze
◮ Last Autoanalyze

32. Creating Reports: Section 3
================
Index Statistics
================
------------------------------------------------------------ -------- ------------ -------------
Schema.Relation.Index Idx Scan Idx Tup Read Idx Tup Fetch
------------------------------------------------------------ -------- ------------ -------------
pg_catalog.pg_aggregate.pg_aggregate_fnoid_index 2 2 2
pg_catalog.pg_am.pg_am_name_index 0 0 0
pg_catalog.pg_am.pg_am_oid_index 0 0 0
pg_catalog.pg_amop.pg_amop_opc_strat_index 12 36 36
pg_catalog.pg_amop.pg_amop_opr_opc_index 7 10 10
pg_catalog.pg_amproc.pg_amproc_opc_proc_index 11 11 11
pg_catalog.pg_attrdef.pg_attrdef_adrelid_adnum_index 1 2 2
pg_catalog.pg_attrdef.pg_attrdef_oid_index 0 0 0
pg_catalog.pg_attribute.pg_attribute_relid_attnam_index 0 0 0
pg_catalog.pg_attribute.pg_attribute_relid_attnum_index 137 331 331
pg_catalog.pg_auth_members.pg_auth_members_member_role_index 0 0 0
pg_catalog.pg_auth_members.pg_auth_members_role_member_index 0 0 0
pg_catalog.pg_authid.pg_authid_oid_index 0 0 0
pg_catalog.pg_authid.pg_authid_rolname_index 0 0 0
pg_catalog.pg_autovacuum.pg_autovacuum_vacrelid_index 0 0 0
pg_catalog.pg_cast.pg_cast_oid_index 0 0 0
pg_catalog.pg_cast.pg_cast_source_target_index 160 51 51
pg_catalog.pg_class.pg_class_oid_index 71 71 71
pg_catalog.pg_class.pg_class_relname_nsp_index 30 17 17
pg_catalog.pg_constraint.pg_constraint_conname_nsp_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_conrelid_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_contypid_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_oid_index 0 0 0
pg_catalog.pg_conversion.pg_conversion_default_index 0 0 0
...

33. What else can we do with pg proctab?
Enable pg top to monitor remote databases by providing access to
the database system’s operating system process table.

34. pg top

35. __ __
/ ~~~/ . o O ( Thank you! )
,----( oo )
/ __ __/
/| ( |(
^ /___ / |
|__| |__|-"

36. . . . the fine print . . .
◮ Linux-only
◮ Developed on 8.3; still works on 8.4
◮ Download it from:
http://git.postgresql.org/gitweb?p=pg_proctab.git
◮ Change it:
git clone
git://git.postgresql.org/git/pg_proctab.git
◮ Patches welcome! We’ll be in the (Sn—H)ackers’ Lounge!

37. Acknowledgements
Haley Jane Wakenshaw
__ __
/ ~~~/
,----( oo )
/ __ __/
/| ( |(
^ /___ / |
|__| |__|-"

38. License
This work is licensed under a Creative Commons Attribution 3.0
Unported License. To view a copy of this license, (a) visit
http://creativecommons.org/licenses/by/3.0/us/; or, (b)
send a letter to Creative Commons, 171 2nd Street, Suite 300, San
Francisco, California, 94105, USA.