Visualizing Relationships
Between Python Objects
Dinu C. Gherman
gherman@python.net
EuroPython Conference
2008-07-08, Vilnius

Motivation
• Make Python data structures visible
• Provide a simple tool for text/book authors
• Explore GraphViz capabilities

Typical Questions
• How does that data structure look like?
• How does it change after doing … to it?
• What does this module contain?

Background

Example 1
http://inst.eecs.berkeley.edu/
~selfpace/cs9honline/Q2/mutation.html

Example 2
http://www.linux-user.de/ausgabe/
2006/09/080-python-1/

Example 3
X = [1, 2, 3]
L = ['a', X, 'b']
D = {'x':X, 'y':2}
http://oreilly.com/catalog/9780596513986

Example 4
L = ['abc', [(1, 2), ([3], 4)], 5]
http://oreilly.com/catalog/9780596513986

Example 4 revisited
L
WANTED: ref
list: 0 1 2
str: abc list: 0 1 int: 5
tuple: 0 1 tuple: 0 1
int: 1 int: 2 list: 0 int: 4
int: 3

Example 3 revisited
L D
ref ref
list: 0 1 2 dict: keys
X
vals
WANTED: ref
str: a str: b list: 0 1 2 str: y str: x
int: 1 int: 3 int: 2

PDF References (internal)
2:0 1:0
202 19
9:0 8:0
205 19
3:0
106
/Page 5:0
97
7:0
10:0
108 53
11:0 /ExtGState
/Page 98
24:0 4:0
50 80 22:0
20:0 21:0
/Catalog /Pages 211
3116 21
14:0 15:0 6:0 /FontDescriptor
109 98 173
/Page /Font
25:0 /TrueType
trailer 23:0
163
19:0 36
18:0 98
109
26:0 /Page
90
13:0 12:0
208 20
17:0 16:0
208 20

”First System“
os
sys errno posixpath UserDict copy_reg
stat
Imports of module 'os' (simplified)

”First System“
fileinfo
dir mod mod mod
plugins expatbuilder guicocoa compatibility guiwx
mod
mod fileinfo_plugin_xml mod mod dir minidom
mod
fileinfo_plugin_py fileinfo_plugin_qt fileinfo_plugin_mp3 xml mod mod
mod mod mod __all__ __all__ mod
compiler keyword cStringIO tokenize sax test dom
mod parsers dir mod mod mod mod mod
pycodegen consts transformer mod expatreader _exceptions pulldom examples fileinfo_plugin_test test_fileinfo minicompat NodeFilter domreg xmlbuilder
visitor mod mod mod
future symbols syntax pyassem symbol parser token fileinfo_plugin_pdf fileinfo_plugin_ttf saxutils expat imports docstrings pythoncode unittest fileinfo copy
ast new dis misc pyPdf operator xmlreader urllib urlparse traceback plugging investigator
getopt
types pdf generic socket string handler linecache glob time datetime
struct utils filters _ssl _socket os fnmatch
re zlib copy_reg posixpath UserDict
stat
Module Space

GraphViz

Features
• Diagrams of abstract graphs and networks
• Automatic graph drawing
• GUIs, tools, libraries, and language bindings
• Developped by AT&T, Open Source
• DOT format (text) ➞ ps, pdf, svg, svgz, fig,
mif hpgl, pcl, png, gif, dia, imap, cmapx, …

Example: Hello World
Hello
digraph G {
Hello -> World
}
World

Example: Hello World 2
Hello
digraph G {
overlap=false;
Hello [shape="box", color="red"];
World [shape="circle", color="blue"];
Hello -> World [color="green"];
}
World

Example: FSM
digraph finite_state_machine {
rankdir=LR;
size="8,5"
node [shape = doublecircle]; LR_0 LR_3 LR_4 LR_8;
node [shape = circle];
LR_0 -> LR_2 [ label = "SS(B)" ];
LR_0 -> LR_1 [ label = "SS(S)" ];
LR_1 -> LR_3 [ label = "S($end)" ];
LR_2 -> LR_6 [ label = "SS(b)" ];
LR_2 -> LR_5 [ label = "SS(a)" ];
LR_2 -> LR_4 [ label = "S(A)" ];
LR_5 -> LR_7 [ label = "S(b)" ];
LR_5 -> LR_5 [ label = "S(a)" ];
LR_6 -> LR_6 [ label = "S(b)" ];
LR_6 -> LR_5 [ label = "S(a)" ];
LR_7 -> LR_8 [ label = "S(b)" ];
LR_7 -> LR_5 [ label = "S(a)" ];
LR_8 -> LR_6 [ label = "S(b)" ];
LR_8 -> LR_5 [ label = "S(a)" ];
}

Example: FSM
LR_4
S(A)
S(b)
S(b)
SS(b) LR_6
S(a)
LR_2 S(a) LR_8
SS(B) S(a)
SS(a) S(b)
LR_0 LR_5 S(b)
SS(S)
S(a) LR_7
LR_1 S($end)
LR_3

Pyrels

Features
• “Second system” – really simple, so far
• Recursive func. collecting nodes and edges
• Converting to GraphViz nodes and edges

Pyrels Profile
$ cd pyrels
$ fileinfo -p rel -a lc:nmtlines:nclasses:ndefs:ncalls
scripts/pyrels $(find src -name "*.py")
lc nmtlines nclasses ndefs ncalls path
58 14 0 1 9 scripts/pyrels
0 0 0 0 0 src/pyrels/__init__.py
411 70 1 7 163 src/pyrels/pyrels2dot.py
0 0 0 0 0 src/pyrels/test/__init__.py
379 150 0 10 105 src/pyrels/test/pdfposter.py
46 13 0 2 10 src/pyrels/test/test_all.py
99 68 2 3 17 src/pyrels/test/test_basic.py
90 26 1 2 15 src/pyrels/test/test_commandline.py
55 21 1 2 18 src/pyrels/test/test_cyles.py
148 62 1 6 63 src/pyrels/test/test_multi.py
193 66 5 14 50 src/pyrels/test/test_other.py
20 7 0 2 6 src/pyrels/test/testutils.py
12 4 0 1 1 src/pyrels/utils.py
1511 501 11 50 457 total

Example: String Caching
ref
1 v str: XXXXXXXXXXXXXXXXXXXXX
v = 'X' * 21 ref
u str: XXXXXXXXXXXXXXXXXXXXX
u = 'X' * 21
s = 'X' * 20 3
t = 'X' * 20 s ref
str: XXXXXXXXXXXXXXXXXXXX
ref
t
2
digraph G {
name1775744 [label="v", shape="ellipse"];
str3376528 [label="str: XXXXXXXXXXXXXXXXXXXXX", shape="box"];
name1845536 [label="u", shape="ellipse"];
str3284144 [label="str: XXXXXXXXXXXXXXXXXXXXX", shape="box"];
name1686816 [label="s", shape="ellipse"];
str3376480 [label="str: XXXXXXXXXXXXXXXXXXXX", shape="box"];
name1798720 [label="t", shape="ellipse"];
str3376480 [label="str: XXXXXXXXXXXXXXXXXXXX", shape="box"];
name1686816 -> str3376480 [label="ref"];
name1845536 -> str3284144 [label="ref"];
name1798720 -> str3376480 [label="ref"];
name1775744 -> str3376528 [label="ref"];
}

Example: Singleton None
digraph G {
1 2 name3498784 [label="aDict", shape="ellipse"];
dict3660224 [label="dict: | {keys|vals}|
{<k0>|<v0>}", shape="record"];
none1 = None none [label="None", shape="box"];
none2 = None none [label="None", shape="box"];
aList = [None] name3499008 [label="aList", shape="ellipse"];
list3662272 [label="list: | <0> 0",
aTuple = (None, None) shape="record"];
aDict = {None:None} none [label="None", shape="box"];
name3498976 [label="none2", shape="ellipse"];
none [label="None", shape="box"];
dict: name3498944 [label="none1", shape="ellipse"];
aDict
ref
keys vals
none [label="None", shape="box"];
name3317728 [label="aTuple",
shape="ellipse"];
tuple3334392 [label="tuple: | <0> 0|<1> 1",
list: shape="Mrecord"];
ref
aList none [label="None", shape="box"];
0 none [label="None", shape="box"];
none2
ref
None dict3660224:k0 -> none [label=""];
ref dict3660224:v0 -> none [label=""];
3 list3662272:0 -> none [label=""];
none1 tuple3334392:0 -> none [label=""];
tuple3334392:1 -> none [label=""];
name3498784 -> dict3660224 [label="ref"];
tuple: name3499008 -> list3662272 [label="ref"];
aTuple
ref
0 name3498976 -> none [label="ref"];
name3498944 -> none [label="ref"];
1 name3317728 -> tuple3334392 [label="ref"];
}

Example: Module Content (pyPdf)
ref
utils module: pyPdf.utils
import pyPdf
ref locals().update(pyPdf.__dict__)
__doc__ None
del pyPdf
ref
pdf module: pyPdf.pdf
ref
PdfFileWriter new style class: PdfFileWriter
ref
__name__ str: 'pyPdf'
ref
filters module: pyPdf.filters
ref
__file__ str: '/usr/local/lib/python2.5/site-packages/pyPdf/__init__.pyc'
ref
generic module: pyPdf.generic
ref list:
__path__
0 str: '/usr/local/lib/python2.5/site-packages/pyPdf'
ref list:
__all__
0 str: 'pdf'
ref
PdfFileReader new style class: PdfFileReader

Future

Issues
• Testing, debugging & refactoring, …
• Detecting cycles
• Fiddling with GraphViz params (overlap, …)
• Adding other relationships:
- Module Imports or not)
- Inheritance (UML-like
- Call graphs
-…
• Smart graph pruning

Summary
• Development only barely started (v. 0.1)
• Useful already for visualising data structures
• Provides certain insights for Python newbies
• Feedback is warmly welcome!

Links
• http://www.dinu-gherman.net/tmp/pyrels-0.1.0.tar.gz
• http://www.graphviz.org

Questions?

Questions!
• GraphViz editor with branch fold-in/out?
• How to get full path for DOT edges
(in plain format)?