This document discusses creative data analysis using Python. It provides an overview of useful Python tools for data analysis and visualization including Matplotlib, NumPy, Pandas, and SQL Alchemy. It emphasizes extracting small insights from large data sets, presenting truth without deception, and making analysis flexible, readable, and understandable for audiences. Examples are provided for Matplotlib annotations, layered charts, shape placement, and SQL querying using dictionaries. The document encourages iteration and exploring what Python can do for flexible data analysis.

1. Creative Data Analysis
with Python
Grant Paton-Simpson
Senior Data & Implementation Specialist
Optima Corporation
Creator of SOFA Statistics

2. Great Python Tools Available
●
Matplotlib (see Creating Interactive Applications in
Matplotlib by Jake Vanderplas http://vimeo.com/63260224)
●
Numpy
●
Python sets, ordered dicts, named tuples
●
PANDAS
●
SQL Alchemy, adodbapi, dbapi
●
Easy text processing
(e.g. HTML)
●
CSV
●
Python!

3. Get Inspired!

4. Flexibility

5. Use Freedom Responsibly!
See http://blog.revolutionanalytics.com/2010/04/when-infographics-go-bad.html etc
and http://www.netmagazine.com/features/seven-dirty-secrets-data-visualisation

6. The point is in there somewhere –
honest!

7. Simple can be best

8. Make a Simple Point
Make complex things simple
● Extract small information from large data
● Present truth, do not deceive
●
http://www.dataists.com/2010/10/...
… what-data-visualization-should-do-simple-small-truth/

9. Make it easy for the audience

10. Flexible analysis
needs flexible tools

11. Matplotlib can do it

12. is your friend
●
How to shift a legend outside the plot
●
How to have a major and minor axis
●
How to shift x axis labels to the middle of a bar
●
●
How to position a triangle a certain percentage along
the x axis
How to apply a heat map to circles etc etc

13. Annotations, layers, shape placement and much
more!

14. Example with Percentile Lines

15. Iterate

16. Colour adds meaning

17. The power of ...
●
Planned
non-obsolescence
SQL
●
Nothing you can't do
●
Scales
●
Can decouple
●
SQL Alchemy, dbapi, adodbapi etc
●
●
In my current role, I use SQL with safe data where there is no
significant potential for dangerous input. In this case, the most
readable and maintainable way of building SQL strings is to use
dicts and string interpolation:
“SELECT %(fld1)s, %(fld2)s FROM ...” % {“fld1”:
dest_arrive_time, “fld2”: dest_depart_time}.
But this is not a good habit otherwise – search on “SQL injection”
if you don't know why!
Read data using dicts: row[“dest_x”]

18. dbapi
●
con = db.connect(host=...)
●
cur = con.cursor()
●
sql = “SELECT fname
FROM data
WHERE age > 40”
●
cur.execute(sql)
●
print(“, ”.join(x[“fname”] for x in cur.fetchall()))

19. The power of ...
●
●
●
●
●
Text
Nothing you
can't do
HTML
Easy to display tabular data,
hyperlinks, subreports
Clean HTML can be opened
as documents and spreadsheets
Conditional highlighting e.g.
class_str = “class = 'highlight'
if age > 10 else ””
html.append(“<td %(class_str)s>%(age_val)</td>”)

20. Imagine, create, iterate ...