SOFA Statistics - Developing, packaging, & promoting a Python open source project

SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand SOFA Statistics Developing, packaging, & promoting a Python open source project Grant Paton-Simpson sofastatistics.com

SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Overview
Introducing the SOFA project
wxPython
Launchpad, Freshmeat and Sourceforge
Making debian packages
Making a Windows installer

SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand The SOFA Project
Statistics Open For All
Statistics, analysis & reporting
Open source (AGPL3) – no restrictions on commercial use etc
Cross-platform
Video (please interrupt with questions)
Website
Appearances matter
Quick taste of what SOFA is and why they might want it
Easy to download

SOFA Architecture SQLite MySQL MS Access ... SOFA Linking not importing Scripts ready to glue together etc HTML (spreadsheet- friendly)

HTML Output
Tree-based for each dimension
Created an artificial limit of 5000 cells
Scales linearly and Python not the bottleneck

Import from Excel and SPSS etc
Looking at Statlib for basic statistical tests
Want to choose a library for charting
Want to add own interactive visualisations for explaining t-tests etc
Not trying to compete with R (or RK Ward)
Focus on making the most common needs easy to satisfy
Plugin extensions for rest
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Next Steps

50% coding script-based solutions
50% adding GUI
Another 50% enabling installation
Yet another 50% setting up website, bazaar, Launchpad, Sourceforge, Freshmeat, making screencast for ShowMeDo, writing articles, giving presentations etc
Why?
Complete control over the tools I rely on as an analyst
Open source is a positive contribution
Nice to have a large, long-term project to keep refining
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Running a Project

Python-focused
I benefited from the screencast on debian packaging
Great way of coping with bandwidth if your project takes off
Video is like screenshots on steroids – gives people a real taste
Used gtk-recordmydesktop which is very easy
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand ShowMeDo Videos

wxPython GUI Toolkit
Cross-platform and native widgets
Ubuntu (Dust theme) Windows XP

wxPython in Action – Great book
Mailing list (with Robin Dunn a very regular contributor)
Lots of online documentation (but can be a little bit terse)
There is a GUI for making GUIs but I prefer handcoding
Clean
Can make GUI plugins e.g. SOFA database configuration
Lots of sophisticated, configurable widgets
Was able to make a data entry table work like I thought it should e.g. new row has column label of … , behaviour when tabbing
Focus on grid control
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Documentation & Community

When linking to data need to subclass wx.grid.PyGridTableBase
Need to implement methods such as GetNumberCols()
With flexibility and power comes some complexity to handle
The internal documentation is sometimes longer than the code itself
700 lines to make the data grid you saw in the video (including validation, ability to add new rows and edit values etc)
1/6 were comments
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand wx.Grid

Example grid code

SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Custom Controls

When creating a custom control e.g. a composite control, need to subclass wx.PyControl
Getting good documentation takes some resourcefulness – having a good working example the key
The custom control is utilised as part of a custom cell editor
To make a custom cell editor need to subclass wx.grid.PyGridCellEditor
Adequately documented
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand wx.PyControl

Need to be able to cope with enormous tables – as large as MySQL etc can throw at us
GetValue() best with unique key (using ORDER BY and LIMIT does not scale)
Also need caching
Can now scroll large tables with lots of columns e.g. 200,000 records and 20 columns
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Grid Performance

Option of label display e.g. “Male” not 1
Conditional formatting e.g. all values > 1000 red etc
Choice of toolkit very important - can it support what you want to do or will you hit a wall?
If I wanted to display sparklines or pie charts as cells in a table … could I?
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Grid Plans

Not either/or – can still have project releases in Sourceforge as well plus a Freshmeat project
Why Launchpad?
Ubuntu and Canonical good fit for me (my main platform)
Uses modern distributed version control software (Bazaar)
Lots of developer resources and ongoing development
Corporate backing – a safe bet
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Launchpad

Modern distributed version control
Built in Python
Documentation seemed pitched at my level
Very simple commands
bzr add importer.py
bzr commit -m “Added ability to import from Excel”
bzr push
Can grow with the project
Uses modern Ubuntu notifications
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Bazaar

Putting the open into open source
Also written in Python
A web viewer for bazaar projects
Loggerhead

Initially very daunted – could I even do this?
Found ShowMeDo video by Austrian Horst Jens
His example was a Python project so very similar requirements
Ended up with very detailed step-by-step guidelines for packaging SOFA Statistics
Have small handful of files with content I reuse for each package e.g. rules
Using AGPL so needed to edit copyright file by hand
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Making Debian Package (for Ubuntu)

“rules” is mainly for shifting files into destination
NB installing application for all users, not a given user
Files are put into /usr/share/pyshared/sofa/...
Any files needed by an individual user are transferred during first use of application /home/username/sofa/...
“control” is for versioning, developer details, and dependencies e.g. Depends: python (>=2.6.2-0ubuntu1), wx-common (>=2.8.9.2-2),
distro-specific so may not be a simple matter of using alien to make rpms
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Making Debian Package cont ...

Nullsoft Scriptable Install System. Free. Used by Firefox, OpenOffice etc
Weird language – cross between PHP and assembler
Macros can take parameters but functions cannot
No functions inside sections
Plenty of documentation etc but best to start and then extend
One glitch – was including all files in installer, whether or not used by version (solution – different info has to be in different files which are then included)
Issue – file size. Including mysqldb, numpy, wxpython, sqlite, python
Put program in Program Files and user files in Documents and Settingsusernamesofa...
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand NSIS Windows Installer

Running a new project can be exciting
Lots of new learning
Someone has to “take out the trash” (hint – it's you ;-))
Phenomenal resources available in open source world
Python packaging still a work-in-progress but should shake down
SOFA Paton-Simpson & Associates Ltd Auckland, New Zealand Final Thoughts

http://research213.byethost33.com	14
http://research213.comoj.com	3
http://research213.tumblr.com	3
http://www.slideshare.net	2

SOFA Statistics - Developing, packaging, & promoting a Python open source project

by Grant Paton-Simpson on Jun 22, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 22

Statistics

SOFA Statistics - Developing, packaging, & promoting a Python open source project — Presentation Transcript