Presentation to the Open Source Developers Conference 2012. Code snippets and application samples written in Python.

1. 20 Cool Things I
have done in Python
Maurice Maneschi
RedWaratah.com
RedWaratah.com
Open Source Solutions for Business

2. Who am I
●
I am a former mainframe developer
●
I work for Financial Advisory company
●
All Microsoft and Excel
●
However, Python gives me a tool kit that easily
solves dvierse, apparently intractable data and
integration problems
●
I am a hero for the problems I can solve
RedWaratah.com
Open Source Solutions for Business

3. Format
●
I will fly through 20 examples of what I have done
●
Slow me down for the interesting slides, speed me up
for the boring ones
●
For details, grab me afterwards over coffee
●
See the key libraries on the bottom right
– Search the web for samples
– Send me email anything@redwaratah.com
RedWaratah.com
Open Source Solutions for Business

4. Configuration
●
Python header
●
Command line
●
INI file
●
Windows Registry
RedWaratah.com
Open Source Solutions for Business

5. Configuration – in the header
RedWaratah.com
Open Source Solutions for Business

6. Configuration – command line
RedWaratah.com
Open Source Solutions for Business
Using argparse library

7. Configuration – ini file
RedWaratah.com
Open Source Solutions for Business
Using configparser library

8. Configuration – Windows registry
RedWaratah.com
Open Source Solutions for Business
Using winreg library

9. Using DLLs
●
I had to debug a Windows program in
production without a debugger
●
Using Python ctypes, I called each subroutine
until I discovered the defect
RedWaratah.com
Open Source Solutions for Business

10. Using DLLs
RedWaratah.com
Open Source Solutions for Business
Using ctypes library

11. Driving a website
●
Single window website for MSIE only
●
Running reports is fiddly, same report routinely
is painful
●
Python proxy server to log the responses
●
I built an application that mimicked the
browser and download the reports in PDF and
Excel
RedWaratah.com
Open Source Solutions for Business

12. Driving a web site – part 1
RedWaratah.com
Open Source Solutions for Business
Using urllib, html, http and re libraries

13. Driving a web site – part 2
RedWaratah.com
Open Source Solutions for Business

14. Regular Expression Log Parsing
●
MS IIS logs are nightly copied to a folder
●
Python uses RE to capture log ons and report runs
●
Converts IP address to client name via firewall config
●
Saves the data to an SQLite database
●
This data is provided regularly to the wiki article on
active clients
RedWaratah.com
Open Source Solutions for Business

15. Regular Expression Log Parsing
RedWaratah.com
Open Source Solutions for Business
Using re, sqlite3 libraries

16. Regular Expression Magic
●
We have the Mediawiki (Wikipedia) for documentation
and TestTrack for defect tracking
●
Python does a nightly read of the wiki articles and builds
a huge regular expression (3 actually) of all titles
●
Python does a nightly extract of the contents of
TestTrack to HTML and hyper links any reference to the
articles back to the wiki
RedWaratah.com
Open Source Solutions for Business

17. Regular Expression Magic - pt1
RedWaratah.com
Open Source Solutions for Business
Using re, subprocess libraries

18. Regular Expression Magic – pt 2
RedWaratah.com
Open Source Solutions for Business

19. Scheduled Task Status
●
We have about 30 scheduled tasks that run
each day (or more frequently)
●
Python scans the status return codes every 5
minutes and puts the results into a CSV files
●
MSDN defines the format of the job files and
Python can read these formats
●
Nagios warns if any status codes are not 0
RedWaratah.com
Open Source Solutions for Business

20. Scheduled Task Status
RedWaratah.com
Open Source Solutions for Business
Using struct, csv libraries

21. Scheduled Task Status
RedWaratah.com
Open Source Solutions for Business

22. Static Data Checker
●
Our key software tool does not audit every
Static Data table
●
Changes can happen inadvertently
●
Python reads every nominated table, pickles
each row and pops the results into an SQLite
table
●
Daily report of new, changed or deleted rows
RedWaratah.com
Open Source Solutions for Business

23. Static Data Checker
Row
Static data
Pickle
Key Value
SQlite
RedWaratah.com
Open Source Solutions for Business
Using adodb, pickle, sqlite3 libraries

24. Static Data Checker
RedWaratah.com
Open Source Solutions for Business

25. Static Data Checker
RedWaratah.com
Open Source Solutions for Business

26. Conan: Flagging required email
●
Purchased Software is meant to send email
confirming financial transactions that we track
●
It fails to correctly ascertain settlements
●
Python creates a web site that shows which
settlements have to be sent
●
Annotations
●
Validates against the transmission logs
RedWaratah.com
Open Source Solutions for Business

27. Conan
RedWaratah.com
Open Source Solutions for Business
Using html, cgi, sqlite3, adodb libraries

28. Conan
RedWaratah.com
Open Source Solutions for Business

29. Nagios
●
I spoke three years ago about Python for
writing Nagios plugins
●
Our Nagios server got virtualised two years
ago
●
We have a Ubuntu desktop PC showing the
Nagios and RateFeed status
●
Sounds an alarm when new errors emerge
RedWaratah.com
Open Source Solutions for Business

30. Nagios
RedWaratah.com
Open Source Solutions for Business
Using numerous libraries

31. Tkinter
●
I love it
●
Not sexy or super responsive, but fast to code
and reliable
●
Six different Tkinter applications
RedWaratah.com
Open Source Solutions for Business
Using tkinter, tkmessagebox, tkdialog libraries

32. Tkinter – example 1
RedWaratah.com
Open Source Solutions for Business

33. Tkinter – example 2
RedWaratah.com
Open Source Solutions for Business

34. Tkinter – example 3
RedWaratah.com
Open Source Solutions for Business

35. Tkinter – example 4
RedWaratah.com
Open Source Solutions for Business

36. Tkinter – example 5
RedWaratah.com
Open Source Solutions for Business

37. RateFeed
●
Three Tkinter based applications written in Python
●
Download rates from Reuters via FTP
●
Calculate curves from a detailed INI file
●
Keep results and logs in two SQLite files
●
Write to two servers via DCOM
●
Respond to queries via RPC
RedWaratah.com
Open Source Solutions for Business

38. RateFeed
RedWaratah.com
Open Source Solutions for Business
Using numerous libraries

39. RateFeed Export
●
Managers cannot understand our RateFeed
●
Python generates two exports a day in Excel
– Import openpyxl
●
Tabs for different curve types
●
Columns for incoming data, formulae and
outgoing data
●
Used to get sign off on curve changes
RedWaratah.com
Open Source Solutions for Business

40. RateFeed Export
RedWaratah.com
Open Source Solutions for Business
Using openpyxl, sqlite3 libraries

41. Firewall settings
●
We open our firewall to registered customers only
●
Their IP addresses change as they change ISP, proxy
settings, location etc.
●
Checking IP addresses quite error prone.
●
Now after each change, we export our firewall configuration
to a set location
●
On change, Python updates our wiki with current client data
RedWaratah.com
Open Source Solutions for Business

42. Firewall settings
RedWaratah.com
Open Source Solutions for Business
Using re, smtplib libraries

43. Crystal Report Management
●
We have 300 Crystal Reports
●
Our test server databases have different names to production,
but Crystal Reports requires each table to be reset to use a
different database
●
Python shifts changes in the production version and the test
version by using the Crystal API to fix every table
●
All via accessing the DCOM interface into Crystal Reports
RedWaratah.com
Open Source Solutions for Business

44. Crystal Reports Management
RedWaratah.com
Open Source Solutions for Business
Using win32com.client library (pywin32)

45. Change Request Management
●
Change requests go into our defect management system
●
When assessed and approved, the go into the work list.
●
Python picks up newly assessed work and adds it to the
end of Planner XML files for project management
●
Email reports show projections on updates to these plans
RedWaratah.com
Open Source Solutions for Business

46. Change Request Management
RedWaratah.com
Open Source Solutions for Business
Using sax, adodb libraries

47. Crystal Report Testing
●
Crystal Reports used for confirmation of trading
●
These reports have loads of business logic
●
If we need to make a change, what is the impact
●
Python re runs every such confirmation for last
12 months under the new and old template,
exports to Excel and flags any differences
●
Tkinter GUI shows progress
RedWaratah.com
Open Source Solutions for Business

48. Crystal Report Testing
●
Grabs parameters from
past confos
●
Applies them to the new
and old template
●
Exports to HTML and
compare
●
Note differences and
store
RedWaratah.com
Open Source Solutions for Business
Using adodb, html, win32com.client, tkinter libraries

49. Risk scenario building
●
Have special Risk reports that model the impact of
different current shifts
●
28 currencies x 11 scenarios x 15 points per currency
●
GUI is slow, errors high
●
GUI exports to XML and imports
●
Python generates XML with all required scenarios built
from an INI file
●
Tkinter interface
RedWaratah.com
Open Source Solutions for Business

50. Risk scenario building
Given this...
RedWaratah.com
Open Source Solutions for Business

51. Risk scenario building
...builds this:
RedWaratah.com
Open Source Solutions for Business
Using sax, win32com.client, tkinter libraries

52. Backup tool
●
Test servers are not backed up
●
Python scheduled task puts contents of key
folder and anything linked to it into a ZIP file
and puts this on the main file server
●
Follows file links
RedWaratah.com
Open Source Solutions for Business

53. Backup tool
RedWaratah.com
Open Source Solutions for Business
Using os, win32com.client, zip libraries

54. Disaster Recovery
●
Much of our DRP is in soft copy
●
The detail is in a Media Wiki database and
TestTrack database
●
Python exports all MediaWiki articles and
TestTrack items into HTML
●
These are burned to DVDs quarterly along
with ZIPs from the backup tool
RedWaratah.com
Open Source Solutions for Business

55. Disaster Recovery
RedWaratah.com
Open Source Solutions for Business
Using subprocess, urllib libraries

56. RESTful Interface
●
I wrote a dashboard using the Underscore
backbone (in JavaScript). It wanted to interact
via Restful interface
●
I created such an interface as a Python CGI
with the Apache rewriting
RedWaratah.com
Open Source Solutions for Business

57. RESTful Interface
RedWaratah.com
Open Source Solutions for Business
Using cgi, sqlite3, xmlrpc libraries

58. RESTful Interface
RedWaratah.com
Open Source Solutions for Business

59. ImageMagick Scripting
●
I have a Python script running unsupervised
on the server
●
It responds to RPC requests
●
Python drives ImageMagick to build a visual
representation
●
Also, did a target for a competition
RedWaratah.com
Open Source Solutions for Business

60. ImageMagick Scripting
RedWaratah.com
Open Source Solutions for Business
Using subprocess, os libraries

61. ImageMagick Scripting
RedWaratah.com
Open Source Solutions for Business

62. Test Server Symbol Management
●
Each server has different resource locations
– Server name is in the UNC
●
These are in an INI file
●
Python library to access this INI file
●
Caches the results in a pickle file to reduce
network traffic and improve resiliency
RedWaratah.com
Open Source Solutions for Business

63. Test Server Symbol Management
RedWaratah.com
Open Source Solutions for Business

64. Rate Checker
●
Many rates go in each day at different times
●
Required to confirm the success of rates
●
Python compares today's rates with yesterday,
a week ago and last month end
●
Reports via email
●
Emails are logged as proof of our diligence
RedWaratah.com
Open Source Solutions for Business

65. Rate Checker
RedWaratah.com
Open Source Solutions for Business
Using smtplib, adodb, configparser libraries

66. Rate Checker
RedWaratah.com
Open Source Solutions for Business

67. Special Thanks to...
●
Guido Van Rossum – keeping it simple and powerful
●
Mark Hammond – Making it work on Windows
●
Libaries
– Tkinter for easy UI
– Openpyxl for Excel import and export
– email and smtplib for sending email
– Urllib and httplib for capture and parsing
– Re for regular expressions
– argparse, configparser and winreg for parameterisation
– Subprocess for process management
●
And the Python community for sample code all over the internet
RedWaratah.com
Open Source Solutions for Business

68. Go forth and code!
RedWaratah.com
Open Source Solutions for Business