1.
Python and Machine Learning by Semen A. Trygubenko

2.
Machine learning around us <ul><li>Voice recognition

3.
Spam filtering

4.
Player ranking in online games

5.
Vehicle stability systems

6.
Computer vision (barcode, fingerprint and number plate readers)

7.
Optimisation </li></ul>

8.
Learning? Machine Learning? <ul><li>Getting better at a task through practice

9.
The act of remembering: data / experience

10.
Generalisation, similarity and new inputs

11.
Learning and flexibility: adaptation </li></ul>

12.
Objective function game <ul><li>Supervised learning, training </li><ul><li>Regression

13.
Classification: features, decision boundaries </li></ul><li>Reinforcement learning

14.
Semi-supervised learning

15.
Evolutionary learning, fitness

16.
Unsupervised </li></ul>

18.
Everything is miscellaneous <ul><li>Clustering

19.
Graphical models

20.
Artificial neural networks

21.
Kernel methods

22.
Dimensionality reduction

23.
Optimisation </li></ul>

24.
Interdisciplinary & as a branch of CS

25.
ML is a vaaast field

26.
ML Tools: Python <ul><li>NLTK

27.
FANN

28.
Orange

29.
PyMC

30.
PyML

31.
LIBSVM

32.
PyBrain

33.
ffnet </li></ul><ul><li>MDP

34.
Shogun toolbox

35.
Theano

36.
mlpy

37.
Elefant

38.
Bayes Blocks

39.
Monte Python

40.
hcluster </li></ul><ul><li>Plearn

41.
Pycplex

42.
pymorph </li></ul>

43.
We need total coverage ...

44.
Nothing about everything <ul><li>A sprint usability study:

45.
(1) assess how easy it is </li><ul><li>to install and obtain source code;

46.
to get going with toy examples </li></ul><li>(2) check the quality of documentation and source code

47.
(3) Establish where the project is in its lifecycle

48.
(4) Demo stuff that I think is cool </li></ul>

49.
Testbed <ul><li>OS and package repo: </li><ul><li>Debian

50.
Ubuntu 10.04 </li></ul><li>Make 3.81: </li><ul><li>$make install

51.
$make source </li></ul><li>Free as in freedom </li></ul>

52.
Orange <ul><li>A machine learning and data mining suite

53.
Visual programming framework + scriptable environment

54.
Orange canvas: widgets and channels

55.
Communication via tokens

56.
Implements sampling, filtering, scaling, discretisation, regression, classification, clustering, scoring functions, SVMs </li></ul>

57.
Orange <ul><li>Faculty of computer and information science

58.
AI lab, University of Ljubljana, Slovenia

59.
(L)GPL

60.
C++ components accessible from Python

61.
98K LOCs in Python, 66K LOCs in C++

62.
700 revisions in 2010, 14 developers

63.
10000 commits since 2003 when migrated from cvs </li></ul>

64.
Orange # Homepage: http://www.ailab.si/orange/ # Dependencies: Python, PythonWin, NumPy, PyQt, PyQwt... .PHONY: install install: repo sudo apt-get install orange-svn python-orange-svn .PHONY: source source: svn checkout http://www.ailab.si/svn/orange/trunk Orange .PHONY: repos repos: sudo echo -e &quot;deb http://www.ailab.si/orange/debian lenny main
deb-src http://www.ailab.si/orange/debian lenny main
deb http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu lucid main
deb-src http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu lucid main&quot;| sudo tee -a /etc/apt/sources.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5BB92C09DB82666C && sudo apt-get update sudo apt-get install python2.5

65.
import orange data = orange.ExampleTable(&quot;in&quot;) print data.domain for item in data: print item orange.saveTabDelimited(&quot;out.tab&quot;,data) I/O

66.
selection = orange.MakeRandomIndices2(data, 0.03) sample = data.select(selection, 0) sample.save(&quot;sample.tab&quot;) Basic statistics

67.
Learners and classifiers classifier = orange.BayesLearner(data) print classifier(newItem)

69.
MDP <ul><li>Modular toolkit for data processing

70.
Supervised and unsupervised

71.
PCA and ICA, Slow feature analysis, LLE, restricted Boltzmann machine

72.
Hierarchical networks

73.
Support vector machines (through wrappers around Shogun and LIBSVM)

74.
Concept of nodes and flows; basic are parallelized </li></ul>

75.
MDP <ul><li>Pietro Berkes, Rike-Benjamin Schuppner, Niko Wilbert, Tiziano Zito, community contributions

76.
Institute for Theoretical Biology of the Humboldt University, Berlin

77.
Schuppner, Wilbert and Zito are active, 150 commits this year

78.
LGPL, version 2.6 released in May this year

79.
30K LOCs of Python, 3K of comments in the code! </li></ul>

80.
MDP # Homepage: http://mdp-toolkit.sourceforge.net/ # Dependencies: NumPy, SciPy .PHONY: install install: sudo apt-get install python-mdp .PHONY: source source: git clone git://mdp-toolkit.git.sourceforge.net/gitroot/mdp-toolkit/mdp-toolkit

81.
Dimensionality reduction import mdp x = mdp.numx_rand.random((100, 25)) # 25 variables, 100 observations y = mdp.pca(x) z = mdp.fastica(x, dtype='float32')

82.
Nodes: training and usage n = mdp.nodes.PCANode() n.train(x) # learn PC of x n.stop_training() print n1.output_dim print n1.explained_variance z = n.execute(y) # project y on PC learned in training

83.
Inverting the flow print n.is_invertible() # true for PCA node print n.inverse(z) # get y back

84.
Flows: feed-forward architectures flow = mdp.Flow([ mdp.nodes.PCANode(output_dim=5) ,mdp.nodes.CuBICANode() ]) flow = mdp.nodes.PCANode(output_dim=5) + mdp.nodes.CuBICANode() flow.train(x)

85.
More dimensions: defs <ul><li>Layer = wrapper for a set of nodes trained and executed in parallel

86.
FlowNode = a node with internal structure

87.
Switchboard = DS for arbitrary routing </li></ul>

88.
Hierarchical networks a = mdp.nodes.PCANode() b = mdp.nodes.SFANode() c = mdp.hinet.FlowNode( mdp.Flow([a,b]) ) layer = mdp.hinet.Layer([n, c])

89.
PyMC <ul><li>Markov chain Monte Carlo for Python

90.
Can fit Bayesian statistical models with MCMC

91.
Large suite of statistical distributions

92.
Building blocks to construct probability models: stochastic, deterministic and potential

93.
Python for scalar vars, numpy and hand-optimised Fortran code for arrays </li></ul>

94.
PyMC <ul><li>Christopher Fonnesbeck, David Huard, Anand Patil

95.
USA

96.
MIT lic

97.
3 developer, 100 commits this year, all in Jan

98.
Version: 2.1 released in Jan

99.
26K LOCs of Python, 10K of Fortran and 3K LOCs of C </li></ul>

100.
PyMC # Homepage: http://code.google.com/p/pymc/ # Dependencies: NumPy,SciPy,matplotlib,pytables,pydot,nose .PHONY: install # grr... 2.1alpha is missing gp_submodel.py and step_methods.py ##dir:=pymc-2.1alpha dir:=pymc install: source ##${dir}.tar.gz sudo apt-get install python-dev gcc gfortran ## tar -xzvf $< cd ${dir} && python setup.py config_fc --fcompiler gnu95 build && sudo python setup.py install ##pymc-2.1alpha.tar.gz: ## wget ##http://pymc.googlecode.com/files/pymc-2.1alpha.tar.gz .PHONY: source source: svn checkout http://pymc.googlecode.com/svn/trunk/ pymc

101.
Model module, part 1 x = numpy.array([-.86,-.3,-.05,.73]) alpha = pymc.Normal('alpha',mu=0,tau=.01) beta = pymc.Normal('beta',mu=0,tau=.01) @pymc.deterministic def theta(a=alpha,b=beta): return pymc.invlogit(a+b*x)

102.
Model module, part 2 # Binomial likelihood for data d = pymc.Binomial( 'd' ,n=numpy.ones(4,dtype=int)*5 ,p=theta ,value=numpy.array([0.,1.,3.,5.]) ,observed=True )

103.
Sampling from a distribution import pymc import model S=pymc.MCMC(model,db='pickle') S.sample(iter=10000,burn=5000,thin=2)

104.
PyML <ul><li>A Python machine learning package

105.
Interactive OO framework, focuses on SVM and other kernel methods

106.
tools for feature selection, model selection, syntax for combining classifiers

107.
methods for assessing classifier performance </li></ul>

108.
PyML <ul><li>Asa Ben-Hur, Colorado, USA

109.
Depends: matplotlib, numpy

110.
1 developer (+ research group?) + NSF funding

111.
0.7.4.1 out last month

112.
18K LOCs of Python, 8K LOCs of C++

113.
LGPL </li></ul>

114.
PyML # Homepage: http://pyml.sourceforge.net/ # Dependencies: numpy, matplotlib .PHONY: install install: PyML-0.7.4.1.tar.gz tar -xzvf $< cd PyML-0.7.4.1 && python setup.py build && sudo python setup.py install PyML-0.7.4.1.tar.gz: wget https://sourceforge.net/projects/pyml/files/PyML-0.7.4.1.tar.gz/download

115.
hcluster <ul><li>Damian Eads, UCSC

116.
11K LOCs of Python, 8K of C and 500 of C++

117.
2500 lines of documentation

118.
1 developer, active 2007-2008

119.
New BSD lic

120.
8 routines for agglomerative clustering, routines that compute statistics on hierarchies, 21 distance functions </li></ul>

121.
hcluster # Homepage: http://code.google.com/p/scipy-cluster/ .PHONY: install install: sudo apt-get install python-hcluster .PHONY: source source: svn checkout http://scipy-cluster.googlecode.com/svn/trunk/ hcluster

122.
NLTK # Homepage: http://www.nltk.org/ # Dependencies: PyYAML .PHONY: install install: sudo apt-get install python-nltk .PHONY: source source: sudo apt-get source python-nltk

123.
mlpy # Homepage: https://mlpy.fbk.eu/ # Dependencies: NumPy, GSL .PHONY: install install: sudo echo &quot;deb http://ppa.launchpad.net/davide-albanese/ppa/ubuntu jaunty main&quot;| sudo tee -a /etc/apt/sources.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4AEC3064 sudo apt-get update sudo apt-get install python-mlpy .PHONY: source source: svn co https://mlpy.fbk.eu/svn/ mlpy

124.
LIBSVM # Homepage: http://www.csie.ntu.edu.tw/~cjlin/libsvm/ .PHONY: install install: sudo apt-get install libsvm2 python-libsvm .PHONY: source source: sudo apt-get source libsvm2

125.
PyEvolve # Homepage: http://pyevolve.sourceforge.net/ # Dependencies: matplotlib .PHONY: install install: wget http://downloads.sourceforge.net/pyevolve/Pyevolve-0.5-py2.6.egg sudo easy_install Pyevolve-0.5-py2.6.egg .PHONY: source source: svn co https://pyevolve.svn.sourceforge.net/svnroot/pyevolve

126.
FANN library # Homepage: http://www.nltk.org/ # Dependencies: PyYAML .PHONY: install install: sudo apt-get install libfann1-dev .PHONY: source source: sudo apt-get source libfann1 .PHONY: pythonBindings: pythonBindings: # need to build from source!

127.
Theano # Homepage: http://deeplearning.net/software/theano # Dependencies: blas .PHONY: install install: sudo apt-get install libblas-dev cd Theano && sudo python setup.py install .PHONY: source source: hg clone http://hg.assembla.com/theano Theano

128.
PyBrain # Homepage: http://www.pybrain.org/pages/download .PHONY: install install: cd pybrain && sudo python setup.py install .PHONY: source source: git clone git://github.com/pybrain/pybrain.git

129.
Shogun # Homepage: http://www.shogun-toolbox.org .PHONY: install install: sudo apt-get install shogun-doc-en shogun-cmdline shogun-python .PHONY: source source: svn checkout https://svn.tuebingen.mpg.de:/shogun/trunk shogun

130.
ffnet # Homepage: http://ffnet.sourceforge.net .PHONY: install install: cd ffnet && sudo python setup.py install .PHONY: source source: svn co https://ffnet.svn.sourceforge.net/svnroot/ffnet/trunk ffnet

131.
Observations <ul><li>There is a lot of activity in ML + Python

132.
Not everything is packaged for your favourite OS

133.
But: it is pretty easy to get going / help

134.
Documentation is of varying quality, heterogeneous

135.
But: there's source

136.
Good libraries are usually centred on research groups / backed up by universities </li></ul>

137.
Links <ul><li>http://mloss.org

138.
http://videolectures.net

139.
http://archive.ics.uci.edu/ml/ </li></ul>

140.
Books <ul><li>Machine Learning: An Algorithmic Perspective

141.
by Stephen Marsland, 2009

142.
Artificial Intelligence: A Modern Approach

143.
by Stuart Russell and Peter Norvig, 2009

144.
Programming Collective Intelligence

145.
by Toby Segaran, 2007

146.
Information Theory, Inference, and Learning Algorithms

147.
by David McKay, 2005 </li></ul>

148.
The End

149.
Are we there yet? <ul>Todd L. Veldhuisen, in “Software libraries and their reuse: entropy, Kolmogorov's complexity and Zipf's law”, 2005: <li>program size is a rough lower bound for effort

150.
using a component from a library we achieve reduction in size

151.
components maybe whatever form of abstraction we may invent: subroutines, generators of subroutines, generators of generators of sub ... </li></ul>

152.
Grunge end cont'd <ul><li>complexity of an object is measured by the length of the smallest program that generates it

153.
libraries allow us to “compress the incompressible” by taking the advantage of the commonality exhibited by programs within a problem domain

154.
problem domain entropy is defined as a measure of program diversity </li></ul>

155.
Grunge end cont'd <ul><li>“libraries are essentially incomplete, and there always be room for more components in any problem domain”

156.
“better tools and culture can have only marginal impact on reuse rates if the domain is inherently resistant to reuse” </li></ul>

157.
Machine learning is a high-entropy domain

158.
The future: doable and fascinating <ul><li>“There is a fantastic existence proof that learning is possible, which is the bag of water and electricity (together with a few trace chemicals) sitting between your ears.” – Stephen Marsland

159.
“You are a creative genius. Your creative genius is so accomplished that it appears, to you and to others, as effortless. Yet it far outstrips the most valiant efforts of today's fastest supercomputers. To invoke it, you need only open your eyes.” – Donald D. Hoffman </li></ul>

160.
<ul>Thank you! </ul>