4. Computer Science
• The scientific method
• Make a hypothesis about the world
• Generate predictions based on this hypothesis
• Design experiments to verify/falsify the prediction
• Predictions verified: hypothesis might be true
• Predictions falsified: hypothesis is wrong
5. Computer Science
• The scientific method (for ML)
• Make a hypothesis about (the structure of) given data
• Generate models based on this hypothesis
• Design experiments to measure accuracy of the models
• Good performance: It works (on this data)
• Bad performance: It doesn’t work on this data
• Aggregates (it works 60% of the time) not useful
6. Computer Science
• The scientific method (for ML)
• Make a hypothesis about (the structure of) given data
• Generate models based on this hypothesis
• Design experiments to measure accuracy of the models
• Good performance: It works (on this data)
n zed o
acteri doesn’t work on this data
char
• Badtperformance: Its well?
nd a a be work
H ow •a Aggregatesm works 60% of the time) not useful
c
orith (it
hich t he alg
w
7. Computer Science
• The scientific method (for ML)
• Make a hypothesis about (the structure of) given data
• Generate models based on this hypothesis
• Design experiments to measure accuracy of the models
• Good performance: It works (on this data)
n
teri zed o ct of
harac It doesn’t work on thissdata effe
• Badtperformance: s well?
a be c
n da work hat i the tings?
H ca
ow • Aggregatesm works 60% of the time) not eter set
rith (it W
th e algo aram useful
w hich p
8. Meta-Learning
• The science of understanding which algorithms work
well on which types of data
• Hard: thorough understanding of data and algorithms
• Requires good data: extensive experimentation
• Why is this separate from other ML research?
• A thorough algorithm evaluation = a meta-learning study
• Original authors know algorithms and data best, have large sets
of experiments, are (presumably) interested in knowing on
which data their algorithms work well (or not)
9. Meta-Learning
With the right tools, can we make everyone a
meta-learner?
datasets algorithm comparison
data insight
learning curves
Large sets of experiments
algorithm selection
ML algorithm
meta-learning
design
algorithm characterization
algorithm insight
data characterization
source code
bias-variance analysis
14. Open machine learning?
• We can also be `open’
• Simple, common formats to describe experiments, workflows,
algorithms,...
• Platform to share, store, query, interact
• We can go (much) further
• Share experiments automatically (open source ML tools)
• Experiment on-the-fly (cheap, no expensive instruments)
• Controlled experimentation (experimentation engine)
15. Formalizing
machine learning
• Unique names for algorithms, datasets, evaluation
measures, data characterizations,... (ontology)
• Based on DMOP, OntoDM, KDOntology, EXPO,...
• Simple, structured way to describe algorithm setups,
workflows and experiment runs
• Detailed enough to reproduce all experiments
40. Workflow Setup
part of
ta
so
rge
ur
setup
t
ce
algorithm workflow connection
setup
Workflow: components, connections,
and parameters (inputs)
41. Workflow Setup
part of
Also:
ta
ports
so
rge
ur
setup
t
datatype
ce
algorithm workflow connection
setup
Workflow: components, connections,
and parameters (inputs)
42. Workflow
Example
Weka. Weka. Weka.SMO
url Weka.RBF eval evalu-
ARFFLoader Evaluation
data ations
par p=! location= p=! F=10 p=! C=0.01 p=! G=0.01
http://... data
logRuns=true p=! S=1 f(x) 5:kernel
pred predic-
logRuns=false
tions
2:loadData logRuns=true 4:learner
3:crossValidate
1:mainFlow
43. Workflow
Example
Weka. Weka. Weka.SMO
url Weka.RBF eval evalu-
ARFFLoader Evaluation
data ations
par p=! location= p=! F=10 p=! C=0.01 p=! G=0.01
http://... data
logRuns=true p=! S=1 f(x) 5:kernel
pred predic-
logRuns=false
tions
2:loadData logRuns=true 4:learner
3:crossValidate
1:mainFlow
evaluations 6
eval Evaluations
data 8 data pred
Weka.Instances predictions 7
Predictions
44. Setup
part of
setup
f(x)
algorithm function workflow experiment
setup setup
45. Experiment
Setup
part of
setup
<X>
algorithm workflow experiment experiment
setup variable
46. Experiment
Setup
part of
se
tu
p
setup
<X>
algorithm workflow experiment experiment
setup variable
Also: experiment design, description,
literature reference, author,...
61. Taking it further
Seamless integration
• Webservice for sharing, querying experiments
• Integrate experiment sharing in ML tools (WEKA,
KNIME, RapidMiner, R, ....)
• Mapping implementations, evaluation measures,...
• Online platform for custom querying, community
interaction
• Semantic wiki: algorithm/data descriptions, rankings, ...
62. Experimentation Engine
• Controlled experimentation (Delve, MLComp)
• Download datasets, build training/test sets
• Feed training and test sets to algorithms, retrieve predictions/
models
• Run broad set of evaluation measures
• Benchmarking (Cross-Validation), learning curve analysis,
bias-variance analysis, workflows(!)
• Compute data properties for new datasets
63. Why would you use it?
(seeding)
• Let the system run the experiments for you
• Immediate, highly detailed benchmarks (no repeats)
• Up to date, detailed results (vs. static, aggregated in journals)
• All your results organized online (private?), anytime, anywhere
• Interact with people (weird results?)
• Get credit for all your results (e.g. citations), unexpected results
• Visibility, new collaborations
• Check if your algorithm really the best (e.g. active testing)
• On which datasets does it perform well/badly?
65. Merci
Danke Thanks
Xie Xie
Diolch
Toda
Dank U
Grazie
Spasiba
Efharisto
Gracias
Arigato
Köszönöm
Tesekkurler
Kia ora
Dhanyavaad
Hvala
http://expdb.cs.kuleuven.be