This talk explores the possibility of turning machine learning research into open science and proposed concrete approaches to achieve this goal

1. the open experiment database
meta-learning for the masses
Joaquin Vanschoren @joavanschoren

2. The Polymath story
Tim Gowers

3. Machine Learning
are we doing it right?

4. Computer Science
• The scientific method
• Make a hypothesis about the world
• Generate predictions based on this hypothesis
• Design experiments to verify/falsify the prediction
• Predictions verified: hypothesis might be true
• Predictions falsified: hypothesis is wrong

5. Computer Science
• The scientific method (for ML)
• Make a hypothesis about (the structure of) given data
• Generate models based on this hypothesis
• Design experiments to measure accuracy of the models
• Good performance: It works (on this data)
• Bad performance: It doesn’t work on this data
• Aggregates (it works 60% of the time) not useful

6. Computer Science
• The scientific method (for ML)
• Make a hypothesis about (the structure of) given data
• Generate models based on this hypothesis
• Design experiments to measure accuracy of the models
• Good performance: It works (on this data)
n zed o
acteri doesn’t work on this data
char
• Badtperformance: Its well?
nd a a be work
H ow •a Aggregatesm works 60% of the time) not useful
c
orith (it
hich t he alg
w

7. Computer Science
• The scientific method (for ML)
• Make a hypothesis about (the structure of) given data
• Generate models based on this hypothesis
• Design experiments to measure accuracy of the models
• Good performance: It works (on this data)
n
teri zed o ct of
harac It doesn’t work on thissdata effe
• Badtperformance: s well?
a be c
n da work hat i the tings?
H ca
ow • Aggregatesm works 60% of the time) not eter set
rith (it W
th e algo aram useful
w hich p

8. Meta-Learning
• The science of understanding which algorithms work
well on which types of data
• Hard: thorough understanding of data and algorithms
• Requires good data: extensive experimentation
• Why is this separate from other ML research?
• A thorough algorithm evaluation = a meta-learning study
• Original authors know algorithms and data best, have large sets
of experiments, are (presumably) interested in knowing on
which data their algorithms work well (or not)

9. Meta-Learning
With the right tools, can we make everyone a
meta-learner?
datasets algorithm comparison
data insight
learning curves
Large sets of experiments
algorithm selection
ML algorithm
meta-learning
design
algorithm characterization
algorithm insight
data characterization
source code
bias-variance analysis

10. Open Machine Learning

11. Open science
World-wide Telescope

12. Open science
Microarray Databases

13. Open science
GenBank

14. Open machine learning?
• We can also be `open’
• Simple, common formats to describe experiments, workflows,
algorithms,...
• Platform to share, store, query, interact
• We can go (much) further
• Share experiments automatically (open source ML tools)
• Experiment on-the-fly (cheap, no expensive instruments)
• Controlled experimentation (experimentation engine)

15. Formalizing
machine learning
• Unique names for algorithms, datasets, evaluation
measures, data characterizations,... (ontology)
• Based on DMOP, OntoDM, KDOntology, EXPO,...
• Simple, structured way to describe algorithm setups,
workflows and experiment runs
• Detailed enough to reproduce all experiments

16. Run
run

17. Run
Execution of a
predefined setup
run

18. Run
Execution of a
predefined setup
run
setup

19. Run
run
setup

20. Run
in
data run
setup

21. Run
machine
in
data run
setup

22. Run
machine
in out
data run data
setup

23. Run
machine
in out
data run data
Also:
start time
setup author
status,...

24. Setup
setup

25. Setup
Plan of what
we want to do
setup

26. Setup
Plan of what
we want to do
setup
f(x)
algorithm function workflow experiment
setup setup

27. Setup
Hierarchical
part of
setup
f(x)
algorithm function workflow experiment
setup setup

28. Setup
Hierarchical
Parameterized
part of
p=!
setup parameter setting
f(x)
algorithm function workflow experiment
setup setup

29. Setup
Hierarchical
Parameterized
Abstract/concrete
part of
p=!
setup parameter setting
f(x)
algorithm function workflow experiment
setup setup

30. Algorithm Setup
algorithm
setup

31. Algorithm Setup
Fully defined algorithm
part of configuration
algorithm
setup

32. Algorithm Setup
Fully defined algorithm
part of configuration
algorithm
setup
p=! f(x)
implementation parameter setting function
setup

33. Algorithm Setup
Fully defined algorithm
part of configuration
algorithm
setup
p=! f(x)
implementation parameter setting function
setup

34. Algorithm Setup
part of
algorithm
setup
p=! f(x)
implementation parameter setting function
setup

35. Algorithm Setup
part of
algorithm
setup
p=! f(x)
implementation parameter setting function
setup
p=? f(x)
algorithm quality algorithm parameter mathematical function

36. Algorithm Setup
part of
unique
names
algorithm
setup
p=! f(x)
implementation parameter setting function
setup
p=? f(x)
algorithm quality algorithm parameter mathematical function

37. Algorithm Setup
part of
unique Roles:
names learner,
algorithm base-learner,
setup kernel,...
p=! f(x)
implementation parameter setting function
setup
p=? f(x)
algorithm quality algorithm parameter mathematical function

38. Setup
part of
setup
f(x)
algorithm function workflow experiment
setup setup

39. Workflow Setup
part of
setup
algorithm workflow
setup

40. Workflow Setup
part of
ta
so
rge
ur
setup
t
ce
algorithm workflow connection
setup
Workflow: components, connections,
and parameters (inputs)

41. Workflow Setup
part of
Also:
ta
ports
so
rge
ur
setup
t
datatype
ce
algorithm workflow connection
setup
Workflow: components, connections,
and parameters (inputs)

42. Workflow
Example
Weka. Weka. Weka.SMO
url Weka.RBF eval evalu-
ARFFLoader Evaluation
data ations
par p=! location= p=! F=10 p=! C=0.01 p=! G=0.01
http://... data
logRuns=true p=! S=1 f(x) 5:kernel
pred predic-
logRuns=false
tions
2:loadData logRuns=true 4:learner
3:crossValidate
1:mainFlow

43. Workflow
Example
Weka. Weka. Weka.SMO
url Weka.RBF eval evalu-
ARFFLoader Evaluation
data ations
par p=! location= p=! F=10 p=! C=0.01 p=! G=0.01
http://... data
logRuns=true p=! S=1 f(x) 5:kernel
pred predic-
logRuns=false
tions
2:loadData logRuns=true 4:learner
3:crossValidate
1:mainFlow
evaluations 6
eval Evaluations
data 8 data pred
Weka.Instances predictions 7
Predictions

44. Setup
part of
setup
f(x)
algorithm function workflow experiment
setup setup

45. Experiment
Setup
part of
setup
<X>
algorithm workflow experiment experiment
setup variable

46. Experiment
Setup
part of
se
tu
p
setup
<X>
algorithm workflow experiment experiment
setup variable
Also: experiment design, description,
literature reference, author,...

47. Experiment Setup

48. Experiment Setup
Variables: labeled tuples which can be
referenced in setups

49. Run
machine
in out
data run data
Also:
start time
setup author
status,...

50. Run
data
dataset evaluation model predictions

51. Run
source
data run
dataset evaluation model predictions

52. Run
source
data quality data run
dataset evaluation model predictions

53. EXPML
Weka. Weka. Weka.SMO
url Weka.RBF eval evalu-
ARFFLoader Evaluation
data ations
par p=! location= p=! F=10 p=! C=0.01 p=! G=0.01
http://... data
logRuns=true p=! S=1 f(x) 5:kernel
pred predic-
logRuns=false
tions
2:loadData logRuns=true 4:learner
3:crossValidate
1:mainFlow

54. Demo
(preview)

55. Examples
1$
0.9$
0.8$
predic've)accuracy)
0.7$
0.6$
RandomForest$
0.5$ C45$
Logis<cRegression$
0.4$
RacedIncrementalLogitBoostAStump$
0.3$ NaiveBayes$
SVMARBF$
0.2$
10$ 20$ 30$ 40$ 50$ 60$ 70$ 80$ 90$ 100$
percentage)of)original)dataset)size)
Learning curves

56. Examples
When does one algorithm outperform another?

57. Examples
When does one algorithm outperform another?

58. Examples
Bias-variance profile + effect of dataset size

59. Examples
boosting
bagging
Bias-variance profile + effect of dataset size

60. Examples
Bias-variance profile + effect of dataset size

61. Taking it further
Seamless integration
• Webservice for sharing, querying experiments
• Integrate experiment sharing in ML tools (WEKA,
KNIME, RapidMiner, R, ....)
• Mapping implementations, evaluation measures,...
• Online platform for custom querying, community
interaction
• Semantic wiki: algorithm/data descriptions, rankings, ...

62. Experimentation Engine
• Controlled experimentation (Delve, MLComp)
• Download datasets, build training/test sets
• Feed training and test sets to algorithms, retrieve predictions/
models
• Run broad set of evaluation measures
• Benchmarking (Cross-Validation), learning curve analysis,
bias-variance analysis, workflows(!)
• Compute data properties for new datasets

63. Why would you use it?
(seeding)
• Let the system run the experiments for you
• Immediate, highly detailed benchmarks (no repeats)
• Up to date, detailed results (vs. static, aggregated in journals)
• All your results organized online (private?), anytime, anywhere
• Interact with people (weird results?)
• Get credit for all your results (e.g. citations), unexpected results
• Visibility, new collaborations
• Check if your algorithm really the best (e.g. active testing)
• On which datasets does it perform well/badly?

64. Question
Is
open
machine learning
possible?

65. Merci
Danke Thanks
Xie Xie
Diolch
Toda
Dank U
Grazie
Spasiba
Efharisto
Gracias
Arigato
Köszönöm
Tesekkurler
Kia ora
Dhanyavaad
Hvala
http://expdb.cs.kuleuven.be