Tutorial given at the European Conference for Machine Learning (ECMLPKDD 2015). It covers OpenML, how to use it in your research, interfaces in Java, R, Python, use through machine learning tools such as WEKA and MOA. Also covers topics in open science and reproducible research.

1. OpenML
N E T W O R K E D M A C H I N E L E A R N I N G

2. Metalearning and algorithm selection require:
• Large amounts of real datasets
• Wide range of state-of-the-art algorithms
• Huge amounts of experimentation: explore how
algorithms behave on many kinds of data
• Integrated tools, databases, services to organise
all this

3. Gregg Gordon
Data by itself has no value. It’s the ever-changing
ecosystem surrounding data that gives it meaning.
So much data, experiments are going around.
Can we connect online to
share data, code, results?

4. N E T W O R K E D S C I E N C E
Polymaths: Mathematicians solved
centuries-old problems within weeks
by collaborating online, via blog
SDSS: Thousands of papers published
on data server from single telescope.
Also: GenBank, ChEMBL, GEO, …
Galaxy Zoo: Amateur astronomers
make new discoveries by looking
through thousands of images

5. F R I C T I O N - L E S S , N E T W O R K E D E N V I R O N M E N T
F O R M A C H I N E L E A R N I N G R E S E A R C H
Organized data: Experiments connected to data, code, people. Data and
code can live on existing repositories, services (e.g. GitHub)
Easy to use: Integrated in existing ML environments for easy reuse, and
automated, reproducible sharing
Easy to contribute: Post single dataset, algorithm, experiment, comment
Reward structure*: Build reputation and trust (auto-tracking of reuse, likes)

6. Data (ARFF) uploaded or referenced, versioned
analysed, characterized, organised online

7. Tasks contain data, goals, procedures.
Readable by tools, automates experimentation
Results organized online: realtime challenge
Train-test
splits
Evaluation
measure

8. Flows (code) from various tools, versioned.
Integrations + APIs (REST, Java, R, Python,…)

9. Experiments auto-uploaded, evaluated online,
reproducible, linked to data, flows and authors

10. Experiments auto-uploaded, evaluated online,
reproducible, linked to data, flows and authors

11. Explore
Reuse
Share

13. • Search by
keywords or
properties
• Filters
• Tagging
Data

14. • Wiki-like
descriptions
• Analysis and
visualisation of
features
Data

15. • Wiki-like
descriptions
• Analysis and
visualisation of
features
• Auto-calculation
of large range
of meta-features
Data

16. • Example: Classification
on click prediction
dataset, using 10-fold
CV and AUC
• People submit results
(e.g. predictions)
• Server-side evaluation
(many measures)
• All results organized
online, per algorithm,
parameter setting
• Online visualizations:
every dot is a run
plotted by score
Tasks

17. • Leaderboards visualize progress over time: who delivered breakthroughs
when, who built on top of previous solutions
• Collaborative: all code and data available, learn from others, form teams
• Real-time: who submits first gets credit, others can improve immediately

18. Classroom challenges

19. Classroom challenges

20. • All results obtained with same flow organised online
• Results linked to data sets, parameter settings -> trends/comparisons
• Visualisations (dots are models, ranked by score, colored by parameters)

21. • Detailed run info
• Author, data, flow,
parameter settings,
result files, …
• Evaluation details
(e.g., results per
sample)

22. REST API Requires OpenML account.

23. REST API
Guide on: www.openml.org/guide
List datasets: /data/list
List tasks: /task/list
List flows: /flow/list
List runs: /run/list
Get dataset: /data/<id>
Get task: /task/<id>
Get flow: /flow/<id>
Get run: /run/<id>
…
Endpoint: api.openml.org/v1/

24. R API
Tutorial:
https://github.com/openml/r/blob/master/doc/knitted/1-Introduction.md
List datasets

25. R API
List tasks

26. R API
List flows

27. R API
List runs and results (under construction)

28. Java API
Guide: https://www.openml.org/guide
Python API
Guide: https://www.openml.org/guide

29. Share
Explore
Reuse

30. Website
Search,
CVS export,
query

31. R API
Download data

32. R API
Download task

33. R API
Download task, data

34. R API
Download flow

35. R API
Download run
Download run with predictions

36. Python API
Simple example:

37. Reuse
Explore
Share

38. WEKA
OpenML extension
in plugin manager

39. MOA

40. RapidMiner: 3 new operators
See talk by Jan van Rijn

41. R API
Run a task
(with mlr):

42. R API
Run a task
(with mlr):
And upload:

43. R API
Simple example:

44. Studies (e-papers)
- Online counterpart of a paper, linkable
- Add data, code, experiments (new or old)
- Public or shared within circle
Circles
Create collaborations with trusted researchers
Share results within team prior to publication
Altmetrics
- Measure real impact of your work
- Reuse, downloads, likes of data, code, projects,…
- Online reputation (more sharing)
Online collaboration (soon)

45. Collaboratory: bring data
scientists and domain scientists
together online (and their data
and tools)
Easy, large-scale collab:
Extract actionable datasets, key
tools. Scientists shares data and
get help, DS can test technique
on many current datasets.
Real-time collab: share
experiments automatically,
discuss online.Automate
experimentation.
Data
scientist
Domain
scientist
Tool
developer
data+
code
Bridging gaps in data-driven science

46. Collaboratory: bring data
scientists and domain scientists
together online (and their data
and tools)
Easy, large-scale collab:
Extract actionable datasets, key
tools. Scientists shares data and
get help, DS can test technique
on many current datasets.
Real-time collab: share
experiments automatically,
discuss online.Automate
experimentation.
Data
scientist
Domain
scientist
Tool
developer
data+
code
Bridging gaps in data-driven science

47. OpenML Community
Jan-Jun 2015
Used all over the world
400 7-day, 1700 30-day active users, growing
1000s of datasets, flows, 450000+ runs

48. Algorithm selection, configuration
OpenML offers major opportunities for automated
algorithms selection and configuration
• Many datasets, flows, experiments: allows much larger
meta-learning studies than possible before
• APIs:Algorithm selection and configuration services can
be built on top of OpenML, reusing and sharing data
• These services can be integrated, accessible through
OpenML, continuously improving the more they are used.

49. Automating machine learning
Human scientists
meta-data, models,
evaluations
Automated processes
Data cleaning
Algorithm Selection
Parameter Optimization
Workflow construction
Post processing
API

50. A U T O M AT E D M A C H I N E L E A R N I N G
D
ata
Collection
D
ata
Cleaning
M
etric
Selection
Algorithm
Selection
Param
eter O
ptim
ization
Post-processing
O
nline
evaluation
D
ata
Coding
S E M I -
Thanks to R. Caruana

51. A U T O M AT E D M A C H I N E L E A R N I N G
D
ata
Collection
D
ata
Cleaning
M
etric
Selection
Algorithm
Selection
Param
eter O
ptim
ization
Post-processing
O
nline
evaluation
D
ata
Coding
S E M I -
Thanks to R. Caruana

52. A U T O M AT E D M A C H I N E L E A R N I N G
D
ata
Collection
D
ata
Cleaning
M
etric
Selection
Algorithm
Selection
Param
eter O
ptim
ization
Post-processing
O
nline
evaluation
D
ata
Coding
S E M I -
Thanks to R. Caruana

53. Bruce Mau
When everything is connected to everything else,
for better or for worse, everything matters.

54. - Open Source, on GitHub
- Regular workshops, hackathons
Join OpenML
Next workshop:
- Lorentz Center (Leiden),
14-18 March 2016

55. T H A N K Y O U
Joaquin Vanschoren
Jan van Rijn
Bernd Bischl
Matthias Feurer
Michel Lang
Nenad Tomašev
Giuseppe Casalicchio
Luis Torgo
You?
#OpenMLFarzan Majdani
Jakob Bossek