MEX Vocabulary - A Lightweight Interchange Format for Machine Learning Experiments

MEX Vocabulary
A Lightweight Interchange Format for Machine Learning Experiments
Diego Esteves et al.
Department of Computer Science, AKSW
University of Leipzig
17 Sep 2015 - SEMANTiCS
Diego Esteves et al. (University of Leipzig) MEX Vocabulary 17 Sep 2015 - SEMANTiCS 1 / 30

Outline
1 Introduction
Problem
Motivation
Challenges
State of the Art
2 MEX
The Inspiration
The Architecture
Examples
3 Conclusion and Future Work

Motivation
The Problem
The Problem
How should we represent results of machine learning experiments in a
common, comprehensive and interoperable format?

Motivation
Example 1: Collaborative Project
Three Universities are working collaboratively in a research project
How to achieve a high level of interoperability?
A uses the Weka1 toolkit.
B uses DL-Learner2
C uses the Accord Framework3
1
http://www.cs.waikato.ac.nz/ml/weka/
2
http://dl-learner.org/
3
http://accord-framework.net/

Motivation
Example 2: hands on...
A complex script-based scenario
You are working on your research about stock market predictions and want
to store the data for further analysis?
eg.: a script which takes 2 days to run a multi-level machine learning
algorithm.

Motivation
Example 3: Reading or Reviewing a paper
You are a reviewer or scientist...
sometimes it’s hard to understand the proposed solution of a research
paper.
.
The ACL POS Tagging website (State of the art)
exempliﬁes a good use case for MEX on the web 1.
Furthermore, in both cases the task/reading is error-prone and
time-consuming.
1
http://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_
the_art)

Motivation
Solution
Machine-readable data

Motivation
Solution
Existing Standards:
Comma-Separated Values (CSV)
eXtensible Markup Language (XML)
JavaScript Object Notation (JSON)
Value-Object (VO)
Data-Transfer-Objects (DTO)
Database Management System (DBMS)

Motivation
Solution

Motivation
3 drawbacks
1 The lack of schema deﬁnition: you always have to deﬁne the
schema by yourself and share your model afterwards.
2 DBMS is technology-dependent and does not provides reasoning
and inference capabilities.
3 the lack of semantic information.

Motivation
Problem: an example

Motivation
The Problem
The Optimal Scenario
How should we represent results of machine learning experiments in a
common1, comprehensive (but not complex)2, lightweight3,
interoperable4 and flexible5 format, taking into consideration a low
effort-level6 for implementation?

State of the Art
Related Work

State of the Art
Platforms for e-science workﬂows
Name Description
MyExperiment
[DeRoure2009 ]
A collaborative environment
where scientists can
publish their workflows and
experiment plans
Wings
[Gil2011 ]
A Semantic Approach to
creating very large
scientific workflows
OpenTOX
[Tcheremenskaia2012 ]
An interoperable predictive
toxicology framework
OpenML
[Vanschoren2014 ]
A frictionless,
collaborative environment
for exploring machine
learning

State of the Art
Ontologies
Name Description
Expos´e
[Vanschoren2010 ]
Data mining experiments
used in conjunction with
Experiment Databases
OntoDM
[Panov2013 ]
Data mining investigations
DMOP
[Keet2015 ]
Data Mining OPtimization
Ontology: It supports
informed decision-making
at various choice points of
the data mining process

MEX.aksw.org

The abstraction
What we want to describe
Machine Learning Deﬁnition by T.Mitchell
“A computer program is said to learn from experience E with respect to
some task T and some performance measure P, if its performance on
T, as measured by P, improves with experience E” – Tom Mitchell
ML Concepts MEX Classes
experience E mexcore:ExecutionCollection
task T mexalgo:Algorithm
performance measure P mexperf:ExecutionPerformance

MEX
3 Vocabularies
MEX Core
formalizes the key entities for representing the basic
steps on machine learning executions
MEX Algorithm
representing the context of machine learning algorithms and
their associated characteristics
MEX Performance
provides the basic entities for representing the
experimental results of executions of machine learning
algorithms

MEX Vocabulary (:mexalgo + :mexcore + :mexperf)
and Related Ontologies
402
778
858
757
MEX (7+14+10=31)
ONTO-DM
Expos´e
DMOP

MEX
Interlinking the 3 layers: mexalgo, mexcore and mexperf

:mexalgo

:mexcore

:mexperf

MEX
ACL POS Tagging website metadata

Next chapter ;-)
RDF? Ontology? Jena?
Dublin Core...? SPARQL?
OWL? PROV-O, What?

p u b l i c s t a t i c void main ( S t r i n g [ ] args ) {
MyMEX 10 mex = new MyMEX 10 ( ) ;
mex . setAuthorName (”D Esteves ” ) ;
S t r i n g e i d = ”E001S001 ”;
mex . addConf ( e i d ) . s e t D e s c r i p t i o n (” h e l l o world experiment ” ) ;
mex . Conf ( e i d ) . addFeature (” min ; max ; op ; c l o s e ” ) ;
mex . Conf ( e i d ) . Implementation ( ) . s e t ( enumImplementation . Weka ) ;
mex . Conf ( e i d ) . addAlgorithm ( enumAlgorithm . SupportVectorMachines ) ;
mex . Conf ( e i d ) . addAlgorithm ( enumAlgorithm . NaiveBayes ) ;
mex . Conf ( e i d ) . Algorithm ( enumAlgorithm . SupportVectorMachines ) . addParameter (”C” , ”10ˆ3”);
mex . Conf ( e i d ) . Algorithm ( enumAlgorithm . SupportVectorMachines ) . addParameter (” alpha ” , ” 0 . 2 ” ) ;
. . .
}
/∗ your code here ∗/
. . .
S t r i n g e x i d = mex . Conf ( e i d ) . a d d E x e c u t i o n O v e r a l l . addPerformance ( enumMeasures .ACCURACY, . 9 6 ) ;
S t r i n g e x i d = mex . Conf ( e i d ) . E x e c u t i o n O v e r a l l ( e x i d ) . addPerformance ( enumMeasures .TPR, . 7 8 ) ;
. . .
MEXSerializer 10 . g e t I n s t a n c e ( ) . parse (mex ) ;

Conclusion
D.Esteves et al.
Requirement Argumentation
lightweight 7: this is the minimal number of
classes you need for representing a
basic execution. 31: this is the
number of the most important entities
in the 3 layers
flexible Single or Overall Executions
Choose your inputs/outputs
low
effort-level
MEX provides APIs which encapsulate
the semantic knowledge. So you can
avoid extra implementation-effort and
just log your inputs and outputs

Conclusion
D.Esteves et al.
Requirement Argumentation
common The concepts behind vocabularies
allow us to achieve a high level
of abstraction, generalization and
formalization of concepts
interoperable Vocabularies are the current best
choice for representing real-world
entities
comprehensive classification, regression and
clustering problems are covered

Conclusion
D.Esteves et al.
1 Produces Provenance Metadata.
2 Allows Querying Results.
3 Defines an Interoperable Format for Sharing Machine Learning
Experiments.
4 Benefits Meta-Learning [Vilalta2002 ] Approaches.
5 Tends to minimize the misinterpretation probability rate
on persuasive and informative aspects [Gillen2006 ].
6 MEX is flexible and lightweight.
7 Experiment Databases [Blockeel2007 ][Vanschoren2012 ] need
an interchange format for experiments.
8 MEX provides APIs which facilitate the file generation process.
9 Benchmark Systems[Usbeck2014 ] can benefit from a standard
format.
10 Generate your LaTeX table automatically.

MEX
D.Esteves et al.
Thank you so much for your attention!
mex.aksw.org

MEX Vocabulary - A Lightweight Interchange Format for Machine Learning Experiments

Recommended

Recommended

More Related Content

Similar to MEX Vocabulary - A Lightweight Interchange Format for Machine Learning Experiments

Similar to MEX Vocabulary - A Lightweight Interchange Format for Machine Learning Experiments (20)

Recently uploaded

Recently uploaded (20)

MEX Vocabulary - A Lightweight Interchange Format for Machine Learning Experiments