1. MPOWERS a Multi Point of vieW Evaluation
Refinement Studio for Spoken Dialogue Systems
Ambitions
‐ Alignment of evalua�on approaches (not standardiza�on)
‐ Enforcement of norms to describe the evalua�on protocols, common thinking models and terminology
for evaluators to make their procedures explicit
Er
Ho go
s n
Pro ting l Ex om
vid ica r pe ics Marketing
er chn ppe
Te lo
rt
Experts
ve
Custom- De
made
dashboards
commonly
d efined KPIs
(IT U-T Rec.
Context
sup.24) ‐ No one size fits‐all‐evalua�on to cover all needs
‐ Nomadism of evalua�on approaches
‐ Need for coopera�on between stakeholders
A unique Features
corpus
of data ‐ Implements design conven�ons in a user‐friendly interface
for all ‐ Ensures all evaluator‐users manipulate a unique shared corpus of data
‐ A shared set of parameters for evalua�ons design and retrieval
‐ Supports the convergence of the evalua�on prac�ces
‐ Considers several analy�cal points of view
Let's Go! ‐ Supports a both data‐driven and goal‐driven process
system corpus ‐ Business‐intelligence solu�on based
(CMU)
Evaluation "Process through which one defines, obtains
.
and delivers useful pieces of informa�on to se�le between
the alterna�ve possible decisions." Daniel Stufflebeam, 1980.
Visual dashboards display pre-processed data according to
Screenshots pre-defined evaluation profiles. Ex: Dashboard dedicated to a
high-level view on usability performances
A summary of high-level KPIs provides a general view on the evaluated
system with ”red-light indicators”. Links to more detailed charts or analysis
tools are displayed next to each of them.
Marianne.laurent@orange-ftgroup.com, Philippe.Bretier@orange-ftgroup.com We thank Telecom Bretagne, Q. Jin, X. Chen, S.Zarrad, F. Agez
Orange Labs, 2 avenue Pierre Marzin, 22307 Lannion, France and A. Bolze for their contribution to the platform deployment.