RAMP approach to analytics: Rapid Analytics and Model Prototyping; collaborative data challenges with in-built data science process management tools and analytics; An observatory of data science and scientists. Presented at the Design Theory Special Interest Group of International Design Society. Mines ParisTech and Centre for Data Science.

1. A data science
observatory
Akin Kazakci, Mines ParisTech
Balazs Kégl, CNRS

2. Team
Balázs Kégl
CNRS
Alexandre Gramfort
Télécom ParisTech
Akın Kazakçı
Mines ParisTech
Camille Marini
Télécom ParisTech
Mehdi Cherti
UP Saclay
Yohann Sitruk
Mines ParisTech
Djalel Benbouzid
UPMC

3. 1
The research objective & questions

4. Enough with the chairs
• Design research is falling behind in dealing with contemporary
challenges
• Claim 1a: Too much in-breeding and repetition
• Claim 1b: Huge amount of work is based on ideas from 80s’
• Design is not about objects, but about reasoning

5. - Physics (Particle physics, Plasma
physics, astrophysics…)
- Biology (Genetics, Epidemiology…)
- Chemistry
- Economics, Finance, Banking
- Manufacturing, Industrial Internet
- Internet of things, Connected Devices
- Social media
- Transport & Mobility
- …
There is not enough
data scientists to
handle this much data
Revealing the potential of data: what role for design?

6. Last year: Crowdsourcing data challenges
1785 teams
Kazakci, A., Data science as a new frontier for design, ICED’15, Milan
Reasonable doubts about
the effectiveness of data
science contests

7. Crowdsourcing /?/ Design
crowd
/kraʊd/ noun
1.a large number of people gathered together in a
disorganized or unruly way
1. How to study the design process of a crowd?
2. How to manage the design process of a crowd?

8. SeekerSolvers
?
Crowdsourcing: C-K dynamics
Crowdsourced contests Crowdsourced collaboration

9. Analysis of design strategies
Achieve 5σ Dicovery condition: A discovery is claimed when we find a ‘region’
of the space where there is significant excess of ‘signal’ events.
(rejecting background-only hypothesis with a p value less than 2,9 x
10-7, corresponding to 5 Sigma).
Problem formulation: Traditional classification setting: « the task of
the participants is to train a classifier g based on the training data D
with the goal of maximizing the AMS (7) on a held-out (test) data
set » (HiggsML documentation)
With 2 tweaks:
- Training set events are « weighted »
- Maximize « Approximate Median Significance »:
Select a classification
method
Pre-processing
Choose hyper-params
Train
Optimize for
X
SVM Decision Trees NN…..…..
Performance metrics: During the overall learning process
performance metrics are used to supervise the quality and convergence
of a learned model. A traditional metric is accuracy:
where
Note that for HiggsML AMS, TP (s) and FP (b) are of particular
importance.
Boosting Bagging
others
Ensemble
Methods
(Extended) Dominant
Design
Traditional workflow = Dominant design
C space K Space
Fixating others…
Achieve 5σ
Select a classification
method
Pre-processing
Choose hyper-params
Train
Optimize for accuracy
SVM Decision Trees NN…..…..
Integrate AMS
directly in training
during Gradient
Boosting
(John)
Dicovery condition: A discovery is claimed
when we …
Problem formulation: Traditional
classification setting…
Cross-Validation: Techniques for evaluating
how a …
Ensemble Methods
during node
split in
random
forest
(John)
Weighted
Classification
Cascades
? ? ? ? ?
Optimization of AMS
Design for statistical efficiency
The biggest challenge is the unstability of
AMS. Competition results clearly show
that only participants who dealt effectively
with this issue have had higher ranks.
1st
2nd
3rd
Ensembles + CV monitoring
+ cutoff threshold seem to be
a winning strategy
monitoring
progress with
CV
+
ensembles
+
selecting a cutoff
threshold that optimise
(or stabilise AMS)
Public guide to AMS 3.6
« moves » many participants to the
given path
Fixation vs. Creative Authority
(Agogué et al, 2014)
Generating new design strategies
Data science as a new frontier for design
A. Kazakci, ICED’15 (submitted)
• Available data for HiggsML
- Forums ➔ 136 topics, 1400+ posts
- Documentation
- Participants’ blog entries
- GitHub codes
• Qualitative interpretation combined with
C-K modelling of participants’ strategies
Data challenges are hard to analyze

10. How do you put a crowd under a microscope?

11. 2
The research instrument

12. RAMP - Rapid Analytics and Model Prototyping
A Collaborative Development Platform for Data Science
Instant access to all submitted code - for
participants & organizers

13. RAMP allows us to collect data on the
data science model development
process
1
A Collaborative Development Platform for Data Science
2
3
We prepare a ‘starting kit’
Continuous access to code:
Organizers can follow real-time
what’s happening - and react
Participants can analyse and
build on every submission
Submissions are trained and
performances are displayed
4 Users actions and
interactions are recorded
5 Main Output: Dozens of
predictive models and
performance benchmark
RAMP - Rapid Analytics and Model Prototyping

14. Collecting data with RAMP
- Number of submissions
- Frequency of submissions
- Timing of submissions
- User interactions
- Performance of submissions
- Submitted code
- …
We are interested in
- the variety (code space +
prediction space)
- the mutual influences and
inheritance (code space)
- score and delta score
(impact)
- …

15. 3
Some applications, preliminary
observations & findings

16. Climatology
Time Series Based Event
Prediction on Geo-tagged data
Two workshops: Improvement
in RMSE score: from 0.90 to
0.43
El Nino Prediction
- Temperature data
- Geo-tagged time series
- Prediction: 6 months ahead
George Washington University
George Mason University

17. Astrophysics
Classification of variable stars
One day workshop: Accuracy
improvement: %89 to %96
Light curves (luminosity vs
time profiles)
- Static features
- Functional data
Marc Monier (LAL),
Gilles Faÿ (Centrale-Supelec)

18. Ecology
Finding & Classifying Insects
One day event:
Improvement in prediction
accuracy: from 0.31 to 0.70
from Image Data
Pollenating Insects
- Image data (20K images)
- 18 types of insects
- Deep neural net models
Paris Museum of Natural History,
SPIPOLL.org, NVDIA,
Université de Champagne-
Ardenne ROMEO HPC Center

19. Cancer Research

20. A graph of model similarities
- Steady progression. They have built systematically on a submission they previously created, without being influenced by
the others. Their performance may go either up (constantly) or down (constantly).
- Breakdowns or jumps. There are other groups, where the performance increased or decreased strongly from one
modification submission to the next). There may be some robustness/vulnerability issue with their approach - to be further
investigated.
- Successful expansions. An important “break” has happened at 12:00. This corresponds to “cropping” idea. Strangely,
two very similar submissions (small distance) have been submitted at the same time - one of them did not improve the
score at all (around 0.35, while the leader was around 0.55), whereas the other improved considerably (0.65).
- Currently, we see no dependency between this break and the winning solution. This might be related to the way we
have measured the code similarity.
Some observations

21. “Note that, following the RAMP approach, this model is
the result of a succession of small improvements
made on top of other participants’ contributions. We
did not reach a prediction score of 0.71 in one shot, but
after applying several tricks and manually tuning some
parameters.”
Heuritech,
Winner of Insect Challenge
Blog entry

22. How to compare design concepts - represented as code?
~
?

23. Various distance measures

24. Comparing performance profiles: Promoting novelty search
- Greyness: Model’s raw score
- Size: model’s contribution
- Position: similarity/dissimilarity in predictions
2D projection (MDS) of model’s prediction profiles

25. • Monitoring & Modelling “contribution”
(Pierre Fleckinger, Economic Agents & Incentive Theory)
• Pushing towards “Novelty Search”
(Jean-Bastiste Mouret, Novelty Search)
• Controlled experiments
We Found (to be validated by further studies):
In progress:
• Gravitation: following a given submission, others are hovering around the same
coordinates, by incremental adjustments
• Repulsion: new submission using out-of-the-box code to explore the white space (no
previous close-by submissions exist)
• Hybridation: opportunistic integration of previous submissions, involving/inspired by
at least two different source of code.
RAMP platform
• RAMP platform is meant to be a free tool for researchers and students; this
opens up new perspectives (pedagogy & research) and hopefully brings
closer different communities

26. Akin Kazakci, Mines ParisTech
akin.kazakci@mines-paristech.fr
Thank you