This document discusses challenges and considerations for open data sharing and collaborative analysis. It addresses issues such as [1] balancing top-down and bottom-up hypothesis generation, [2] distinguishing relevant patterns from false correlations, and [3] expanding the space of hypotheses evaluated beyond standard datasets and tools. The document also questions [4] when to abandon hypotheses as non-informative or indeterminate, and [5] how to establish rules for sharing data and crediting contributions in an open collaboration environment.

1. Summary
Data
coding
,
analysis,
archiving,
and
sharing
for
open
collabora9on
Richard
Aslin
University
of
Rochester

2. 1.
What
is
your
hypothesis?
• 9/11
occurred
because
the
intelligence
community
suffered
from
a
“failure
of
imagina9on”
– BoGom-­‐up
data
mining
(“connec9ng
the
dots”)
– Top-­‐down
predic9ons
(“what
are
vulnerabili9es??”)
• Clearly,
you
need
both
• Must
apply
approaches
itera9vely
and
repeatedly

3. 2.
Observa9ons
are
DVs
• Are
the
paGerns
you
“see”
the
ones
that
are
“relevant”
or
causal?
• Problem
of
data
sparsity
and
false
correla9ons
• Hypothesis
tes9ng
requires
an
experiment
(manipula9ng
an
IV)
• Tension
between
“ecology”
and
“control
of
variables”
(sociology
of
preferred
methods)

4. 3.
How
expand
hypothesis
space?
• If
large/standard
datasets,
then
evalua9on
becomes
stagnant
(only
evaluated
with
that
dataset)
• If
evalua9on
only
uses
standard
(sta9s9cal)
tools,
same
problem
of
stagna9on
• Is
clever
visualiza9on
the
key
to
hypothesis
forma9on,
even
if
“simple”
variables?
TED
talk
by
Deb
Roy
from
MIT

5. 4.
When
do
you
give
up?
• Reliance
on
visual
paGern
recogni9on
by
human
coder
may
not
reveal
relevant
(informa9ve)
features
(sound
spectrogram
cannot
be
“read”)
• Failure
at
macro
level
prompts
search
for
info
at
micro
level
(fMRI
univariate
vs.
mul9variate
analysis):
need
to
“drill
down”
• Failure
at
micro
level
may
indicate
indeterminacy
of
causal
hierarchy
(Fodor)

6. 5.
Rules
of
sharing
• When
does
“your”
data
become
accessible
by:
– Your
collaborators
– Friends
who
ask
– Strangers
– Anyone
• Who
gets
credit?
• How
should
junior
researchers
“share”?
Especially
with
senior
labs
that
have
$$$.