A UI concept demo exploring Traumatic Brain Injury (TBI) that Jeff Chang, an ER radiologist, and I presented at a 3D developers conference (zCon in April 2013 hosted by zSpace). We gave our system the name “NeuroElectric and Anatomic Locator,” or “N.E.A.A.L.

1. The
Future
of
Neuroimaging:
A
3D
Explora9on
of
TBI
Hunter
Whitney
Jeffrey
Chang,
MD

2. Speakers
Hunter
Whitney
• UX
Designer
and
Author
of
“Data
Insights”
Jeffrey
Chang,
MD
• ER
Radiologist
2

3. Disclosure
We
have
no
financial
or
commercial
conflicts
of
interest
to
disclose
3

4. Agenda
• Introduc9on
/
WIIFM
• TBI
Overview
• UX
Design
Concepts
• NEAAL
Video
• Improving
on
It
–
Imaging
and
User
Control
• Applica9ons
in
Research
• From
Research
to
Treatment
• Brain
Mapping
–
Current,
Future
and
Complica9ons
• The
Purpose
of
a
PlaXorm
• Hunt
for
Ar9ficial
General
Intelligence
4

5. 5
The
(Real)
Final
Fron9er:
The
Human
Brain
Introduc9on
Blast
medicine
anyway!
We've
learned
to
9e
into
every
organ
in
the
human
body
but
one.
The
brain!
The
brain
is
what
life
is
all
about.
-­‐Dr.
Leonard
H.
McCoy
("Bones")
(from
Star
Trek
TV
series,
The
Menagerie)

6. 6
• Average
weight
of
a
human
brain:
3
pounds
• Number
of
neurons
in
the
brain:
100
Billion
• Miles
of
blood
vessels,
capillaries
and
other
transport
systems
in
the
brain:
100,000
miles
• Number
of
connec9ons
in
the
adult
brain:
1
quadrillion
Gecng
Inside
Your
Head
is
a
Challenge

7. 7
“Since
the
brain
is
unlike
any
other
structure
in
the
known
universe,
it
seems
reasonable
to
expect
that
our
understanding
of
its
func9oning…will
require
approaches
that
are
dras8cally
different
from
the
way
we
understand
other
physical
systems.”
-­‐Richard
M.
Restak
(from
The
Brain.
The
Last
FronEer,
1979)
New
Approaches

8. 8
Crea9ng
useful
new
ways
to
model,
visualize,
and
interact
with
many
layers
of
data
about
the
brain
is
vitally
important
for
many
purposes.
New
Perspec9ves
Needed

9. WIIFM
–
Investors’
Edi9on
9
• This
is
the
Brain
Era
–
many
of
the
next
thirty
years’
technological
breakthroughs
and
their
commercial
applica9ons
will
happen
right
here
• New
industry
innova9ons
revolu9onize
our
world
each
year
–
and
they
each
relate
to
the
future
of
thought,
intui9on
and
analysis
• S&P
500
companies
last
5
years
on
average
–
you
must
glimpse
decades
ahead
in
R&D
• SXSW
2013
–
RIP
Dell,
Groupon,
HP,
B&N,
RIAA

10. I’m
a
Dev
–
WIIFM?
10
• Understand
the
issues
• Research
in
every
field
is
remarkably
varied
• Huge
gulf
between
research
and
applica9on
–
the
dev’s
work
can
bridge
that
gap,
through
UI
and
by
understanding
each
audience’s
needs
• Investment
pouring
in,
startups
and
funded
brain
mapping
projects
all
need
devs

11. Wait,
Isn’t
This
the
Research
Track?
11
• Connec9ng
research
done
by
many
different
groups
–
mul9ple
disciplines
working
as
one
• Breaking
down
silos
within
ins9tu9ons
and
across
the
world
• The
beper
you
understand
the
target
applica9on
and
end-­‐goal,
the
more
likely
you’ll
discover
something
truly
revolu9onary
–
and
the
more
likely
you’ll
get
funding
• Have
a
say
in
the
UI
–
what
do
you
actually
need?

12. Concept
for
Neuroimaging
System
12
We’re
going
to
show
an
early
concept
for
a
neuroimaging
system
that
could
be
used
for
many
purposes,
including
research
into
trauma9c
brain
injury
(TBI)

13. By
examining
the
elements
of
something
that
is
broken…
13

14. …you
can
also
gain
more
insight
into
how
it
normally
works.
14

15. TBI
Overview
15
Source:
Robert
T.
Knight,
M.D.
Professor
of
Psychology
and
Neuroscience
Department
of
Psychology
Helen
Wills
Neuroscience
Ins9tute

16. The
impact
of
the
problem…
16

17. …con9nues
to
emerge.
17

18. Many
Complex
Interac9ons
18
Just
as
a
car
is
made
up
of
a
range
of
different
parts
and
materials
that
will
be
differen9ally
impacted
in
a
collision,
far
more
so
are
the
components
of
the
brain.
It
is
really
only
possible
to
figure
out
the
full
extent
of
damage
retrospec9vely.

19. Mul9ple
Factors
19
EMP
(Electromagne8c
Pulse)

20. Dynamic
Forces
20

21. 21
UX
Design
Concepts

22. Gecng
a
Deeper
Perspec9ve
of
Loca9on
with
3D
22

23. 23
A
3D
neuroimaging
system
that
allows
a
fast,
fluid
inves9ga9on
of
heterogeneous
data
about
the
brain
from
the
popula9on
level
down
to
a
specific
neural
pathway
in
an
individual
pa9ent
Vision

24.
NEAAL
(NeuroElectric
and
Anatomic
Locator)
24

25. 25
High-­‐level
Goals
for
NEAAL
• Integra8on
-­‐
incorporate
3D
anatomical
visualiza9ons
with
related
non-­‐physical,
data
in
a
simple,
elegant
display
• Interac8on
-­‐
maximize
visual
display
for
primary
work
goals
and
employ
verbal
and
gestural
input
for
the
func9onal
tasks
(NEAAL
is
no
“WIMP”)
• Orienta8on
-­‐
help
users
maintain
context
as
they
move
through
an
analy9c
process
while
s9ll
not
overloading
the
display
(ephemeral
context)

26. 26
• Localiza8on
-­‐
allow
users
to
quickly
and
easily
hone
in
on
and
mark
points
of
interest
• Accelera8on
-­‐
enable
faster
workflows
and
more
rapid,
itera9ve
hypothesis
tes9ng.
High-­‐level
Goals
for
NEAAL
(Cont.)

27. NEAAL
Applies
Ben
Shneiderman’s
Mantra
“Overview
first,
zoom
and
filter,
then
details-­‐on-­‐
demand”
27

28. 28
Scenario
for
NEAAL

29. In
2008,
a
soldier
in
Afghanistan
suffers
a
TBI…
29

30. …and
subsequent
depression
and
PTE
(post-­‐trauma9c
epilepsy).
Certain
notable
features
of
the
case
are
flagged
by
the
clinician
and
aggregated
with
similar
cases
History
&
Physical
30

31. Certain
PTE
Cases
with
Characteris9c
Apributes
are
Aggregated
History
&
Physical
Aggregated
H&P
Data
Aggregated
View
31

32. A
Researcher
Starts
with
the
Aggregate
and
then
Moves
to
the
Individual
Case
Individual
View
32
Aggregate
View

33. 33
Integra8on
Paths
Loca9on
and
Scale
Structural,
Func9onal,
Cogni9ve,
Demographic
Views
Inves9ga9ve
and
Anatomical
The
Big
Picture
H&P
CT

34. 34
CT
Popula9on
Cog
ANAM
H&P
PTE
Inves8ga8on
Steps
of
an
Inves9ga9on
(PTE)

35. 35
CT
Popula9on
Cog
ANAM
Steps
of
an
Inves9ga9on
(Depression)
H&P
Depression
Inves8ga8on

36. Traveling
on
an
Anatomical
Path
with
Different
Imaging
Modali9es
MRI
+
DTI
DTI
36

37. Scale
Changes
from
Large
Structural
Features
to
Discrete
Neural
Tracts
37

38. 38
Remember
the
Future?

39. Deckard’s
Image
Scanner
39
“Enhance
15
to
23”

40.
Image
Scanner
Next
Gen
What
can
be
improved?
• The
Blade
Runner
vision
is
interes9ng
but
would
be
cumbersome
for
the
researcher
in
our
scenarios;
mul9modal
3D
is
a
more
robust
and
easier
to
use
vision.
• Another
Image
Scanner
Next
Gen
idea…
“Print
a
hard
copy.”
Why
not
do
that
with
a
3D
print
of
the
brain
and
locus
of
injury?
40

41. 41
NEAAL
Video

42. Disclaimer:
Imaging
Limita9ons
42
San&ago
Ramón
y
Cajal,
Drawing
of
a
single
neuron,
1899
Jiang
X
et
al.
The
organizaEon
of
two
new
corEcal
interneuronal
circuits,
Nature
Neuroscience
2013

43. MVP
Concept
Disclaimer
43
• Consider
a
dynamic
interface
– Gestural
control
of
the
flyover
– Rapid
gestural
or
voice-­‐driven
zoom
and
manipula9on
– Instant
localiza9on
of
any
part
of
the
brain
– Tracks
mul9ple
modali9es
at
once,
and
remembers
which
overlays
provide
complementary
informa9on

44. Imaging
will
change
…
44
Improved
Stroke
Imaging
Techniques,
JAMA
1999
Zhang,
W.
et
al.
Landmark-­‐referenced
voxel-­‐based
analysis
of
diffusion
tensor
images
of
the
brainstem
white
ma]er
tracts.
NeuroImage
2009
Laundre,
B
et
al.
Diffusion
Tensor
Imaging
of
the
CorEcospinal
Tract
before
and
a^er
Mass
ResecEon.
AJNR
2005
Christoforidis,
G.
et
al.
“Tumoral
Pseudoblush”
IdenEfied
within
Gliomas
at
High-­‐SpaEal-­‐
ResoluEon
Ultrahigh-­‐Field-­‐Strength
Gradient-­‐Echo
MR
Imaging.
Radiology
2012

45. We
will
live
to
see
the
end
of
the
mouse
…
45

46. Non-­‐Invasive
BCI
46
Emo8v
EPOC
-­‐
2008
g.Tec
intendiX-­‐
SPELLER
-­‐
2012
EPOC
with
AutoNOMOS-­‐Labs

47. 47
What
Other
Applica9ons
Need
Robust
Tools?
One
plaXorm,
mul9ple
possibili9es
• Tissue
Bioengineering
• Organism
Simula9on

48. Tissue
Bioengineering?
48
From
Mosby
Year-­‐Book
Anthony
Atala:
PrinEng
a
Human
Kidney
(TED
2011)

49. Organism
Simula9on
–
for
Aging,
Disease
and
Pharma
49
Modeling
of
a
Biological
Cell
Model,
MarEn
Falk,
Universität
Stu]gart
Marcus
Covert
Systems
Biology
Lab,
Stanford

50. From
Research
to
Treatment
50
Dr.
Balaji
Anvekar’s
Neuroradiology
Cases;
SP
Ins9tute
of
Neurosciences,
Solapur,
India
-­‐
2012

51. AI
in
the
Hyperacute
Response
51
Keyhole
neurosurgery
–
EU
ROBOCAST
• Bigger
robot
holding
smaller
robot
July
2011,
Baghdad
–
Wealth
of
Health
/
Neuroscience
News

52. The
Future
of
TBI
Treatment
52
Studies
of
axonal
regeneraEon
in
Drosophila
(fruit
flies),
Melissa
Rolls,
Penn
State
University
Nerve
Replacement
Strategies
for
Cavernous
Nerves
May,
F
et
al.
European
Urology
2005(48:3)
Salvador,
G.
Uranga,
R
and
Giusto,
N.
Iron
and
Mechanisms
of
Neurotoxicity.
InternaEonal
Journal
of
Alzheimer’s
Disease,
2011

53. Hurdles
to
Healing
the
Aging
Mind
53
Scheltens,
Philip.
Imaging
in
Alzheimer’s
Disease.
Dialogues
in
Clinical
Neuroscience
2009(11)

54. • The
road
from
Assis9ve
Robo9cs
to
Automa9on
• Automated
clinical
care
algorithms,
especially
with
a
new
genera9on
of
physicians
• Rapid
tes9ng,
immediate
results
for
more
labs
and
radiology,
shortened
stays
(ACO
models)
Disrup9ng
a
Conserva9ve
Industry
54
Automated
ICU
SedaEon
@
Georgia
Tech
–
Wassim
Haddad,
Allen
Tannenbaum
and
Behnood
Gholami
Prof.
Allison
Okamura’s
HapEc
ExploraEon
Lab
at
JHU
(now
at
Stanford)

55. Brain
Mapping
IBM
Researchers
Create
the
Most
Detailed
Brain
Map
Yet
“A
significant
stride
towards
reverse-­‐engineering
the
darn
thing.”
55
July
27th,
2010
410
papers,
50
years,
CoCoMac
database
of
the
Macaque
brain
383
brain
regions,
6,602
directed
long-­‐
distance
connec9ons
“The
data
is
of
the
monkey,
by
the
people,
and
for
the
people.”
–
Dharmendra
Modha,
SyNAPSE

56. CLARITY
–
innova9on
beckons
56
CLARITY
–
Intact
mouse
brain
stained
with
fluorescent
protein-­‐specific
labels.
Kwanghun
Chung
and
Karl
Deisseroth,
Howard
Hughes
Medical
Ins8tute
/
Stanford
University

57. The
Growing
Wave
57

58. “The
Next
Fron9er”
58
I
think
the
biggest
innova&ons
of
the
21st
century
will
be
at
the
intersec&on
of
biology
and
technology.
A
new
era
is
beginning.”
–
Steve
Jobs,
2011

59. Building
the
Universal
PlaXorm
59
Rita
Carter
–
Mapping
the
Mind:
Revised
and
Updated
EdiEon
(2010)
Milky
Way
will
collide
with
Andromeda
in
4
billion
years;
courtesy
of
NASA

60. Problem
#1
60

61. Problem
#2
61

62. Problem
#3
62

63. 63
It’s
Even
More
Complicated
Sprout
Labs
Australia
Buxhoeveden,
D.
and
Casanova,
M.
The
minicolumn
hypothesis
in
neuroscience.
Oxford
Journals:
Brain
2001

64. Issues
with
the
Mind
64
Men
ought
to
know
that
from
the
brain,
and
from
the
brain
only,
arise
our
pleasures,
joy,
laughter
and
jests,
as
well
as
our
sorrows,
pains,
griefs,
and
tears.
–
Hippocrates
of
Cos
(circa
400
BC)

65. What
Counts
as
a
Mapped
Brain?
65

66. The
PlaXorm
as
a
Guide
66

67. Progress
in
Brain
Mapping
67
Allen
Ins8tute
for
Brain
Science
(2003)
$300M
from
2012-­‐2016
Human
Brain
Atlas
–
2011

68. Progress
in
Brain
Mapping
68
The
Human
Connectome
Project
Started
August
2012,
$30M
UCLA
–
MGH,
Washington
U.
–
U.
Minnesota
LPBA
–
the
ProbabilisEc
Brain
Atlas
at
UCLA

69. Progress
in
Brain
Mapping
69
Aggrega9on
of
1200
brain
MRIs,
including
300
pairs
of
twins
Increasing
resolu9on
of
the
reference
MRI
map
to
1
mm
MarEnos
Center
at
MGH
(Harvard)

70. The
Future
of
Brain
Simula9on
70
“CERN
For
The
Brain”
The
Human
Brain
Project
@
EPFL
(Lausanne,
Switzerland)
Awarded
€1.19B
over
10
years
by
the
EC’s
FET
flagship
Compila9on
of
global
neuroscience
data,
will
build
plaXorm
to
help
researchers
with
neuromorphic
compu9ng
and
designing
neurorobo9cs
Collabora9ve
effort
Blue
Brain
+
87
European
and
interna9onal
partners
10,000
simulated
neurons,
30
million
synapses,
forming
part
of
a
single
corEcal
column
in
the
rat
brain;
from
HBP
in
2008

71. The
Supercomputer
Approach
71
TrueNorth,
on
LLNL’s
Blue
Gene
/
Q
Sequoia
(2nd
fastest
supercomputer
in
the
world)
96
racks
(1,572,864
cores,
1.5PB
memory,
6,291,456
threads)
553.5
billion
neurons
100
trillion
synapses
(DARPA’s
SyNAPSE)
1
/
1542
the
speed
of
the
human
brain
The
actual
human
brain
has
86
–
100
billion
neurons
and
100
trillion
to
1
quadrillion
synapses;
average
es9mate
at
350
trillion
synapses
Simula9on
at
approximately
4.8%,
or
1/20th,
the
synap9c
density
of
the
human
brain
(synapses
per
neuron)
Each
dot
represents
a
neurosynapEc
core,
containing
256
neurons;
1024
synapses
per
neuron.
2.084
billion
cores,
divided
into
77
brain
regions,
using
the
macaque
brain
as
the
template

72. Func9on-­‐Focused
72
Spaun
–
U.
Waterloo
Largest
simula9on
of
a
func9oning
brain,
with
2.5
million
separately
modeled
spiking
neurons
Performs
a
variety
of
tasks;
very
useful
as
a
model
for
managing
the
flow
of
informa9on
through
a
large
system,

73. Culturing
the
Brain
73
To
understand
the
development
of
synapses
and
spontaneous
excita9on
on
a
cellular
level
MIRA
InsEtute,
University
of
Twente
–
November
2012;
neurite
morphology
in
a
simulated
Petri
dish
of
10,000
neurons

74. Living
Neural
Networks
74
Removing
some
‘A’
from
AI:
Embodied
Cultured
Networks
(2004)
–
GA
Tech,
MIT,
U.
Western
Australia,
U.
Florida
(follow-­‐up
global
research
from
2004
to
2012)

75. Simula9ng
the
Brain
in
Real
Time
75
Neurogrid
Modeling
selec&ve
aPen&on
in
the
visual
cortex,
by
increasing
the
gain
of
excitatory
neurons.
Analog
computa9on
(parallel)
to
emulate
ion-­‐channel
ac9vity,
and
digital
synap9c
connec9ons.
Simulates
1
million
neurons
and
6
billion
synapses
in
real-­‐9me,
using
only
5
waps
of
power
Nick
Steinmetz,
2011
@
Stanford

76. AI
–
The
Eternal
Horizon
76

77. The
Road
to
AGI
77
Sandberg,
Anders;
Bostrom,
Nick
(2008).
Whole
Brain
EmulaEon:
A
Roadmap.
Future
of
Humanity
InsEtute,
Oxford
University

78. 78
Lvl
Extent
of
Whole
Brain
Emula8on
#
of
en88es
Storage
Demands
(Tb)
Earliest
Year
($B
projects)
CPU
Demand
(FLOPS)
Earliest
Year
($B
projects)
2
Brain
Region
Connec8vity
105
regions,
107
connec9ons
3
x
10-­‐5
Achieved
-­‐-­‐
Achieved
3
Analog
network
popula8on
model
108
popula9ons,
1013
connec9ons
50
Achieved
1015
Achieved
4
Spiking
neural
network
1011
neurons,
1015
connec9ons
8,000
2016
1018
2018
5
Electrophysiology
1015
compartments
x
10
state
variables
10,000
2016
1022
2030
6
Metabolome
1016
compartments
x
102
metabolites
106
2024
1025
2040
7
Proteome
1016
compartments
x
103
proteins
107
2028
1026
2044
8
State
of
protein
complexes
1016
compartments
x
103
proteins
x
10
states
108
2031
1027
2047
9
Distribu8on
of
complexes
1016
compartments
x
103
proteins
x
100
states
109
2035
1030
2057
10
Stochas8c
behavior
of
single
molecules
1025
molecules
3.1
x
1014
2055
1043
2100
11
Quantum
states
Approx.
1026
atoms
Using
Qbits
?
Using
Qbits
?
Sandberg,
Anders;
Bostrom,
Nick
(2008).
Whole
Brain
EmulaEon:
A
Roadmap.
Future
of
Humanity
InsEtute,
Oxford
University

79. The
AGI
Timeline
79
Sandberg,
Anders;
Bostrom,
Nick
(2008).
Whole
Brain
EmulaEon:
A
Roadmap.
Future
of
Humanity
InsEtute,
Oxford
University

80. The
Singularity?
80
“In
the
future,
search
engines
should
be
as
useful
as
HAL
in
the
movie
2001:
A
Space
Odyssey
–
but
hopefully
they
won’t
kill
people.”
–
Sergey
Brin
“In
the
game
of
life
and
evolu&on
there
are
three
players
at
the
table:
human
beings,
nature,
and
machines.
I
am
firmly
on
the
side
of
nature.
But
nature,
I
suspect,
is
on
the
side
of
the
machines.”
–
George
Dyson

81. AGI
–
Current
Efforts
81
Vicarious,
Genifer,
Numenta,
OpenCog,
OpenNARS,
A2I2,
Cyc,
Soar,
the
Google
Moonshot
Factory
Every
&me
I
talk
about
Google’s
future
with
Larry
Page,
he
argues
that
it
will
become
an
ar&ficial
intelligence.”
–
Steve
Jurvetson,
Draper
Fisher
Jurvetson

82. The
Next
Decade
1. Building
the
necessary
tools,
for
discovery
and
applica8on
82
If
you
invent
a
breakthrough
in
ar&ficial
intelligence,
so
machines
can
learn,
that
is
worth
10
Microsos.”
–
Bill
Gates,
2004
2. Keeping
abreast
of
the
8meline
for
Brain
Mapping
efforts;
finding
the
right
ques8ons
to
ask,
for
new
weak
AI
applica8ons
3. Will
your
startup’s
logo
be
on
this
slide
in
2023?

83. Acknowledgments
83
Special
Thanks:
Michael
Aratow,
MD
Lee
Hall,
M.D.
Jason
Collins,
Canopy
Partners
Jeanne
Rayne,
Canopy
Partners
Veena
Kumar,
MD,
MPH
Paul
Laurien9,
MD
The
zSpace
team
Video
Produc8on:
Spencer
Lindsay,
Lindsay
Digital
Ruby
Rieke