OPEN
CLOUD
CONSORTIUM
IMAGE

PROCESSING
FOR
DISASTER
RELIEF

Image
Cache
and
Image
Delta

GOALS
FOR
PROCESSING
MAP
IMAGERY

•  Make
imagery
available
for
Disaster
Relief

workers
over
the
web

•  Provide
a
mechan...
•  Source
imagery
can
be
very
large

– New
image
formats
can
be
~2G

– Compare
image
sets
easily

•  New
data
daily

– NAS...
USE
CASES
FOR
THIS
FRAMEWORK

•  Disasters

– Fires

– Floods

– Earthquakes

– DeforestaJon

– Drought

– War/Refugees

–...
TOOLS
AND
THE
PROCESSING
PLATFORM

•  OCCTestbed
pla^orm

–  Resources
for
processing
large
data

–  Testbed
of
mulJple
cl...
INTERFACING
WITH
RESULTS

•  Open
GeospaJal
ConsorJum
Web
Map
Service

–  Images
available
through
OGCWMS
open
specificaJon...
ARAL
SEA
1989
AND
2008

ZOOM
LEVELS
/
BOUNDS

Zoom
Level
1:
4
images
 Zoom
Level
2:
16
images

Zoom
Level
3:
64
images
 Zoom
Level
4:
256
images

Mapper
Input
Key:
Bounding
Box

Mapper
Input
Value:

Mapper
Output
Key:
Bounding
Box

Mapper
Output
Value:

Mapper
resizes...
Reducer
Key
Input:
Bounding
Box

(minx
=
‐45.0
miny
=
‐2.8125
maxx
=
‐43.59375
maxy
=
‐2.109375)

Reducer
Value
Input:

St...
Mapper
Input
Key:
Bounding
Box

Mapper
Input
Value:

Mapper
Output
Key:
Bounding
Box

Mapper
Output
Value:

Mapper
resizes...
Reducer
Key
Input:
Bounding
Box

(minx
=
‐45.0
miny
=
‐2.8125
maxx
=
‐43.59375
maxy
=
‐2.109375)

Reducer
Value
Input:

St...
GULF
OIL
SPILL

Day
115
 Day
128
 Delta

SAMPLES
/
FLOODS
IN
PAKISTAN

2010
day
197
 2010
day
263

Delta

SAMPLES
/
FLOODS
IN
PAKISTAN

2010
day
197
 2010
day
263
 Delta

HBASE
TABLES

•  OGC
WMS
Query
translates
to
HBase
scheme

– Layers,
Styles,
ProjecJon,
Size

•  Table
name:
WMS
Layer

– ...
Upcoming SlideShare
Loading in...5
×

TexelTek - Andrew Levine - Hadoop World 2010

1,612

Published on

Hadoop Image Processing for Disaster Relief

Andrew Levine
TexelTek

Published in: Technology, Art & Photos
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,612
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "TexelTek - Andrew Levine - Hadoop World 2010"

  1. 1. OPEN
CLOUD
CONSORTIUM
IMAGE
 PROCESSING
FOR
DISASTER
RELIEF
 Image
Cache
and
Image
Delta

  2. 2. GOALS
FOR
PROCESSING
MAP
IMAGERY
 •  Make
imagery
available
for
Disaster
Relief
 workers
over
the
web
 •  Provide
a
mechanism
for
large
scale
image
 processing
Satellite/Map
Imagery
 •  Provide
image
deltas
for
temporally
different
 and
geospaJally
idenJcal
image
sets

  3. 3. •  Source
imagery
can
be
very
large
 – New
image
formats
can
be
~2G
 – Compare
image
sets
easily
 •  New
data
daily
 – NASA
E01
mission
tasking
 for
fires
and
floods
 •  Pass
over
areas
about
every
3rd
day
 •  High
availability
for
results
 HaiJ
image:
 18,878px
by
34,782px
 TOOLS
AND
THE
PROCESSING
PLATFORM
MOTIVATION
FOR
CLOUD
IMPLEMENTATION

  4. 4. USE
CASES
FOR
THIS
FRAMEWORK
 •  Disasters
 – Fires
 – Floods
 – Earthquakes
 – DeforestaJon
 – Drought
 – War/Refugees
 – Tornados
 •  Other
Processing
 – Medical
Imagery
 – Anomaly
 DetecJon
 – Full
MoJon
Video
 – Tracking
 – Digital
Cinema

  5. 5. TOOLS
AND
THE
PROCESSING
PLATFORM
 •  OCCTestbed
pla^orm
 –  Resources
for
processing
large
data
 –  Testbed
of
mulJple
clouds
 –  UIC
cloud
is
32
nodes
 •  Quad
Core,
16GB
RAM,
GigE,
HDFS
on
256GB
 •  Apache
Hadoop:
MapReduce
and
HBase
 –  Algorithm
adheres
to
MapReduce
framework
 –  hcp://hadoop.apache.org/
 •  OCC
Image
Processing
tools
(open
source)
 –  hcp://code.google.com/p/matsu‐project/
 –  Image
comparison

  6. 6. INTERFACING
WITH
RESULTS
 •  Open
GeospaJal
ConsorJum
Web
Map
Service
 –  Images
available
through
OGCWMS
open
specificaJon
 –  hcp://www.opengeospaJal.org/
 •  OCC
WMS
Servlet
(open
source)
 –  hcp://code.google.com/p/matsu‐project/
 •  Various
Map
Viewing
Tools
 –  OpenLayers,
Google
Maps,
others

  7. 7. ARAL
SEA
1989
AND
2008

  8. 8. ZOOM
LEVELS
/
BOUNDS
 Zoom
Level
1:
4
images
 Zoom
Level
2:
16
images
 Zoom
Level
3:
64
images
 Zoom
Level
4:
256
images

  9. 9. Mapper
Input
Key:
Bounding
Box
 Mapper
Input
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
resizes
and/or
cuts
up
the
original
 image
into
pieces
to
output
Bounding
Boxes
 (minx
=
‐135.0
miny
=
45.0
maxx
=
‐112.5
maxy
=
67.5)
 Step
1:
Input
to
Mapper
 Step
2:
Processing
in
Mapper
 Step
3:
Mapper
Output
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Build
Tile
Cache
in
the
Cloud
‐
Mapper

  10. 10. Reducer
Key
Input:
Bounding
Box
 (minx
=
‐45.0
miny
=
‐2.8125
maxx
=
‐43.59375
maxy
=
‐2.109375)
 Reducer
Value
Input:
 Step
1:
Input
to
Reducer
 …
 Step
2:
Reducer
Output
 Assemble
Images
based
on
bounding
box
 •  Output
to
HBase
 •  Builds
up
Layers
 for
WMS
for
 various
datasets
 Build
Tile
Cache
in
the
Cloud
‐
Reducer

  11. 11. Mapper
Input
Key:
Bounding
Box
 Mapper
Input
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
resizes
and/or
cuts
up
the
original
 image
into
pieces
to
output
Bounding
Boxes
 (minx
=
‐135.0
miny
=
45.0
maxx
=
‐112.5
maxy
=
67.5)
 Step
1:
Input
to
Mapper
 Step
2:
Processing
in
Mapper
 Step
3:
Mapper
Output
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 Mapper
Output
Key:
Bounding
Box
 Mapper
Output
Value:
 +
Timestamp
 +
Timestamp
 +
Timestamp
 +
Timestamp
 +
Timestamp
 +
Timestamp
 +
Timestamp
 +
Timestamp
 +
Timestamp
 Image
Processing
in
the
Cloud
‐
Mapper

  12. 12. Reducer
Key
Input:
Bounding
Box
 (minx
=
‐45.0
miny
=
‐2.8125
maxx
=
‐43.59375
maxy
=
‐2.109375)
 Reducer
Value
Input:
 Step
1:
Input
to
Reducer
 …
 …
 Step
2:
Process
difference
in
Reducer
 Assemble
Images
based
on
Jmestamps
and
compared
 Result
is
a
delta
of
the
two
Images
 Step
3:
Reducer
Output
 All
images
go
to
different
map
layers
set
of
images
for
display
in
WMS
 Timestamp
1
 Set
 Timestamp
2
 Set
 Delta
Set
 Image
Processing
in
the
Cloud
‐
Reducer

  13. 13. GULF
OIL
SPILL
 Day
115
 Day
128
 Delta

  14. 14. SAMPLES
/
FLOODS
IN
PAKISTAN
 2010
day
197
 2010
day
263
 Delta

  15. 15. SAMPLES
/
FLOODS
IN
PAKISTAN
 2010
day
197
 2010
day
263
 Delta

  16. 16. HBASE
TABLES
 •  OGC
WMS
Query
translates
to
HBase
scheme
 – Layers,
Styles,
ProjecJon,
Size
 •  Table
name:
WMS
Layer
 – Row
ID:
Bounding
Box
of
image
 ‐Column
Family:
Style
Name
and
ProjecJon
 


‐Column
Qualifier:
Width
x
Height
 





‐Value:
Buffered
Image


×