Terabyte scale Sensor Network data analysis using MapReduce/ Hadoop

1. Clustering
• Sensor signal is not labeled. For classification, we need to
label first, e.g. by clustering events

2. Clustering
• Sensor signal is not labeled. For classification, we need to
label first, e.g. by clustering events
Truck? Car? Noise?

3. Clustering
Windowing Windowing Windowing Windowing Windowing
Convolute Convolute Convolute Convolute Convolute
random Calculate Calculate Calculate Calculate Calculate update
clusters distance distance distance distance distance clusters
Average Average Average Average Average
per cluster per cluster per cluster per cluster per cluster

4. Clustering in Hadoop
Subsequences
Raw data Data massage
(with lead-in/out)
...
<ts, {s1, s2, ...}>
<tsi-1,{1,s1}>
<ts1, >
<tsi , {0,s1}> Reduce
split 1 Map <ts ,{-1,s }>
i+1 1 (per tsi ) <ts2, >
...
<ts3, >
split 2 Map Reduce <ts4, >
... <ts5, >
split n Map Reduce <ts6, >
lead-in ts lead-out
lead-in/out needed to center
bumps (snapping)

5. Clustering in Hadoop
Subsequences
Raw data Data massage
(with lead-in/out)
...
<ts, {s1, s2, ...}>
<tsi-1,{1,s1}>
<ts1, >
<tsi , {0,s1}> Reduce
split 1 Map <ts ,{-1,s }>
i+1 1 (per tsi ) <ts2, >
...
<ts3, >
split 2 Map Reduce <ts4, >
... <ts5, >
split n Map Reduce <ts6, >
Some clever
partitioning of keys lead-in ts lead-out
lead-in/out needed to center
bumps (snapping)

6. Clustering in Hadoop
Distance calculation parallel
Subsequences Cluster
Clustering
(with lead-in/out) centroids
<clusi,partialsums>
<ts1, > split 1 Map Reduce
<clus1, >
<ts2, > (per clusi )
split 2 Map
<ts3, >
... Reduce <clus2, >
<ts4, > split n Map
<ts5, >
<ts6, > current
cluster
update current cluster centroids
centroids
iterate
k (random)
lead-in ts lead-out

7. Clustering

8. Clustering
Truck (traffic jam)

9. Clustering
Truck (traffic jam)
Small truck (traffic jam)

10. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam)

11. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam)
Heavy truck

12. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam)
Heavy truck
Medium truck

13. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam)
Heavy truck
Medium truck
Small truck

14. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam)
Heavy truck
Medium truck
Small truck
Car

15. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam)
Heavy truck
Medium truck
Small truck
Car
Idle (noise)

16. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam)
Heavy truck
Medium truck
Small truck
Car
Idle (noise)

17. Clustering
Truck (traffic jam)
Small truck (traffic jam)
Cars (traffic jam) Truck!
Car!
Heavy truck
Medium truck
Small truck
Car
Idle (noise)

18. Performance
• MapReduce: Techniques scale linearly (6 node cluster)
• Noticeable overhead on small amounts of data
Convolution Clustering
40,00
Runtime (hours)
30,00
20,00
10,00
0
3 days 10 days 1 month 3 months
Amount of sensor data

19. Performance
• MapReduce: Techniques scale linearly (6 node cluster)
• Noticeable overhead on small amounts of data
Convolution Clustering
40,00
Runtime (hours)
30,00
20,00 66 node
10,00
cluster
0
3 days 10 days 1 month 3 months
Amount of sensor data

20. Performance
• MapReduce: Techniques scale linearly (6 node cluster)
• Noticeable overhead on small amounts of data
Convolution Clustering
40,00
Runtime (hours)
30,00
20,00 66 node
10,00
cluster
0
3 days 10 days 1 month 3 months
Amount of sensor data

21. Multi-scale analysis
• Sensor signal is composite of events that happen at
different time-scales
• Passing truck (small), traffic jam (medium), seasonal
change (long scale)
• Try to de-compose signals in ‘natural’ timescales
• Basic idea:
• Convolute data at different scales (scale space)
• Subtract key convolutions (band-pass filters)

22. Scale space
Amount of sensor data

23. Multi-scale analysis
• Subtraction of two such convolutions (band-pass filter)
Amount of sensor data

24. Scale space

25. Decomposition
S2
S4-S2
S4

26. Up to speed...
• Large-scale preprocessing allows advanced analysis
• Equation discovery
s100(t) = 1.196 s101(t) - 0.272 s102(t) + 0.156 s106(t)
• Long-term trends (regression)
• E.g. change in response, eigenfrequency,...
• Correlations:
strain
• ...
temperature

27. Dank U
Hvala Thanks
Xie Xie
Diolch
Toda
Merci
Grazie
Spasiba
Efharisto
Obrigado
Arigato
Köszönöm
Tesekkurler Danke
Dhanyavaad Gracias