This document discusses big data and the Internet of Things (IoT). It begins with an overview of big data, including how it was started by Google and the need to store and process large amounts of data from various sources that is generated quickly. It then discusses Hadoop and Apache Storm as platforms for batch and real-time processing. The next section discusses using data from "Things" in IoT as a data source for big data systems, how to control devices based on the data, and storing data for complex analysis. Finally, it raises some ideas, risks, and gains regarding using big data platforms for IoT.

2. WhoAmI
Entrepreneurial geek
Belgian Community
Micro-clusters for BigData enthousiasts
BigData
Services

3. Agenda
Part 1 : BigData
Part 2 : IoT and BigData
Part 3 : Discussions

4. BigData
Part 1

5. The Buzz
Started by Google
●
●
GFS Paper (2003)
MapReduce Paper (2004)
Storing and processing “Big” Data
~ multiple petabytes

6. The Need
Large amount of data
IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOI
IOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOI
IOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI
OIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOI
OIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOII
IOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOI
OOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIIIOIOO
OOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOO
OIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIO
IOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOI
OIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOII
OIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOII
OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI
OIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOI
OIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIII
OIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOI
OOOOIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOI
OIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOI
OIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOI
OIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOI
OIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOI
OIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOII
OIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOII
OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI
OIOIIIOIOOOOIOIIOIOIIOIIOIOIOIIIOIOOOOI

7. The Need
OIII
IOII
IOII OO
OIII
OIII
High Speed at which
IOII
data becomes available
IOII OO
OIII
OIII
IOII
IOII OO
OIII

8. The Need
IOII
IOII
The Different ways data
is structured
IOII
IOII
IOII
IOII IOIIIOIIIOII
IOII
IOII
IOII
IOII
IOII
IOII
IOII
IOII
IOII
IOII
IOIIIOII
IOII
IOII
IOII
IOII
IOII IOII
IOII
IOII

9. Technologies

10. Technologies
Batch Processing:
Realtime Processing:
STORM

11. Hadoop
Platform for Batch Processing
Yahoo!
Doug Cutting
Mike Cafarella
Fetching and processing webpages

12. Storage: HDFS

13. Processing: YARN

14. Processing: MapReduce

15. Apache Storm
Platform for Realtime Processing
Looks like Complex Event Processing (CEP)
Backtype
Nathan Marz
Acquired by Twitter
Like “ever-running” MapReduce jobs

16. Apache Storm

17. Apache Storm

18. Apache Storm

19. Apache Storm

20. Apache Storm

21. Myths
“BigData is all about storing data”
“BigData is only for big companies having lots of data”
“There are no BigData use cases in Belgium”
“BigData is only being used for analytics”

22. Dangers
People are hard to find
Technologies are “different”
Too much technologies
You need hardware
Incorrect information

23. IoT & BigData
Part 2

24. Iot as datasource
Use data from “Things”
As raw as possible
No matter which format
Store everything

25. Iot as datasource

26. Iot as datasource
Adapt the processing flow based on the data flowing
through it:
- Disable faulty sensors
- Take processing shortcuts

27. Iot as datasource
Control other “Things” based on the data flowing
through:
- Rain predicted
=> disable sprinklers
- Check-out at work, trip home will take 60 min
=> start heating in 30 min
Would be great to link with a rule-engine (Drools e.g.)

28. Iot as datasource
Store data for more complex processing:
- Pattern or behaviour recognition
- Forecasting
- Prevention

29. Iot as datasource
Storage:
Every event is a record
Every event is immutable
Every event has:
a source
a timestamp
a type
a value

30. Iot as datasource

31. Iot as Platform - Idea
Quantity > Quality
One platform to
Store
Process
Nodes can be
Anything with a controller
Anything that can store data

32. Iot as Platform - Risks
Complex Resource Management
Exploit Capabilities
Complex Data Management
Fragmented Data
Complex Security Model
Obfuscate Data

33. Iot as Platform - Gains
Keep your friends close, but your data closer.
Delegate processing
Use resources in the best way possible

34. What do you
think?
Part 3