WhoAmI
Entrepreneurial geek

Belgian Community

Micro-clusters for BigData enthousiasts

BigData
Services
Agenda

Part 1 : BigData
Part 2 : IoT and BigData
Part 3 : Discussions
BigData
Part 1
The Buzz
Started by Google
●
●

GFS Paper (2003)
MapReduce Paper (2004)

Storing and processing “Big” Data
~ multiple peta...
The Need

Large amount of data

IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOI
IOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOII...
The Need

OIII
IOII
IOII OO
OIII
OIII
High Speed at which
IOII
data becomes available
IOII OO
OIII
OIII
IOII
IOII OO
OIII
The Need

IOII

IOII

The Different ways data
is structured

IOII
IOII

IOII

IOII IOIIIOIIIOII
IOII

IOII
IOII

IOII

IOI...
Technologies
Technologies
Batch Processing:

Realtime Processing:

STORM
Hadoop

Platform for Batch Processing
Yahoo!
Doug Cutting
Mike Cafarella
Fetching and processing webpages
Storage: HDFS
Processing: YARN
Processing: MapReduce
Apache Storm

Platform for Realtime Processing
Looks like Complex Event Processing (CEP)
Backtype
Nathan Marz
Acquired by ...
Apache Storm
Apache Storm
Apache Storm
Apache Storm
Apache Storm
Myths

“BigData is all about storing data”
“BigData is only for big companies having lots of data”
“There are no BigData u...
Dangers
People are hard to find
Technologies are “different”
Too much technologies
You need hardware
Incorrect information
IoT & BigData
Part 2
Iot as datasource

Use data from “Things”
As raw as possible
No matter which format
Store everything
Iot as datasource
Iot as datasource

Adapt the processing flow based on the data flowing
through it:
- Disable faulty sensors
- Take process...
Iot as datasource
Control other “Things” based on the data flowing
through:
- Rain predicted
=> disable sprinklers
- Check...
Iot as datasource

Store data for more complex processing:
- Pattern or behaviour recognition
- Forecasting
- Prevention
Iot as datasource
Storage:
Every event is a record
Every event is immutable
Every event has:
a source
a timestamp
a type
a...
Iot as datasource
Iot as Platform - Idea
Quantity > Quality
One platform to
Store
Process
Nodes can be
Anything with a controller
Anything t...
Iot as Platform - Risks
Complex Resource Management
Exploit Capabilities
Complex Data Management
Fragmented Data
Complex S...
Iot as Platform - Gains
Keep your friends close, but your data closer.
Delegate processing
Use resources in the best way p...
What do you
think?
Part 3
IoT and BigData
Upcoming SlideShare
Loading in...5
×

IoT and BigData

939

Published on

An introduction to BigData and how it can be used in combination with the internet of things.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
939
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
79
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

IoT and BigData

  1. 1. WhoAmI Entrepreneurial geek Belgian Community Micro-clusters for BigData enthousiasts BigData Services
  2. 2. Agenda Part 1 : BigData Part 2 : IoT and BigData Part 3 : Discussions
  3. 3. BigData Part 1
  4. 4. The Buzz Started by Google ● ● GFS Paper (2003) MapReduce Paper (2004) Storing and processing “Big” Data ~ multiple petabytes
  5. 5. The Need Large amount of data IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOI IOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOI IOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI OIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOI OIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOII IOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOI OOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIIIOIOO OOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOO OIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIO IOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOI OIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOII OIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOII OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI OIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOI OIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIII OIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOI OOOOIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOI OIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOI OIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOI OIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOI OIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOI OIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOII OIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOII OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI OIOIIIOIOOOOIOIIOIOIIOIIOIOIOIIIOIOOOOI
  6. 6. The Need OIII IOII IOII OO OIII OIII High Speed at which IOII data becomes available IOII OO OIII OIII IOII IOII OO OIII
  7. 7. The Need IOII IOII The Different ways data is structured IOII IOII IOII IOII IOIIIOIIIOII IOII IOII IOII IOII IOII IOII IOII IOII IOII IOII IOIIIOII IOII IOII IOII IOII IOII IOII IOII IOII
  8. 8. Technologies
  9. 9. Technologies Batch Processing: Realtime Processing: STORM
  10. 10. Hadoop Platform for Batch Processing Yahoo! Doug Cutting Mike Cafarella Fetching and processing webpages
  11. 11. Storage: HDFS
  12. 12. Processing: YARN
  13. 13. Processing: MapReduce
  14. 14. Apache Storm Platform for Realtime Processing Looks like Complex Event Processing (CEP) Backtype Nathan Marz Acquired by Twitter Like “ever-running” MapReduce jobs
  15. 15. Apache Storm
  16. 16. Apache Storm
  17. 17. Apache Storm
  18. 18. Apache Storm
  19. 19. Apache Storm
  20. 20. Myths “BigData is all about storing data” “BigData is only for big companies having lots of data” “There are no BigData use cases in Belgium” “BigData is only being used for analytics”
  21. 21. Dangers People are hard to find Technologies are “different” Too much technologies You need hardware Incorrect information
  22. 22. IoT & BigData Part 2
  23. 23. Iot as datasource Use data from “Things” As raw as possible No matter which format Store everything
  24. 24. Iot as datasource
  25. 25. Iot as datasource Adapt the processing flow based on the data flowing through it: - Disable faulty sensors - Take processing shortcuts
  26. 26. Iot as datasource Control other “Things” based on the data flowing through: - Rain predicted => disable sprinklers - Check-out at work, trip home will take 60 min => start heating in 30 min Would be great to link with a rule-engine (Drools e.g.)
  27. 27. Iot as datasource Store data for more complex processing: - Pattern or behaviour recognition - Forecasting - Prevention
  28. 28. Iot as datasource Storage: Every event is a record Every event is immutable Every event has: a source a timestamp a type a value
  29. 29. Iot as datasource
  30. 30. Iot as Platform - Idea Quantity > Quality One platform to Store Process Nodes can be Anything with a controller Anything that can store data
  31. 31. Iot as Platform - Risks Complex Resource Management Exploit Capabilities Complex Data Management Fragmented Data Complex Security Model Obfuscate Data
  32. 32. Iot as Platform - Gains Keep your friends close, but your data closer. Delegate processing Use resources in the best way possible
  33. 33. What do you think? Part 3
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×