IoT and BigData

1,370 views

Published on

An introduction to BigData and how it can be used in combination with the internet of things.

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,370
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
98
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

IoT and BigData

  1. 1. WhoAmI Entrepreneurial geek Belgian Community Micro-clusters for BigData enthousiasts BigData Services
  2. 2. Agenda Part 1 : BigData Part 2 : IoT and BigData Part 3 : Discussions
  3. 3. BigData Part 1
  4. 4. The Buzz Started by Google ● ● GFS Paper (2003) MapReduce Paper (2004) Storing and processing “Big” Data ~ multiple petabytes
  5. 5. The Need Large amount of data IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOI IOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOI IOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI OIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOI OIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOII IOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOI OOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIIIOIOO OOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOO OIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIO IOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOI OIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOII OIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOII OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI OIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOI OIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIII OIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOI OOOOIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOI OIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOI OIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOI OIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOI OIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOI OIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOII OIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOII OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOI OIOIIIOIOOOOIOIIOIOIIOIIOIOIOIIIOIOOOOI
  6. 6. The Need OIII IOII IOII OO OIII OIII High Speed at which IOII data becomes available IOII OO OIII OIII IOII IOII OO OIII
  7. 7. The Need IOII IOII The Different ways data is structured IOII IOII IOII IOII IOIIIOIIIOII IOII IOII IOII IOII IOII IOII IOII IOII IOII IOII IOIIIOII IOII IOII IOII IOII IOII IOII IOII IOII
  8. 8. Technologies
  9. 9. Technologies Batch Processing: Realtime Processing: STORM
  10. 10. Hadoop Platform for Batch Processing Yahoo! Doug Cutting Mike Cafarella Fetching and processing webpages
  11. 11. Storage: HDFS
  12. 12. Processing: YARN
  13. 13. Processing: MapReduce
  14. 14. Apache Storm Platform for Realtime Processing Looks like Complex Event Processing (CEP) Backtype Nathan Marz Acquired by Twitter Like “ever-running” MapReduce jobs
  15. 15. Apache Storm
  16. 16. Apache Storm
  17. 17. Apache Storm
  18. 18. Apache Storm
  19. 19. Apache Storm
  20. 20. Myths “BigData is all about storing data” “BigData is only for big companies having lots of data” “There are no BigData use cases in Belgium” “BigData is only being used for analytics”
  21. 21. Dangers People are hard to find Technologies are “different” Too much technologies You need hardware Incorrect information
  22. 22. IoT & BigData Part 2
  23. 23. Iot as datasource Use data from “Things” As raw as possible No matter which format Store everything
  24. 24. Iot as datasource
  25. 25. Iot as datasource Adapt the processing flow based on the data flowing through it: - Disable faulty sensors - Take processing shortcuts
  26. 26. Iot as datasource Control other “Things” based on the data flowing through: - Rain predicted => disable sprinklers - Check-out at work, trip home will take 60 min => start heating in 30 min Would be great to link with a rule-engine (Drools e.g.)
  27. 27. Iot as datasource Store data for more complex processing: - Pattern or behaviour recognition - Forecasting - Prevention
  28. 28. Iot as datasource Storage: Every event is a record Every event is immutable Every event has: a source a timestamp a type a value
  29. 29. Iot as datasource
  30. 30. Iot as Platform - Idea Quantity > Quality One platform to Store Process Nodes can be Anything with a controller Anything that can store data
  31. 31. Iot as Platform - Risks Complex Resource Management Exploit Capabilities Complex Data Management Fragmented Data Complex Security Model Obfuscate Data
  32. 32. Iot as Platform - Gains Keep your friends close, but your data closer. Delegate processing Use resources in the best way possible
  33. 33. What do you think? Part 3

×