IoT Data Stream Processing Architecture
•
1 like•192 views
IoT and DataStream: How to solve the data processing for the IoT Devices.
Recommended
Recommended
More Related Content
Similar to IoT Data Stream Processing Architecture
Similar to IoT Data Stream Processing Architecture (20)
Recently uploaded
Recently uploaded (20)
IoT Data Stream Processing Architecture
- 1. slide 1 July, 2008 IoT e Data Stream Jorge Cavalcanti Barbosa Fonsêca (jorge.fonseca@upe.br)
- 2. Quem sou eu? Professor UPE SD, SD, SD Jorge Fonsêca
- 3. Computação Ubíqua “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” (Mark Weiser – The Computer for the 21st Century)
- 5. Computação Ubíqua Objetos Físicos Objetos Eletrônicos
- 6. 6slide 6
- 7. 7slide 7
- 8. 8slide 8
- 9. 9slide 9 IoT
- 10. 10slide 10 IoT
- 11. 11slide 11 IoT
- 12. IoT Fonte: (Bonomi et al; 2014)
- 15. 15slide 15 Internet of Things Fonte: http://www.3g.co.uk
- 16. 16slide 16 Internet of Things Fonte: http://www.123rf.com Desafios: Heterogeneidade de dispositivos Escalabilidade Troca de dados ubíqua Otimização de energia Localização e rastreamento Autogerenciamento Gerenciamento de dados Semântica de interoperabilidade Segurança e privacidade Mobilidade
- 17. 17slide 17 IoT e Big Data = IoT Data
- 19. 19slide 19 IoT Typical data flow in an IoT solution https://www.youtube.com/watch?v=y9KMkjAn_b0
- 20. 20slide 20 IoT Solution Architecture
- 21. 21slide 21 Gartner Hype Cycle
- 22. 22slide 22 KNoT
- 25. 25slide 25 Fog Computing Fonte: (Bonomi et al; 2014)
- 26. 26slide 26 Como processar esses dados? 2004 MapReduce é um modelo de programação, e framework introduzido pelo Google para suportar computações paralelas em grandes coleções de dados em clusters de computadores
- 27. 27slide 27 Internet of Things
- 28. 28slide 28 Map Reduce – Word Count Example
- 30. 30slide 30 Map Reduce • But MR is Batch Processing… – V = Volume – Time is not an issue – Has access to all data – Might compute something big and complex – Is generally more concerned with throughput than latency of individual components of the computation – Has latency measured in seconds/minutes • Depends on the size of the data
- 34. 34slide 34 Cenários Network Intrusion Detection
- 38. 38slide 38 Cenários O que esses cenários tem em comum? Fluxo contínuo de dados Como você projetaria/implementaria um sistema que fica monitorando dados bancários, checando dados do twitter e variação da bolsa de valores?
- 39. 39slide 39 Streams Tim Bass – DEBS 2007 - Slideshare
- 40. 40slide 40 Streams Tim Bass – DEBS 2007 - Slideshare
- 41. 41slide 41 Streams Tim Bass – DEBS 2007 - Slideshare
- 42. 42slide 42 Streams Tim Bass – DEBS 2007 - Slideshare
- 43. 43slide 43 Streams Tim Bass – DEBS 2007 - Slideshare
- 44. 44slide 44 Streams Tim Bass – DEBS 2007 - Slideshare
- 45. 45slide 45 Streams Tim Bass – DEBS 2007 - Slideshare
- 46. 46slide 46 Streams • We developed a set of event processing principles and techniques for analyzing a cloud of events CEP 1996
- 47. 47slide 47 CEP EventoS simples Evento Complexo
- 48. 48slide 48 Streams Tim Bass – DEBS 2007 - Slideshare
- 49. 49slide 49 CEP Tim Bass – DEBS 2007 - Slideshare
- 50. 50slide 50 CEP Tim Bass – DEBS 2007 - Slideshare
- 51. 51slide 51 Data Stream (Cugola) 70 páginas – Visão Geral
- 52. 52slide 52 História • Human Active - Database-Passive (HADP) vs. • Database Active - Human-Passive (DAHP) DBMS Active DBMS DSMS CEP (?)
- 53. 53slide 53 IFPs IFP (Information Flow Processing) Systems
- 54. CEP (Complex Event Processing) • Based on publish-subscribe – Topic vs. content-based – subscriptions may refer to single events only and cannot take into account the history of already received events or relationships between events • CEP as an extension of publish-subscribe (register for composite events) • detect complex patterns of incoming items, involving sequencing, ordering relationships creating patterns (different from DSMS)
- 55. CEP (Functional Model) Internal memory not present in all IFP engines (Smoke, temperature) Fire Alert (Notify chemical alert)
- 56. detection phase production phase CEP (Functional Model)
- 57. EPL (Rules) • The Event Processing Language (EPL) is a SQL- standard language with extensions – offering SELECT, FROM, WHERE, GROUP BY, HAVING and ORDER BY clauses. – Streams Tables – Events Rows • Since events are composed of data – joins, filtering and aggregation through grouping
- 58. EPL (Rules) "select * from Carro where placa like '%1234'" User-defined functions
- 59. 59slide 59 CEP - Esper
- 60. 60slide 60 CEP - Window
- 61. Tools - CEP Storm + CEP
- 62. Tools – Como distribuir? Introducing Pattern
- 63. Tools – Como distribuir?
- 64. Twitter Heron Twitter has completely replaced Storm with Heron which is currently processing “several tens of terabytes of data, generating billions of output tuples” per day, “delivering 6-14X improvements in throughput, and 5-10X reductions in tuple latencies” on a standard word count test, and resulting in a 3X reduction in hardware
- 65. ZooKeeper…
- 66. 66slide 66 Ciência / Engenharia de Dados
- 67. 67slide 67 Ciência de Dados - Onde Estudar?
- 69. Proposta Um arquitetura orientada a políticas de coordenação para processamento distribuído de eventos complexos na IoT slide 71
- 70. Arquitetura Fog CEP Fog CEP Fog CEP Fog CEP Cloud Cluster Cloud – Layer 3Fog – Layer 2Devices – Layer 1 Cloud CEP CEP CEP
- 71. Cloud Layer 1 Layer 2 Layer 3 Leader Sink node Cluster Orquestrador de Agentes
- 72. 74slide 74 Agent Orchestrator Deployment Map Policies CEP Rules Context Agents and Clusters Função de Tomada de Decisão (Ftd)
- 73. 75slide 75 Políticas Environment Agent Orchestrator Clusters Policy Scope definition Policy Repository Enviroment Policies Cluster Policies PMT Policy Decision Point CEP Engine (d) Filter Filter (b) (c) (e) (f) (a) Executed into clusters and environment PEP
- 74. slide 76 July, 2008 IoT e Data Stream Jorge Cavalcanti Barbosa Fonsêca