Ocean Data Interoperability Platform - Big Data - Streams & Workflows
•Download as PPTX, PDF•
0 likes•347 views
Presentation on streaming data at the ODIP-II workshop Big Data session, September 2015
Recommended
More Related Content
Viewers also liked
Viewers also liked (9)
Similar to Ocean Data Interoperability Platform - Big Data - Streams & Workflows
Similar to Ocean Data Interoperability Platform - Big Data - Streams & Workflows (20)
More from Adam Leadbetter
More from Adam Leadbetter (20)
Recently uploaded
Recently uploaded (20)
Ocean Data Interoperability Platform - Big Data - Streams & Workflows
- 1. ODIP II – Data Streams and Flows September 2015 Adam Leadbetter, Damian Smyth, Rob Fuller adam.leadbetter@marine.ie
- 9. 20th Century 21st Century Single machine Distributed processing One-to-one communication Publish-subscribe pattern No fault tolerance Replication, auto-recovery Fixed schema, encoding Schema management, evolvable encoding
- 10. 20th Century 21st Century Byte stream Messages In-memory Durable Blocking / backpressure Buffering One-to-one Multi-subscriber
- 21. Further Work • Look at the architectures for reprocessing historic data • Incorporate Sensor Web Enablement, OM-JSON • Deploy on vessels / mobile, remote platforms • Investigation of Apache NiFi
Editor's Notes
- Thanks – Damian Smyth & Rob Fuller (both - MI)
- We know how to do real-time. Or do we? Our real-time is often periodic, or near-real time. Can we do it at scale? Can we plug turn on a new sensor, and have it feeding web displays straight off? Throughput in real-time at scale – social networks: Twitter, Facebook, LinkedIn
- Apologies to Martin Kleppmann for the plagiarism…
- Plenty of tools out there to support this approach!
- Explore stream composition through the context of the Galway Bay cable observatory project Background on observatory
- Galway Bay observatory CTD – On a serial port with a hardware Moxa switch to make the serial connection available to multiple machines Docker container on shore station server with serial2kafka app running Shore station Kafka holds on to the data for a fortnight Kafka queue replicated across the network to HQ Raw data – stored in Cassandra – available through ERDDAP Or, some augmentation through stream processing in Storm – back on to a Kafka queue, exposed through WebSockets
- NiFi – Flow based programming Drag and drop + configuration in a workbench, i.e. as little code as possible Includes ability to fire off, say, the individual R processes from the previous slide However, may be mainly of use for ingesting data as far as a message queue..
- The Internet of Things (IoT), is the network of physical objects or "things" embedded with electronics, software, sensors, and connectivity to enable objects to collect and exchange data. Typically, IoT is expected to offer advanced connectivity of devices, systems, and services that goes beyond machine-to-machine communications (M2M) The interconnection of these embedded devices (including smart objects), is expected to usher in automation in nearly all fields
- Queues etc… like working with JSON, especially JSON with JSON-Schema Get very interesting when… OGC SWE Domain Working Group – 16th September 2015 – Simon Cox (& Peter Taylor)
- First step to SOS/SWE Demo rudimentary nginx SOS(?)