Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FIWARE Global Summit - Real-time Processing of Historic Context Information using Apache Flink

241 views

Published on

Presentation by the UPM Team
Sonsoles López, Andres Muñoz, Joaquin Salvachua and Gabriel Huecas

FIWARE Global Summit
27-28 November 2018
Malaga, Spain

Published in: Technology
  • Be the first to comment

FIWARE Global Summit - Real-time Processing of Historic Context Information using Apache Flink

  1. 1. FIWARE Orion Flink Connector Sonsoles López (slopez@dit.upm.es) Andres Muñoz (jamunoz@dit.upm.es) Joaquin Salvachua (jsalvachua@dit.upm.es) Gabriel Huecas (ghuecas@dit.upm.es) Universidad Politécnica de Madrid @jsalvachua, @FIWARE
  2. 2. Flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. 2
  3. 3. Previous Architecture 3
  4. 4. Current Architecture 4
  5. 5. fiware-cosmos-orion-flink-connector https://github.com/ging/fiware-cosmos-orion-flink-connector https://fiware-cosmos-flink.readthedocs.io Two parts: ● OrionSource: Receive notifications from the Context Broker ● OrionSink: Write data into the Context Broker 5
  6. 6. Connection 6 ORION Context Broker Flink Cluster Flink Job (JAR) orion-flink-connector HTTP POST (Notification) HTTP POST/PUT/PATCH OrionSource OrionSink
  7. 7. OrionSource Receives data from the Orion Context Broker from a given port. The received data is a Stream of NgsiEvent objects val eventStream = env.addSource(new OrionSource(9001)) 7
  8. 8. OrionSink Sends data back to the Orion Context Broker Takes a stream of OrionSinkObjects as a source: ● content: Message content in String format. If it is a JSON, it needs to be stringified ● url: URL to which the message should be sent ● contentType: Type of HTTP content of the message (JSON, Plain) ● method: HTTP method of the message (POST, PUT, PATCH) OrionSink.addSink( processedDataStream ) 8
  9. 9. Basic Example final val URL_CB = "http://flinkexample_orion_1:1026/v2/entities/" final val CONTENT_TYPE = ContentType.JSON final val METHOD = HTTPMethod.POST def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment // Create Orion Source. Receive notifications on port 9001 val eventStream = env.addSource(new OrionSource(9001)) // Process event stream val processedDataStream = eventStream .flatMap(event => event.entities) .map(entity => { val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue() new Temp_Node(entity.id, temp) }) .keyBy("id") .timeWindow(Time.seconds(5), Time.seconds(2)) .min("temperature") .map(tempNode => { val url = URL_CB + tempNode.id + "/attrs" OrionSinkObject(tempNode.toString, url, CONTENT_TYPE, METHOD) }) // Add Orion Sink OrionSink.addSink( processedDataStream ) // … } 9
  10. 10. Examples https://github.com/ging/fiware-cosmos-orion-flink-connector-examples https://fiware-cosmos-flink-examples.readthedocs.io ● Example 1: Simulated Orion Source Notification ● Example 2: Complete Orion Scenario with docker-compose ● Example 3: Packaging the code and submitting it to the Flink Job Manager ● Example 4: More complex operations (Flink AggregateFunction) ● Example 5: Structured values for attributes (objects) 10
  11. 11. def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment // Create Orion Source. Receive notifications on port 9001 val eventStream = env.addSource(new OrionSource(9001)) // Process event stream val processedDataStream = eventStream .flatMap(event => event.entities) .map(entity => { val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue() new Temp_Node( entity.id, temp) }) .keyBy("id") .timeWindow(Time.seconds(10)) .aggregate(new Average) // print the results with a single thread, rather than in parallel processedDataStream.print().setParallelism(1) env.execute("Socket Window NgsiEvent") } Demo: Average temperature for each entity 13
  12. 12. Demo: Maximum temperature overall def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment // Create Orion Source. Receive notifications on port 9001 val eventStream = env.addSource(new OrionSource(9001)) // Process event stream val processedDataStream = eventStream .flatMap(event => event.entities) .map(entity => { val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue() println("Entity",entity.id, "Temperature", temp) new Temp_Node( entity.id, temp) }) .timeWindowAll(Time.seconds(10) ) .max("temperature") // print the results with a single thread, rather than in parallel processedDataStream.print().setParallelism(1) env.execute("Socket Window NgsiEvent") } 15
  13. 13. Thank you! http://fiware.org Follow @FIWARE on Twitter 17

×