Presentation on the FIWARE Cosmos Generic Enabler: Performing Streaming Analytics using Flink and the Orion-Flink Connector. Presented at the FIWARE Global Summit Málaga 2018 by the UPM-ETSIT Team

1. FIWARE Orion Flink Connector
Sonsoles López (slopez@dit.upm.es)
Andres Muñoz (jamunoz@dit.upm.es)
Joaquin Salvachua (jsalvachua@dit.upm.es)
Gabriel Huecas (ghuecas@dit.upm.es)
Universidad Politécnica de Madrid
@jsalvachua, @FIWARE

2. Flink
Apache Flink is a framework and distributed processing engine for stateful
computations over unbounded and bounded data streams. Flink has been
designed to run in all common cluster environments, perform computations at
in-memory speed and at any scale.
2

3. Previous Architecture
3

4. Current Architecture
4

5. fiware-cosmos-orion-flink-connector
https://github.com/ging/fiware-cosmos-orion-flink-connector
https://fiware-cosmos-flink.readthedocs.io
Two parts:
● OrionSource: Receive notifications from the Context Broker
● OrionSink: Write data into the Context Broker
5

6. Connection
6
ORION
Context Broker
Flink Cluster
Flink Job (JAR)
orion-flink-connector
HTTP POST (Notification)
HTTP POST/PUT/PATCH
OrionSource
OrionSink

7. OrionSource
Receives data from the Orion Context Broker from a given port.
The received data is a Stream of NgsiEvent objects
val eventStream = env.addSource(new OrionSource(9001))
7

8. OrionSink
Sends data back to the Orion Context Broker
Takes a stream of OrionSinkObjects as a source:
● content: Message content in String format. If it is a JSON, it needs to be
stringified
● url: URL to which the message should be sent
● contentType: Type of HTTP content of the message (JSON, Plain)
● method: HTTP method of the message (POST, PUT, PATCH)
OrionSink.addSink( processedDataStream )
8

9. Basic Example
final val URL_CB = "http://flinkexample_orion_1:1026/v2/entities/"
final val CONTENT_TYPE = ContentType.JSON
final val METHOD = HTTPMethod.POST
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
new Temp_Node(entity.id, temp)
})
.keyBy("id")
.timeWindow(Time.seconds(5), Time.seconds(2))
.min("temperature")
.map(tempNode => {
val url = URL_CB + tempNode.id + "/attrs"
OrionSinkObject(tempNode.toString, url, CONTENT_TYPE, METHOD)
})
// Add Orion Sink
OrionSink.addSink( processedDataStream )
// …
}
9

10. Examples
https://github.com/ging/fiware-cosmos-orion-flink-connector-examples
https://fiware-cosmos-flink-examples.readthedocs.io
● Example 1: Simulated Orion Source Notification
● Example 2: Complete Orion Scenario with docker-compose
● Example 3: Packaging the code and submitting it to the Flink Job Manager
● Example 4: More complex operations (Flink AggregateFunction)
● Example 5: Structured values for attributes (objects)
10

11. def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
new Temp_Node( entity.id, temp)
})
.keyBy("id")
.timeWindow(Time.seconds(10))
.aggregate(new Average)
// print the results with a single thread, rather than in parallel
processedDataStream.print().setParallelism(1)
env.execute("Socket Window NgsiEvent")
}
Demo: Average temperature for each entity
13

12. Demo: Maximum temperature overall
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
println("Entity",entity.id, "Temperature", temp)
new Temp_Node( entity.id, temp)
})
.timeWindowAll(Time.seconds(10) )
.max("temperature")
// print the results with a single thread, rather than in parallel
processedDataStream.print().setParallelism(1)
env.execute("Socket Window NgsiEvent")
}
15

13. Thank you!
http://fiware.org
Follow @FIWARE on Twitter
17