Presentation by the UPM Team
Sonsoles López, Andres Muñoz, Joaquin Salvachua and Gabriel Huecas
FIWARE Global Summit
27-28 November 2018
Malaga, Spain
FIWARE Global Summit - Real-time Processing of Historic Context Information using Apache Flink
1. FIWARE Orion Flink Connector
Sonsoles López (slopez@dit.upm.es)
Andres Muñoz (jamunoz@dit.upm.es)
Joaquin Salvachua (jsalvachua@dit.upm.es)
Gabriel Huecas (ghuecas@dit.upm.es)
Universidad Politécnica de Madrid
@jsalvachua, @FIWARE
2. Flink
Apache Flink is a framework and distributed processing engine for stateful
computations over unbounded and bounded data streams. Flink has been
designed to run in all common cluster environments, perform computations at
in-memory speed and at any scale.
2
7. OrionSource
Receives data from the Orion Context Broker from a given port.
The received data is a Stream of NgsiEvent objects
val eventStream = env.addSource(new OrionSource(9001))
7
8. OrionSink
Sends data back to the Orion Context Broker
Takes a stream of OrionSinkObjects as a source:
● content: Message content in String format. If it is a JSON, it needs to be
stringified
● url: URL to which the message should be sent
● contentType: Type of HTTP content of the message (JSON, Plain)
● method: HTTP method of the message (POST, PUT, PATCH)
OrionSink.addSink( processedDataStream )
8
9. Basic Example
final val URL_CB = "http://flinkexample_orion_1:1026/v2/entities/"
final val CONTENT_TYPE = ContentType.JSON
final val METHOD = HTTPMethod.POST
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
new Temp_Node(entity.id, temp)
})
.keyBy("id")
.timeWindow(Time.seconds(5), Time.seconds(2))
.min("temperature")
.map(tempNode => {
val url = URL_CB + tempNode.id + "/attrs"
OrionSinkObject(tempNode.toString, url, CONTENT_TYPE, METHOD)
})
// Add Orion Sink
OrionSink.addSink( processedDataStream )
// …
}
9
11. def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
new Temp_Node( entity.id, temp)
})
.keyBy("id")
.timeWindow(Time.seconds(10))
.aggregate(new Average)
// print the results with a single thread, rather than in parallel
processedDataStream.print().setParallelism(1)
env.execute("Socket Window NgsiEvent")
}
Demo: Average temperature for each entity
13
12. Demo: Maximum temperature overall
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
println("Entity",entity.id, "Temperature", temp)
new Temp_Node( entity.id, temp)
})
.timeWindowAll(Time.seconds(10) )
.max("temperature")
// print the results with a single thread, rather than in parallel
processedDataStream.print().setParallelism(1)
env.execute("Socket Window NgsiEvent")
}
15