Modern race cars produce lot of data, and all this in real time. In this presentation I will show you how data could be generated and used by various applications in the car, on the track or team head quarter. The demonstration will show how to move data using messaging systems like Apache Kafka, process the data using Apache Spark and use various storage technics: Distributed File System, NoSQL Database. This presentation is a great opportunity to see how to build a " near real time big data application". The code from this talk will be made available as open source.

1. Fast Cars, Big Data
How Streaming Can Help Formula1 ?
Tugdual Grall - MapR - @tgrall
ROME 24-25 MARCH 2017

2. © 2017 MapR TechnologiesMapR Confidential 2
Agenda
• What’s the point of data in motorsports?
• Demo
• Architecture
• What’s next?
• Q&A

3. © 2017 MapR TechnologiesMapR Confidential 3
What’s the point of data in motorsports?

4. © 2017 MapR TechnologiesMapR Confidential 4

5. © 2017 MapR TechnologiesMapR Confidential 5
Data in Motorsports
F1 Framework - http://f1framework.blogspot.de/2013/08/short-guide-to-f1-telemetry-spa-circuit.html

6. © 2017 MapR TechnologiesMapR Confidential 6
Race Strategy
F1 Framework - http://f1framework.blogspot.be/2013/08/race-strategy-explained.html

7. © 2017 MapR TechnologiesMapR Confidential 7
Got examples?

8. © 2017 MapR TechnologiesMapR Confidential 8
Got examples?
• Up to 300 sensors per car

9. © 2017 MapR TechnologiesMapR Confidential 9
Got examples?
• Up to 300 sensors per car
• Up to 2000 channels

10. © 2017 MapR TechnologiesMapR Confidential 10
Got examples?
• Up to 300 sensors per car
• Up to 2000 channels
• Sensor data are sent to the paddock in 2ms

11. © 2017 MapR TechnologiesMapR Confidential 11
Got examples?
• Up to 300 sensors per car
• Up to 2000 channels
• Sensor data are sent to the paddock in 2ms
• 1.5 billions of data points for a race

12. © 2017 MapR TechnologiesMapR Confidential 12
Got examples?
• Up to 300 sensors per car
• Up to 2000 channels
• Sensor data are sent to the paddock in 2ms
• 1.5 billions of data points for a race
• 5 billions for a full race weekend

13. © 2017 MapR TechnologiesMapR Confidential 13
Got examples?
• Up to 300 sensors per car
• Up to 2000 channels
• Sensor data are sent to the paddock in 2ms
• 1.5 billions of data points for a race
• 5 billions for a full race weekend
• 5/6Gb of compressed data per car for 90mn

14. © 2017 MapR TechnologiesMapR Confidential 14
Got examples?
• Up to 300 sensors per car
• Up to 2000 channels
• Sensor data are sent to the paddock in 2ms
• 1.5 billions of data points for a race
• 5 billions for a full race weekend
• 5/6Gb of compressed data per car for 90mn
US Grand Prix 2014 : 243 Tb (race teams combined)

15. © 2017 MapR TechnologiesMapR Confidential 15
How does that work?

16. © 2017 MapR TechnologiesMapR Confidential 16
Production System Outline

17. © 2017 MapR TechnologiesMapR Confidential 17
Simplified Demo System Outline
Archive
MapR DB
Jetty /
Bootstrap /
d3
Apache Drill
(SQL access)
TORCS race
simulator
MapR
Streams

18. © 2017 MapR TechnologiesMapR Confidential 18
TORCS for Cars, Physics & Drivers
• The Open Racing Car Simulator
– http://torcs.sourceforge.net/
• Car racing game
• AI & Research Platform

19. © 2017 MapR TechnologiesMapR Confidential 19
Demo

20. © 2017 MapR TechnologiesMapR Confidential 20
Architecture
Events Producers Events Consumers
sensors data
Real Time
Analytics
Analytics with SQL
https://github.com/mapr-demos/racing-time-series

21. © 2017 MapR TechnologiesMapR Confidential 21
Architecture
Events Producers Events Consumers
sensors data
Real Time
Analytics
https://github.com/mapr-demos/racing-time-series
JSONProducer
Consumer

22. © 2017 MapR TechnologiesMapR Confidential 22
Where/How to store data ?

23. © 2017 MapR TechnologiesMapR Confidential 23
Big Datastore
Distributed File System
HDFS/MapR-FS
NoSQL Database
HBase/MapR-DB
….

24. © 2017 MapR TechnologiesMapR Confidential 24
Data Streaming

25. © 2017 MapR TechnologiesMapR Confidential 25
Data Streaming
Moving millions of events per h|mn|s

26. © 2017 MapR TechnologiesMapR Confidential 26
What is Kafka?
• http://kafka.apache.org/
• Created at LinkedIn, open sourced in 2011
• Implemented in Scala / Java
• Distributed messaging system built to scale

27. © 2017 MapR TechnologiesMapR Confidential 27
Big Picture
Producer
Producer
Producer
Consumer
Consumer
Consumer

28. © 2017 MapR TechnologiesMapR Confidential 28
Topic & Partitions
0 1 2 3 4 5 6 7 8
0 1 2 3 4 5
0 1 2 3 4 5 6 7
Partition 0
Partition 1
Partition 2
Writes

29. © 2017 MapR TechnologiesMapR Confidential 29
Consumer Groups

30. © 2017 MapR TechnologiesMapR Confidential 30
More real life Kafka …
Zookeeper
Broker 1
Topic A Topic B
Broker 2
Topic A Topic B
Broker 3
Topic A Topic B
Producer
Producer
Producer
Consumer
Consumer
Consumer

31. © 2017 MapR TechnologiesMapR Confidential 31
MapR Streams
• Distributed messaging system built to scale
• Use Apache Kafka API 0.9.0
• No code change
• Does not use the same “broker” architecture
– Log stored in MapR Storage (Scalable, Secured, Fast, Multi DC)
– No Zookeeper

32. © 2017 MapR TechnologiesMapR Confidential 32
Produce Messages
ProducerRecord<String, byte[]> rec = new ProducerRecord<>(
“/apps/racing/stream:sensors_data“,
eventName,
value.toString().getBytes());
producer.send(rec, (recordMetadata, e) -> {
if (e != null) { … });
producer.flush();

33. © 2017 MapR TechnologiesMapR Confidential 33
Consume Messages
long pollTimeOut = 800;
while(true) {
ConsumerRecords<String, String> records = consumer.poll(pollTimeOut);
if (!records.isEmpty()) {
Iterable<ConsumerRecord<String, String>> iterable = records::iterator;
StreamSupport
.stream(iterable.spliterator(), false)
.forEach((record) -> {
// work with record object
record.value();
…
});
consumer.commitAsync();
}
}

34. © 2017 MapR TechnologiesMapR Confidential 34
What’s next?

35. © 2017 MapR TechnologiesMapR Confidential 35
Sensor Data V1
• 3 main data points:
– Speed (m/s)
– RPM
– Distance (m)
• Buffered
{ "_id":"1.458141858E9/0.324",
"car" = "car1",
"timestamp":1458141858,
"racetime”:0.324,
"records":
[
{
"sensors":{
"Speed":3.588583,
"Distance":2003.023071,
"RPM":1896.575806
},
"racetime":0.324,
"timestamp":1458141858
},
{
"sensors":{
"Speed":6.755624,
"Distance":2004.084717,
"RPM":1673.264526
},
"racetime":0.556,
"timestamp":1458141858
},

36. © 2017 MapR TechnologiesMapR Confidential 36
Capture more data

37. © 2017 MapR TechnologiesMapR Confidential 37
Sensor Data V2
• 3 main data points:
– Speed (m/s)
– RPM
– Distance (m)
– Throttle
– Gears
– …
• Buffered
{ "_id":"1.458141858E9/0.324",
"car" = "car1",
"timestamp":1458141858,
"racetime”:0.324,
"records":
[
{
"sensors":{
"Speed":3.588583,
"Distance":2003.023071,
"RPM":1896.575806,
"Throttle" : 33,
"Gear" : 2
},
"racetime":0.324,
"timestamp":1458141858
},
{
"sensors":{
"Speed":6.755624,
"Distance":2004.084717,
“RPM”:1673.264526,
"Throttle" : 37,
"Gear" : 2
},
"racetime":0.556,
"timestamp":1458141858
},

38. © 2017 MapR TechnologiesMapR Confidential 38
Add new services

39. © 2017 MapR TechnologiesMapR Confidential 39
New Data Service
sensors data
Alerts
Stream
Processing

40. © 2017 MapR TechnologiesMapR Confidential 40
• Cluster Computing Platform
• Extends “MapReduce” with extensions
– Streaming
– Interactive Analytics
• Run in Memory
• http://spark.apache.org/

41. © 2017 MapR TechnologiesMapR Confidential 41
• Streaming Dataflow Engine
– Datastream/Dataset APIs
– CEP, Graph, ML
• Run in Memory
• https://flink.apache.org/

42. © 2017 MapR TechnologiesMapR Confidential 42
Demo
Stream Processing with Flink

43. © 2017 MapR TechnologiesMapR Confidential 43
Streaming Architecture & Formula 1
• Stream & Process data in real time
– Use distributed & scalable processing and storage
– NoSQL Database, Distributed File System
• Decouple the source from the consumer(s)
– Dashboard, Analytics, Machine Learning
– Add new use case….

44. © 2017 MapR TechnologiesMapR Confidential 44
Streaming Architecture & Formula 1
• Stream & Process data in real
time
– Use distributed & scalable
processing and storage
– NoSQL Database, Distributed
File System
• Decouple the source from the
consumer(s)
– Dashboard, Analytics, Machine
Learning
– Add new use case….
This is not only about Formula 1!
(Telco, Finance, Retail, Content, IT)

45. © 2017 MapR TechnologiesMapR Confidential 45
One Last Thing…
Events Producers Events Consumers
Web Socket
https://github.com/mapr-demos/racing-time-series
BLE
MapR-Streams
Apache Kafka

46. © 2017 MapR TechnologiesMapR Confidential 46