Download as PDF, PPTX
![Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 14
1A) CURRENT SOLUTIONS FOR CAR DATA ACCESS PROVIDED BY OEM
BMW Labs
[1] https://labs.bmw.com/
[2] https://developer.mercedes-benz.com/,
[3] http://www.porsche-next-oi-competition.com/
[4] https://developer.psa-peugeot-citroen.com/inc/
[5] https://www.w3.org/auto/wg/
Web services [1] APIs in development [2-4]
Standards in development
ISO 20078:
neutral server
ISO 22901: data
model for
diagnosis
VISS, VIAS [5]
VISS: Vehicle Information Server Spec (W3C)
VIAS: Vehicle Information API Spec (W3C)
“Gen 2”
In the current charter
V2X communication
Competing standards:
- 5G (cellular network)
- ITS G5 (peer-to-peer)
Interconnected fleets will rely on them](https://image.slidesharecdn.com/defense-benjamin-klotz-2019-releaseversion-191003154407/75/Semantic-Technologies-for-Vehicle-Data-Defense-14-2048.jpg)
![Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 27
2A) CONTRIBUTION: DRIVING CONTEXT MODELING PATTERN
Hazard
Ontologies [3]
Mental state
Ontologies [2]
Traffic
Ontologies [7]
Weather
Ontologies [6]
Places
Ontologies [5]
Road
Ontologies [4]
Driving context
Modeling pattern
Spatiotemporal
Events
and States
[1] http://automotive.eurecom.fr/vsso
[2] https://bioportal.bioontology.org/ontologies/MFOEM
[3] https://transportdisruption.github.io/
[4] http://ci.emse.fr/opensensingcity/ns/sca/vocabulary_81/
[5] http://mapserv.kt.agh.edu.pl/ontologies/osm.owl
[6] https://ci.mines-stetienne.fr/seas/WeatherOntology
[7] http://vocab.datex.org/terms/
VSSo
(=SSN/SOSA+VSS)
~ 300 signals
~ 50 attributes
~ 80 branches
Car signal
Ontologies [1]](https://image.slidesharecdn.com/defense-benjamin-klotz-2019-releaseversion-191003154407/75/Semantic-Technologies-for-Vehicle-Data-Defense-27-2048.jpg)
![2B-2C)DATA INTEGRATION AND ROUTE SELECTION: POI RECOMMENDATIONS
Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 38
Recommendation of places to stop and rest on
long trips (http://drivescover.eurecom.fr)
According to driving features:
- Traffic
- Weather
- (opening) time
- Ratings
POI score:
score = [WeightClass *Class]*hoursOpen
*(Weather + Rating + Traffic)
Evaluation: contextual driving features enable
driver recommendations
- 18 journeys (35 stops) by giving on a poll by 18
users: rank the top between 10-20 POIs
- Average P@4=31,5%, R@4=14,1%
Klotz, B., Lisena, P., Troncy, R., Wilms, D., & Bonnet, C. (2017, October). DriveSCOVER: A Tourism
Recommender System Based on External Driving Factors. In International Semantic Web](https://image.slidesharecdn.com/defense-benjamin-klotz-2019-releaseversion-191003154407/75/Semantic-Technologies-for-Vehicle-Data-Defense-38-2048.jpg)
![The fastest route is safe A safer alternative is better
Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 40
2B-2C) CONTEXT-BASED ROUTE RECOMMENDER: IMPLEMENTATION
Patterns
- Semantic Trajectories: DatAcron
- Driving Context: VDC,
- Risk: Hazard pattern (VoCamp)
Inferences
- Triple matching and
generation
- SPARQL Construct
Best route=trade-off between
- Routing time T
- Normalized risk factor R
Minimize f(T,R)
Experiment: show usability and explainability of an ontology-based route recommender
2 possible outputs
Provide a list of
differentiating
risk factors
Evaluation:
- Test different functions f to have about as many of both outputs
- F(T,R)=TR² is a working heuristic
- Test different time and driver profiles with fixed origin and destination
- Check output information validity in the semantic trajectory
Heuristics/parameters
Experience threshold: 10 years (half the average license age)
Snow/rain threshold: 4cm (rule of thumb on wheel sinkage)
Visibility maximum: 10km (METAR scale upper limit)
Night/day factor: 0.8 (arbitrary heuristics)
Road type risks: 0.1 to 1 (arbitrary heuristics) from average
stress impact [1]
[1] Schneegass, S., Pfleging, B., Broy, N., Heinrich, F., & Schmidt, A. (2013, October). A data set of
real world driving to assess driver workload. In Proceedings of the 5th international conference on
automotive user interfaces and interactive vehicular applications (pp. 150-157). ACM.](https://image.slidesharecdn.com/defense-benjamin-klotz-2019-releaseversion-191003154407/75/Semantic-Technologies-for-Vehicle-Data-Defense-40-2048.jpg)
![AUTOMOTIVE WEB THINGS: CHALLENGES
Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 46
Domain vs Nature of
−Things
−Interactions
Complexity of vehicles
−Different access control and
security
−Different expertise
"@id": "property/acceleration“,
"@type": ["Property","vsso:LongitudinalAcceleration","iot:Property"],
Data access
−External hardware
−Legacy solutions](https://image.slidesharecdn.com/defense-benjamin-klotz-2019-releaseversion-191003154407/75/Semantic-Technologies-for-Vehicle-Data-Defense-46-2048.jpg)
More Related Content
Similar to Semantic Technologies for Vehicle Data - Defense
Semantic Technologies for Vehicle Data - Defense
- 1. SEMANTIC TECHNOLOGIES FORVEHICLE DATA HOW CAN AUTOMOTIVE SERVICES BENEFIT FROM SEMANTICS IN THEIR DATA MODEL? Benjamin Klotz klotz@eurecom.fr Academic supervisors: Christian Bonnet, Raphael Troncy Industry supervisors: Daniel Wilms, Martin Arend, Michael Würtenberger
- 3. SEMANTIC TECHNOLOGIES FORVEHICLE DATA Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 3 Freedom? Driving experience? Risk prediction in cross- domain environments Trajectory data mining DATAStandard Access Understanding Interoperability
- 4. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 4 PLAN 01 02 03 04 Motivation Complex vehicle modeling Automatic contextualization Interactions in the Web of Things 05 Conclusion and future perspectives
- 5. MOTIVATION Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 5
- 6. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 6 DATA AROUND THE AUTOMOTIVE DOMAIN Maps POIs: “Home”, “Work” Personal information Contacts information Weather dataTraffic data Accident reports Navigation
- 7. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 7 SENSOR DATA IN THE AUTOMOTIVE DOMAIN Front camera Radar Tire pressure sensor Park assistant Steering angle sensor Wheel speed sensor Blind spot detection Adaptive cruise control Temperature sensor Oil temperature sensor Vehicle height sensor {"name":"accelerator_pedal_position","value":0,"timestamp":1361454211.483000} {"name":"fuel_level","value":23.478279,"timestamp":1361454211.485000} {"name":"torque_at_transmission","value":1,"timestamp":1361454211.488000} {"acceleratorPedal":{"position":"4095","ecoPosition":"3"},"brakeContact":"16","sp eedActual":"0“}, "timeStamp":"2018-01-10T17:01:27.297Z",} Signal name? Units? Datetime?
- 8. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 8 INTEROPERABILITY IN A FRAGMENTED IOT ECOSYSTEM Auto WG Technology providers
- 9. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 9 SCOPE OF THIS WORK Semantic Technologies - Data access - Data integration - Data modeling (Knowledge engineering) - Data processing for inferences - Interoperability - Standardization - Security - Privacy - … “OEM wantto interactwith cars,understand theircontext, and have carsinteracting with otherthings.” Main topics Out of scope in this thesis Manifold Air Pressure map
- 10. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 10 Interactions Prediction Trajectory Data model Knowledge generation/extraction Access Data model OVERVIEW VSS: Vehicle Signal Specification (GENIVI) VISS: Vehicle Information Server Spec (W3C) VIAS: Vehicle Information API Spec (W3C) Dataaccess API,carserver Enrichment, annotation Applications and analytics Motivating examples Data abstraction Expert- fixed rules RQ1: How can we provide an interoperable access and description of vehicle data ? RQ2: How can we learn human-understandable concepts from sensor data ? RQ3: How can we enable the integration of vehicle-generated data with external dataset using ontologies and schemata ? VISS, VIAS VSS ontology Signal quality Driving context ontology Machine Learning models WoT concepts & interactions Context-based recommendation Aggressive driving Maneuvers1a 1b 1c 2a 2b 2c 3
- 11. 02 COMPLEX VEHICLE MODELING SemanticTechnologies for Vehicle Data | Benjamin Klotz | 24/09/2019 11
- 12. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 12 Access Data model OVERVIEW VSS: Vehicle Signal Specification (GENIVI) VISS: Vehicle Information Server Spec (W3C) VIAS: Vehicle Information API Spec (W3C) Dataaccess API,carserver Data abstraction RQ1: How can we provide an interoperable access and description of vehicle data ? VISS, VIAS VSS ontology Signal quality 1a 1b 1c
- 13. REQUIREMENTS: COMPETENCY QUESTIONS SemanticTechnologies for Vehicle Data | Benjamin Klotz | 24/09/2019 13 Get information about attributes and signals on connected vehicles Complete list: https://github.com/klotzbenjamin/vss-ontology Telematics Garage/diagnosis Seamless experience 32 competency questions… Attributes What type of fuel does this car need? What is the model of this car? How old is this car? What type of transmission does this car have? Signals and sensors Is there a signal measuring the steering wheel angle? How many different speedometers does this car contain? Dynamic signals What is the current gear? What is the local temperature on the driver side? E-commerce … generated from domain needs on vehicle signals and attributes
- 14. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 14 1A) CURRENT SOLUTIONS FOR CAR DATA ACCESS PROVIDED BY OEM BMW Labs [1] https://labs.bmw.com/ [2] https://developer.mercedes-benz.com/, [3] http://www.porsche-next-oi-competition.com/ [4] https://developer.psa-peugeot-citroen.com/inc/ [5] https://www.w3.org/auto/wg/ Web services [1] APIs in development [2-4] Standards in development ISO 20078: neutral server ISO 22901: data model for diagnosis VISS, VIAS [5] VISS: Vehicle Information Server Spec (W3C) VIAS: Vehicle Information API Spec (W3C) “Gen 2” In the current charter V2X communication Competing standards: - 5G (cellular network) - ITS G5 (peer-to-peer) Interconnected fleets will rely on them
- 15. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 15 1B) STATE OF THE ART: VEHICLE SIGNAL SPECIFICATION (VSS) Figure: −451 branches −1103 leaves: − 43 attributes − 1060 signals: including − (700 seat-related), − 268 with unitSignal/Attribute Body Weight Raindetection Intensity Type: UInt8 Unit: percent Description: “…” Value: restriction or free ADAS Cabin Chassis Drivetrain OBD Vehicle Attribute Signal Signal entries Examples: Gearbox-sensed speed: .Drivetrain.Transmission.Speed Engine speed: .Drivetrain.Engine.Speed GPS-sensed speed: .Cabin.Infotainment.Speed Left door lock: .Body.Row1.Door.Left.IsLocked Right mirror tilt: .Cabin.Mirror.Right.Tilt https://github.com/GENIVI/vehicle_signal_specification
- 16. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 16 1B) STATE OF THE ART: SEMANTIC TECHNOLOGIES FOR CONNECTED THINGS Connected things - Have sensors/actuators - Produce/consume signals - Have communication means Early attempts : SensorML1, XML description of sensors and measurement processes, 2007-2014 Semantic Sensor Network2 (SSN), expressive representation of sensors and observations, 2004 Limitations : SSN was never really accepted in the industry (big, dependent on DUL…), complex standardization… 1: http://www.opengeospatial.org/standards/sensorml 2: http://purl.oclc.org/NET/ssnx/ssn The Sensor, Observation, Sample, and Actuator (SOSA) ontology was released in 2017, it is: - Lightweight - Self-contained - Domain-independent - Simple central pattern for SSN https://www.w3.org/TR/vocab-ssn/
- 17. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 17 1B) HYPOTHESIS: SOSA PATTERN FOR VEHICLE SIGNALS sosa:ObservableProperty sosa:Sensor sosa:Observationsosa:isObservedBy rdf:type sosa:Result 22.2 km/h^^cdt:speed 22.2^^xsd:double qudt:KilometerPerHour qudt:numericValuequdt:unit sosa:phenomenonTime geo:lat :Speed - Definition of a signal - Definition of a sensor - Formally-defined units - Geolocation No formal definition of: - “speed” or other observable properties - “speedometer” or other car sensors/actuators - “Vehicle” or vehicle parts BUT "2018-04-18T13:36:12Z"^^xsd:dateTime geo:lon sosa:hasSimpleResult 43.614386 7.071125
- 18. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 18 1B) VSSO DEVELOPMENT VSS Add sensors and actuators Reuse design patterns - SSN/SOSA - QUDT (unit) Fixing problems Generate definition of VSS concepts Manually validate and clean the generated ontology VSSontology (VSSo) Fixing problems 1. VSS concepts have unique names 2. All signals are either observable, actuatable or both 3. Different signals can yield the same phenomenon (e.g. speed) 4. All branches are part of the top “vsso:Vehicle” branch 5. All position-dependent branches have a property “position” Benjamin Klotz, Raphael Troncy, Daniel Wilms, and Christian Bonnet. VSSo: A Vehicle Signal and Attribute Ontology. 9th International Semantic Sensor Networks Workshop (SSN), Monterey, California, October 2018.
- 19. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 19 1B) VSSO EXAMPLE sosa:Observable Property sosa:Sensor sosa:Observation sosa:isObservedBy :Speed rdf:type sosa:Result 22.2 km/h^^cdt:speed 22.2^^xsd:double qudt:Unit qudt:unit sosa:hasSimple Result vsso:Speedometer vsso:Speed vsso:ObservableSignal rdfs:subClassOf rdfs:subClassOf rdfs:subClassOf vsso:Transmission vsso:Drivetrain vsso:Internal Combustion vss:partOf vss:partOf vss:hasSignal vsso:Branch rdfs:subClassOf 4 Nm^^cdt:torque vss:maxTorque rdf:type https://ci.mines-stetienne.fr/lindt/ Maxime Lefrançois, Antoine Zimmermann Supporting Arbitrary Custom Datatypes in RDF and SPARQL, In Proc. Extended Semantic Web Conference, ESWC, Heraklion, Greece, 2016
- 20. 1B) VSSO CONTRIBUTIONAND EVALUATION Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 20 VSSo: a Vehicle Signal and Attribute ontology (http://automotive.eurecom.fr/vsso) −OWL ontology of DL expressivity: ALUHOI+ −483 classes (~300 signals); 63 properties (~50 attributes) −Reuse SSN/SOSA modeling patterns Evaluation: Hypothesis: VSSo data enables SPARQL queries answering the set of competency questions Dataset: simulated (random) values for 19 signals and 23 fixed attributes on a sliding window of 3 seconds Experiment: set 2 SPARQL endpoints with VSSo data (with 1 vehicle, with a fleet of 3 vehicles) http://automotive.eurecom.fr/simulator/query http://automotive.eurecom.fr/simulator/fleetquery
- 21. 1B) EVALUATION Semantic Technologiesfor Vehicle Data | Benjamin Klotz | 24/09/2019 21 VSSo expressivity: most requirements can be translated into SPARQL queries What are the dimension of this car? SELECT ?length ?width ?height WHERE { ?branch vsso:length ?length; vsso:width ?width; vsso:height ?height.} SELECT DISTINCT ?localTemperature ?value ?position ?time WHERE { ?wheel a vsso:SteeringWheel; vsso:steeringWheelSide ?steeringWheelSide. ?branch a vsso:LocalHVAC; vsso:position ?position; vsso:hasSignal ?localTemperature. ?localTemperature a vsso:LocalTemperature. FILTER regex(str(?steeringWheelSide),str(?position)) ?obs a sosa:Observation; sosa:observedProperty ?localTemperature; sosa:hasSimpleResult ?value; sosa:phenomenonTime ?time. } ORDER BY DESC(?time) LIMIT 1 What is the current temperature on the driver side? 90% of competency questions can be answered http://automotive.eurecom.fr/simulator/query http://automotive.eurecom.fr/simulator/fleetquery
- 22. 1B) VSS2 Semantic Technologiesfor Vehicle Data | Benjamin Klotz | 24/09/2019 22 Attribute Branch Vehicle Branch VehicleIdentification Branch VIN Attribute Body Branch Drivetrain Branch Signal Branch Body Branch Drivetrain Branch Vehicle Branch AverageSpeed Signal Private Branch BMW Branch PrivateSignal ?? HMI Branch Vehicle Body ADAS Cabin Chassis Drivetrain OBD Private branches and leaves should: - Overwrite pre-existing concepts - Extend the VSS tree VSS needs consistent position patterns VSS1 | VSS2
- 23. Vehicle sensor data Sampling Contract TransmissionMarketplace Privacy 1C) REMAINING COMPONENTS SemanticTechnologies for Vehicle Data | Benjamin Klotz | 24/09/2019 23 https://www.w3.org/community/autowebplatform/wiki/Data_tf https://at.projects.genivi.org/wiki/pages/viewpage.action?pageId=34963516 Degradation Retention Compression Quality Business considerations Consent Policies Sensitivity Access control Signal quality: many components Work items at the W3C Automotive Data Task Force
- 24. 03 AUTOMATIC CONTEXTUALIZATION OFDRIVING DATA Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 24
- 25. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 25 Data model Knowledge generation/extraction Access Data model OVERVIEW VSS: Vehicle Signal Specification (GENIVI) VISS: Vehicle Information Server Spec (W3C) VIAS: Vehicle Information API Spec (W3C) Dataaccess API,carserver Enrichment, annotation Data abstraction RQ2: How can we learn human-understandable concepts from sensor data ? VISS, VIAS VSS ontology Signal quality Driving context ontology 1a 1b 1c 2a
- 26. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 26 REQUIREMENTS FOR A DRIVING CONTEXT MODEL - Sub-domains clustered based on their common models and output of classifiers. - Centered around drivers, passengers, vehicles and roads. 28 competency questions Domain states What is the weather at this location/time? What is the mental state of this passenger? What is the level of fluidity on this road? Domain events What is this hazard? Who is involved in it? Cross-domain states Am I currently overspeeding? What is the most common mental state of the driver on the current trajectory? Cross-domain events What the cause of congestion? How many tti:TurnLeft maneuvers have the driver done the last hour? Driving context domains Complete list: https://github.com/klotzbenjamin/vdc-ontology
- 27. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 27 2A) CONTRIBUTION: DRIVING CONTEXT MODELING PATTERN Hazard Ontologies [3] Mental state Ontologies [2] Traffic Ontologies [7] Weather Ontologies [6] Places Ontologies [5] Road Ontologies [4] Driving context Modeling pattern Spatiotemporal Events and States [1] http://automotive.eurecom.fr/vsso [2] https://bioportal.bioontology.org/ontologies/MFOEM [3] https://transportdisruption.github.io/ [4] http://ci.emse.fr/opensensingcity/ns/sca/vocabulary_81/ [5] http://mapserv.kt.agh.edu.pl/ontologies/osm.owl [6] https://ci.mines-stetienne.fr/seas/WeatherOntology [7] http://vocab.datex.org/terms/ VSSo (=SSN/SOSA+VSS) ~ 300 signals ~ 50 attributes ~ 80 branches Car signal Ontologies [1]
- 28. 2A) MODELING CHOICES:STATES AND EVENTS Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 28 Alignment with domain ontologies/vocabularies owl:equivalentClass On states of features of interest States Events Benjamin Klotz, Raphael Troncy, Daniel Wilms, and Christian Bonnet. VSSo – A vehicle signal and attribute ontology for the Web of Things. Semantic Web journal, 2019.
- 29. 2A) DRIVING CONTEXTCONTRIBUTION AND EVALUATION Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 29 VDC: a Vehicle Driving Context ontology (http://automotive.eurecom.fr/vdc) −OWL ontology of DL expressivity: ALC −22 classes and 3 object properties −Re-use SSN/SOSA modeling patterns and the event ontology Evaluation: Hypothesis: VDC data enables SPARQL queries answering the set of competency questions Datasets converted from: − Collected from a BMW research car: 3 minutes, 3k observations on 10 signals, 103 maneuvers instances − Public dataset of stress, mental load linked to car signals, maneuvers and road types: 45 minutes, 13k observations on 6 states, 31 maneuvers Experiment: set up a SPARQL endpoint with VDC data Benjamin Klotz, Raphaël Troncy, Daniel Wilms, and Christian Bonnet. A driving context ontology for making sense of cross-domain driving data. In BMW Summer school, Raitenhaslach, Germany, 2018 Schneegass, S., Pfleging, B., Broy, N., Heinrich, F., & Schmidt, A. (2013, October). A data set of real world driving to assess driver workload. In Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications (pp. 150-157). ACM.
- 30. 2A) EVALUATION Semantic Technologiesfor Vehicle Data | Benjamin Klotz | 24/09/2019 30 VDC expressivity: most requirements can be translated into SPARQL queries Am I overspeeding? 86% of competency questions can be answered SELECT ?overspeeding WHERE { ?segment a datacron:TrajectoryPart. FILTER NOT EXISTS { ?segment datacron:directlyPrecedes ?nextSegment } signal a tti:SpeedLimit. ?observation a sosa:Observation; sosa:hasObservedProperty ?signal; sosa:hasSimpleResult ?speedLimit; geo:location ?segment. SELECT ?speed WHERE { ?signal a vsso:VehicleSpeed. ?observation a sosa:Observation; sosa:hasObservedProperty ?signal; sosa:hasSimpleResult ?speed; sosa:hasPhenomenonTime ?time. } ORDER BY DESC (?time) LIMIT 1 BIND( IF(?speed>?speedLimit,"yes","no") as ?overspeeding ) } Benjamin Klotz, Raphaël Troncy, Daniel Wilms, and Christian Bonnet. A driving context ontology for making sense of cross-domain driving data. In BMW Summer school, Raitenhaslach, Germany, 2018 SELECT ?flow WHERE { ?signal a datex:TrafficFlow. ?observation a sosa:Observation; sosa:hasObservedProperty ?signal; sosa:hasSimpleResult ?flow; sosa:hasPhenomenonTime ?time. }ORDER BY DESC (?time) LIMIT 1 What is the local traffic flow?
- 31. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 31 Prediction Trajectory Data model Knowledge generation/extraction Access Data model OVERVIEW VSS: Vehicle Signal Specification (GENIVI) VISS: Vehicle Information Server Spec (W3C) VIAS: Vehicle Information API Spec (W3C) Dataaccess API,carserver Enrichment, annotation Applications and analytics Motivating examples Data abstraction Expert- fixed rules RQ2: How can we learn human-understandable concepts from sensor data ? VISS, VIAS VSS ontology Signal quality Driving context ontology Machine Learning models Context-based recommendation Aggressive driving Maneuvers1a 1b 1c 2a 2b 2c
- 32. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 32 STATE OF THE ART: PREDICTION FROM VEHICLE DATA - Aggressiveness - Drowsiness - Driving style - Diagnosis - Topology - Marks - Potholes - Obstacles - Weather - Emotions - Stress - Mental load - Frustration - Distraction - Maneuvers - Intents Vehicle machine learning In-car learning Behavior Mental State Environment Trajectory patterns Fleet learning Data sources: - Car sensors - Smartphones/cameras - Physiological sensors
- 33. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 33 2B) AGGRESSIVE MANEUVER CLASSIFIER Protocol: - A driver told to drive safely on roads of to do dangerous maneuvers on a track - A copilot is using a car application to annotate the recorded data - Sequence: 10 measurements - Split 70/30 AUCRF=0.99 AUCRNN=0.998 Evaluation Dataset - ~13 hours of recorded data - ~3,5 hours of aggressive driving - 183 maneuvers - Sampling period of 0,5s Daniel Alvarez Coello, Benjamin Klotz, Daniel Wilms, Jorge Marx Gómez, and Raphaël Troncy. Modeling dangerous driving events based on in-vehicle data using Random Forest and Recurrent Neural Network. In 1st International Workshop on Data Driven Intelligent Vehicle Applications (DDIVA), Paris, France, 2019
- 34. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 34 2B) RECONSTRUCTION OF DANGEROUS SITUATIONS Important difference between safe and aggressive Track shape and speed far from public roads 2 biases Daniel Alvarez Coello, Benjamin Klotz, Daniel Wilms, Jorge Marx Gómez, and Raphaël Troncy. Modeling dangerous driving events based on in-vehicle data using Random Forest and Recurrent Neural Network. In 1st International Workshop on Data Driven Intelligent Vehicle Applications (DDIVA), Paris, France, 2019
- 35. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 35 2C) STATE OF THE ART: MOBILITY DATA MINING Spatiotemporal trajectories Semantic trajectories Main patterns Places/Regions Connectivity Mining Movement prediction CharacterizationOutlier detection Place recommendation Jean Damascene Mazimpaka and Sabine Timpf. Trajectory data mining-a review of methods and applications. Journal of Spatial Information Science, 2016.
- 36. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 36 2C) CONTRIBUTION SEMANTIC TRAJECTORIES GENERATION http://automotive.eurecom.fr/trajectory Benjamin Klotz, Raphael Troncy, Daniel Wilms, and Christian Bonnet. Generating semantic trajectories using a car signal ontology. In The Web Conference (WebConf'18), Demo Track, April 23-27, 2018, Lyon, France. Rule example acceleration/deceleration/ constant speed with a +/- 5km/h between observations Comparative table of ontology combination from the state of the art Demonstration: threshold-based Horn rules Hypothesis: we can enable semantic trajectory enrichments with signal values for generic applications
- 37. 2B-2C) COMBINING DATAINTEGRATION AND INFERENCES Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 Page 37 p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 Aggressive behavior Neutral Safe behavior Traffic jam Close event Closed road Generalization of semantic trajectories with a driving context Statistical learning from vehicle data Data integration from external sources Rule-based inferences Spatiotemporal traces Contextually-enriched trajectories Example: mental load & maneuver events Benjamin Klotz, Raphaël Troncy, Daniel Wilms, and Christian Bonnet. A driving context ontology for making sense of cross-domain driving data. In BMW Summer school, Raitenhaslach, Germany, 2018
- 38. 2B-2C)DATA INTEGRATION ANDROUTE SELECTION: POI RECOMMENDATIONS Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 38 Recommendation of places to stop and rest on long trips (http://drivescover.eurecom.fr) According to driving features: - Traffic - Weather - (opening) time - Ratings POI score: score = [WeightClass *Class]*hoursOpen *(Weather + Rating + Traffic) Evaluation: contextual driving features enable driver recommendations - 18 journeys (35 stops) by giving on a poll by 18 users: rank the top between 10-20 POIs - Average P@4=31,5%, R@4=14,1% Klotz, B., Lisena, P., Troncy, R., Wilms, D., & Bonnet, C. (2017, October). DriveSCOVER: A Tourism Recommender System Based on External Driving Factors. In International Semantic Web
- 39. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 39 2B-2C) CONTEXT-BASED ROUTE RECOMMENDER: DEMONSTRATION Scope: from scattered data to risk-centric semantic trajectories − Vehicle data and its inferences (location, destination, aggressiveness) combined with contextual data (routing, traffic, weather) − Semantic intended trajectories, include a quantified risk and query it to find the “best route” (safe and short) Assumption: quantifying risky driving features − “Expert views of police officers, lay views of the driving public, and official road accident records” to quantify the contribution of contextual features to driving danger (percentage of those features reported in accidents) Variables: age, gender, experience, aggressiveness, fatigue, time, locations, road information Current position Destination Jonathan J. Rolison, Shirley Regev, Salissou Moutari, Aidan Feeney, What are the factors that contribute to road accidents? An assessment of law enforcement views, ordinary drivers’ opinions, and road accident records, Accident Analysis & Prevention. Volume 115. 2018 Vehicle data + demographics Best N possible routes External context: weather, traffic Driving danger features Best route + explanation origin
- 40. The fastest routeis safe A safer alternative is better Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 40 2B-2C) CONTEXT-BASED ROUTE RECOMMENDER: IMPLEMENTATION Patterns - Semantic Trajectories: DatAcron - Driving Context: VDC, - Risk: Hazard pattern (VoCamp) Inferences - Triple matching and generation - SPARQL Construct Best route=trade-off between - Routing time T - Normalized risk factor R Minimize f(T,R) Experiment: show usability and explainability of an ontology-based route recommender 2 possible outputs Provide a list of differentiating risk factors Evaluation: - Test different functions f to have about as many of both outputs - F(T,R)=TR² is a working heuristic - Test different time and driver profiles with fixed origin and destination - Check output information validity in the semantic trajectory Heuristics/parameters Experience threshold: 10 years (half the average license age) Snow/rain threshold: 4cm (rule of thumb on wheel sinkage) Visibility maximum: 10km (METAR scale upper limit) Night/day factor: 0.8 (arbitrary heuristics) Road type risks: 0.1 to 1 (arbitrary heuristics) from average stress impact [1] [1] Schneegass, S., Pfleging, B., Broy, N., Heinrich, F., & Schmidt, A. (2013, October). A data set of real world driving to assess driver workload. In Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications (pp. 150-157). ACM.
- 41. 04 INTERACTIONS IN THEWEB OF THINGS Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 41
- 42. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 42 Interactions Prediction Trajectory Data model Knowledge generation/extraction Access Data model OVERVIEW VSS: Vehicle Signal Specification (GENIVI) VISS: Vehicle Information Server Spec (W3C) VIAS: Vehicle Information API Spec (W3C) Dataaccess API,carserver Enrichment, annotation Applications and analytics Motivating examples Data abstraction Expert- fixed rules RQ3: How can we enable the integration of vehicle-generated data with external dataset using ontologies and schemata ? VISS, VIAS VSS ontology Signal quality Driving context ontology Machine Learning models WoT concepts & interactions Context-based recommendation Aggressive driving Maneuvers1a 1b 1c 2a 2b 2c 3
- 43. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 W3C WoT servient 43 STATE OF THE ART: SEMANTIC TECHNOLOGIES FOR DATA INTEGRATION Limited vehicle sensor/actuator access SSN/SOSA Domain ontology AutoWG Web Devices IoT platforms
- 44. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 44 STATE OF THE ART: WEB OF THINGS ContextDevice Features of Interest (domain- specific metadata) Thing (Car) Capabilities (HVAC) Interactions (Action heater) Data (data schema) Amelie Gyrard. Designing cross-domain semantic Web of things applications. PhD thesis, EURECOM, 2015
- 45. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 45 STATE OF THE ART: WOT ONTOLOGY −Define a wot:Thing −Centered on wot:interactionPattern − Properties − Actions − Events −Use dataSchema − Literal value − wot:DataType − om:Unit_of_measure http://iot.linkeddata.es/def/wot/index-en.html
- 46. AUTOMOTIVE WEB THINGS:CHALLENGES Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 46 Domain vs Nature of −Things −Interactions Complexity of vehicles −Different access control and security −Different expertise "@id": "property/acceleration“, "@type": ["Property","vsso:LongitudinalAcceleration","iot:Property"], Data access −External hardware −Legacy solutions
- 47. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 47 3) CONTRIBUTION: COMPLEX THINGS IN THE WOT Differing Safety, Security and Privacy ? TD: full vehicle @context securityDefinitions Interactions: 1. property/media- volume 2. action/stop-HVAC 3. event/sow-engine-oil … 2548. action/write-message TD: main TD: safety-critical TD: HVAC TD: infotainment @context securityDefinitions Interactions: 1. property/media- volume … Interactions: 1. action/stop-HVAC … TD: Engine @type @type Interactions: 1. event/sow-engine- oil … securityDefinitions TD: privacy-critical securityDefinitions Requirements included in the latest charter update (June 2019)
- 48. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 48 3) CONTRIBUTION: ALIGNMENT WOT – SOSA FOR THE AUTOMOTIVE DOMAIN Modeling pattern: i. Vehicles are Things ii. Signals are properties Read-write depending on the signal type iii. Actuatable signals are actions iv. DataSchema use the domain Units Benjamin Klotz, Soumya Kanti Datta, Daniel Wilms, Raphael Troncy, and Christian Bonnet. A car as a semantic web thing: Motivation and demonstration. In 2nd Global Internet of Things Summit (GIoTS'18), Bilbao, Spain, June 2018.
- 49. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 49 3) CONTRIBUTION: ALIGNMENT FOR EVENTS Event modeling pattern: i. Mapped to the Event Ontology ii. Wot:Event≡event:Event iii. The FOI (Vehicle) is an Agent Evaluation: - Hypothesis: a combination of WoT and VDC/VSSo enables cross- domain interactions with vehicle attributes, signals and events - Comparative table with state of the art survey Benjamin Klotz, Raphael Troncy, Daniel Wilms, and Christian Bonnet. VSSo – A vehicle signal and attribute ontology for the Web of Things. Semantic Web journal, 2019.
- 50. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 50 3) DEMONSTRATIONS (1): REAL CAR, DIAGNOSIS SIGNALS (2017) Usage: Behind a web browser, we can control the windows, doors and honk Benjamin Klotz. Binding the web of things with LwM2M for a vehicular use case. OpenDay of the W3C Web of Things Working Group, June 2017
- 51. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 51 3) DEMONSTRATIONS (2): TRACES AND APPLICATIONS Automatic applications of rules and interactions of web things: - If a door is open while speed>0 then trigger a warning - If longitudinal and lateral acceleration are high (dangerous driving), then turn on a red LED - If the coordinates of the car are close to a fixed destination, control a garage door an light Application servient Car servient (mocked up) Check rules Other servient Read properties Write properties https://www.youtube.com/watch?v=pjgTLPlAsKQ https://www.youtube.com/watch?v=zkL8Cdgy8PE
- 52. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 52 Legacy 3) DEMONSTRATION (3): INTUITIVE INTERFACE ON LEGACY VEHICLES Device servient Vehicle Web API TD Application servient HTTPS Flow Vehicle Node HTTPS Benjamin Klotz, Daniel Alvarez Coello, Daniel Wilms, and Raphaël Troncy. Abstracting and Interacting with Vehicles in the Web of Things. 2nd W3C Workshop on the Web of Things The Open Web to Challenge IoT Fragmentation, 2019
- 53. 05 CONCLUSION AND FUTUREPERSPECTIVES Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 53
- 54. Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 54 CONCLUSION RQ1: How can we provide an interoperable access and description of vehicle data ? RQ2: How can we learn human- understandable concepts from sensor data ? RQ3: How can we enable the integration of vehicle-generated data with external dataset using ontologies and schemata ? Data models - VSSo (http://automotive.eurecom.fr/vsso) - VDC (http://automotive.eurecom.fr/vdc) - Alignment VSSo/VDC/WoT (axioms to publish) Datasets: semantic trajectories - VSSo data: 5 trajectories, 8 signals, 16k observations (http://automotive.eurecom.fr/trajectory) - VDC data: 2 trajectories, 16 states, 16k observations, 130 events Classifiers (Internal BMW) - Aggressive driving prediction - Maneuvers prediction Demonstrations - DrIveSCOVER - Semantic trajectory annotation - WoT vehicles
- 55. REFLECTION Focus: usage andbenefits of vehicle data with semantic technologies − This research is a step in making connected vehicles more interoperable and understandable for application development Need for open standards − not necessarily for a full solution for connected vehicles Need to find the right level of abstraction from data itself −Limited computing resource embedded in vehicles, and the technical requirements to describe vehicles for the industry Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 55
- 56. REFLECTION Semantic Technologies forVehicle Data | Benjamin Klotz | 24/09/2019 56https://www.w3.org/auto/
- 57. FUTURE WORK: SHORTTERM Ongoing research optimizing the usage of VSSo/VDC − Connected Thesis on “Scalable reasoning on data of heterogenous sources and environments in an automotive context” started in 2018 at BMW by Daniel Alvarez. Experiment with development departments of OEMs the usage of VSSo and VDC ontology Possible work items benefiting from standardization: − W3C − Web of Things: complex things requirements description and best practices − Automotive Data task force: data contract, sampling, quality, marketplaces… − VSS2: specification to implement and new vehicle server to propose − GENIVI − Connected services community to join standardization efforts − OGC − Invitation to an OGC meeting to join efforts on Mobile Location Service (MLS) Domain Working Group (DWG) Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 57
- 58. LONG TERM PERSPECTIVES Researchgoals −Explore pattern alignments of best practices of signal metadata (marketplace, sampling, policies…) with VSSo −Assess the best practices for VDC (granularity, domain expert rules) and formulate recommendations for future VDC versions −Develop testbeds for semantic trajectories and optimize recommender systems Push the current work further to product development and global open standards. − Involve more parties in the standardization of semantic technologies for connected vehicles, including suppliers and data consumers, and provide the missing standard for formal vehicle signals and attributes based on VSSo − Push developer communities and OEMs to apply those standards in developed products at a larger scales Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 58
- 59. PUBLICATIONS AND CONTRIBUTIONSTO STANDARDS (1) Journals and book chapters (2) − Benjamin Klotz, Raphael Troncy, Daniel Wilms, and Christian Bonnet. VSSo – A vehicle signal and attribute ontology for the Web of Things. Semantic Web journal, 2019. − Mahda Noura, Amélie Gyrard, Benjamin Klotz, Raphaël Troncy and Soumya Kanti Datta. How to understand better "smart vehicle"? Knowledge Extraction for the Automotive Sector Using Web of Things, chapter SIoT 2020: Semantic IOT: Theory and Applications - Interoperability, Provenance and Beyond. Studies in Computational Intelligence. Springer, 2020. Submitted International Conferences and workshops (6) − Daniel Wilms, Benjamin Klotz and Adnan Bekan. Enabling innovation through open standards in the transportation domain. W3C Workshop on Transportation Data, 2019 − Daniel Alvarez Coello, Benjamin Klotz, Daniel Wilms, Jorge Marx Gómez, and Raphaël Troncy. Modeling dangerous driving events based on in-vehicle data using Random Forest and Recurrent Neural Network. In 1st International Workshop on Data Driven Intelligent Vehicle Applications (DDIVA), Paris, France, 2019 − Benjamin Klotz, Daniel Alvarez Coello, Daniel Wilms, and Raphaël Troncy. Abstracting and Interacting with Vehicles in the Web of Things. 2nd W3C Workshop on the Web of Things The Open Web to Challenge IoT Fragmentation, 2019 − Benjamin Klotz. Cross-domain interactions for connected cars, 2018. Keynote at the 8th International Conference of Consumer Electronics − Benjamin Klotz, Soumya Kanti Datta, Daniel Wilms, Raphael Troncy, and Christian Bonnet. A Car as a Semantic Web Thing: Motivation and Demonstration. In 2nd Global Internet of Things Summit (GIoTS), Bilbao, Spain, 2018 − Benjamin Klotz, Raphael Troncy, Daniel Wilms, and Christian Bonnet. VSSo – A vehicle signal and attribute ontology. In 9th International Semantic Sensor Networks Workshop (SSN), Monterey, USA, 2018 Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 59
- 60. PUBLICATIONS AND CONTRIBUTIONSTO STANDARDS (2) Demonstrations and Posters at international conferences (3) − Benjamin Klotz, Raphaël Troncy, Daniel Wilms, and Christian Bonnet. A driving context ontology for making sense of cross-domain driving data. In BMW Summer school, Raitenhaslach, Germany, 2018 − Benjamin Klotz, Raphaël Troncy, Daniel Wilms, and Christian Bonnet. Generating Semantic Trajectories Using a Car Signal Ontology. In The Web Conference (WWW), Demo Track, pages 135138, Lyon, France, 2018 − Benjamin Klotz, Pasquale Lisena, Raphaël Troncy, Daniel Wilms, and Christian Bonnet. DrIveSCOVER: A tourism recommender system based on external driving factors. In International Semantic Web Conference (Posters, Demos & Industry Tracks), Vienna, Austria, 2017 Contributions to Standards (7) − Benjamin Klotz. W3C Automotive WG Activities - Next generation specification and VSS. GENIVI All Member Meeting & W3C Automotive Working Group F2F Meeting, May 2019. Munich, Germany − Benjamin Klotz. Ontology alignment for automotive Web Things. Iotschema Community, November 2018. Online Call − Benjamin Klotz, Daniel Wilms, and Ulf Björkengren. VSS2.0 proposal. W3C Automotive Working Group at the TPAC, October 2018. Lyon, France − Benjamin Klotz. WoT and JSON-LD for Vehicle Data. W3C Automotive Working Group, October 2018. Online Call − Benjamin Klotz. VSSo: a car signal ontology - Extension of the Vehicle Signal Specification. GENIVI All Member Meeting & W3C Automotive Working Group F2F Meeting, April 2018. Munich, Germany − Benjamin Klotz. Semantic Technologies for Vehicle Data - VSS proposal. GENIVI Networking Expert Group, March 2018. Online call − Benjamin Klotz. Binding the web of things with LwM2M for a vehicular use case. OpenDay of the W3C Web of Things Working Group, June 2017 Semantic Technologies for Vehicle Data | Benjamin Klotz | 24/09/2019 60
- 61. QUESTIONS ? THANK YOUFOR YOUR ATTENTION 61 Benjamin Klotz klotz@eurecom.fr Academic supervisors: Christian Bonnet, Raphaël Troncy Industry supervisors: Daniel Wilms, Martin Arend, Michael Würtenberger