We’ve built a real-time streaming platform that enables prediction based on user behavior, with events occurring in virtual and augmented reality environments. The solution enables organizations to train people in an extended reality environment, where real-life training may be costly and dangerous. Kafka Streams enables analyzing spatial and event data to detect gestural feature and analyze user behavior in real-time to be able to predict any future mistake the user might make. Kafka is the backbone of our real-time analytics and extended reality communication platform with our cluster and applications being deployed on Kubernetes. In this talk, we will mainly focus on the following: 1. Why Extended Reality with Kafka is a step in the right direction. 2. Architecture & Power of Schema Registry in building a generic platform for pluggable XR apps and analytics models 3. How KSQL and Kafka Streams fits in Kafka Ecosystem to help analyze human motion data and detect features for real-time prediction. 4. Demo of a VR application with real-time analytics feedback, which assists people to be trained in how to work with chemical laboratory equipment.

1. Closing the Loop in Extended
Reality with Kafka Streams and
Machine Learning
Rob Drysdale | Senior Principal | robert.drysdale@accenture.com
Sarah Healy | Software Engineer | s.healy@accenture.com

2. The Dock is Accenture’s flagship
R&D and Global Innovation
Centre
300+ multi-disciplinary team
42% female
35+ nationalities
142 patents filed
Based in Silicon Docks, Dublin.
A hub for many global tech companies

3. • Extended reality in mainstream
• XR Insights : Usecase
• Building Generic Streaming Platform: Architectureand Schema
• Deep dive into our implementation
• Machine Learning 101 : Finding Patterns
• Learnings & Takeaways
• Demo
Agenda

4. XR Industry
• Exponential Growth
• Industry Adoption:
• Healthcare
• Aviation
• Education & Training
• Retail & Advertising
• Training: VR
• Safer Healthcare
• Cheaper
• Monitorable
• On-the-job: AR
• Future Worker
• Hands-free Manuals
• Remote Guidance
• Automated Guidance
• Prediction/Prevention

5. A Streaming platform with capability to
plug & play VR/AR devices and ML models.
XR Insights
Data Stream Capture:
• State Stream
• Event Stream
Data Analyses:
• Data-Driven User Assessment:
o Improving Accuracy of Assessment
o Removing Subjectivity of Assessment
• Insights on Performance and behaviour
• Personalise Training
• Improve Work/Training Process
• Predict/Guide User behaviour

6. TrainingData
WebSocket
Server
(STOMP)
/topic/feedback
XR Insights
/app/state
/app/event
Kafka
Producer
Kafka
Consumer
Kafka Stream Data Transformations
Kafka Connect
Master DB
BuildingML Model
Model & Features
S3 Bucket
Production ML Model
Personalization
Real Time Reporting
XR Devices

7. • Speed of Data Transmation
• Consumer Groups
• Easy Embedding of Kafka clients in existing servicedeployments
• Avro Schema support
• Clients for Java & Python
• Eventdriven & supportto query on real-time data
Why Kafka Streams?

8. Kafka
Producer
Kafka
Consumer
Schema Registry
XR Devices
Schema Registry: Single point of Success

9. • Kubernetes 1.13 & Docker (Statefulsets)
• Spring boot with WebSocket + STOMP
• Unity & C#
• Kafka 2.1.0, Kafka Streams DSL
• Schema Registry, Avro, ConnectCluster
• Couchbase, Grafana, Elastic search
Technologies

10. {
"type": "record",
"namespace": "com.accenture",
"name": "Event",
"version": "1",
"fields": [
{ "name": "eventId", "type": "string", "doc": "Unique Id per record" },
{ "name": "sessionId", "type":"string", "doc": "session Id per record" },
{ "name": "timestamp", "type": "float", "doc": "Timestamp of Event" },
{ "name": "type", "type": "string", "doc" : "Type of Event" },
{ "name": "name", "type": "string", "doc": "Name of Event" },
{ "name": "desc", "type": {"type": "map", "values": "string"} , "doc":
"description of event"}
]
}
{
"topic":"event-topic",
"key":"d433b1db-4f89-4128-896f-95ac204b8681",
"value":{
"eventId":"d433b1db-4f89-4128-896f-95ac204b8681",
"sessionId":"13937ec1-82e1-4071-a6e7-31d31674de25",
"timestamp":148.45999,
"type":"Error",
"name":"Drop",
"desc":{
"collided":"Floor",
"Task id":"0",
"point of contact":"(1.1 1.0 -1.1)",
"lhand":"NA",
"collider":"spray_bottle",
"rhand":"NA"
}
},
"partition":2,
"offset":10
}
Event Schema: Example

11. {
"type": "record",
"namespace": "com.accenture",
"name": "State",
"version": "1",
"fields": [
{ "name": "stateId", "type":["null", "string"], doc:”NA” },
{ "name": "sessionId", "type":["null", "string"], "doc": ”NA" },
{ "name": "timestamp", "type": "double", "doc": ”NA" },
{ "name": "task", "type": "long", "doc": ”NA" },
{ "name": "subtask", "type":["null", "string"], "doc": ”NA" },
{ "name": "lx", "type": "double", "doc": "NA" },
{ "name": "ly", "type": "double", "doc": "NA" },
{ "name": "lz", "type": "double", "doc": "NA" },
{ "name": "rx", "type": "double", "doc": "NA" },
{ "name": "ry", "type": "double", "doc": "NA" },
{ "name": "rz", "type": "double", "doc": "NA" },
{ "name": "hx", "type": "double", "doc": "NA" },
{ "name": "hy", "type": "double", "doc": "NA" },
{ "name": "hz", "type": "double", "doc": "NA" },
{ "name": "rItemHeld", "type":["null", "string"], "doc": "NA"}
{ "name": "lItemHeld", "type":["null", "string"], "doc": "NA"},
{ "name": "ltrigger", "type":["null", "boolean"], "doc": "NA"},
{ "name": "rtrigger", "type":["null", "boolean"], "doc": "NA"} ]
}
{
"topic":"state-topic",
"key":"13937ec1-82e1-4071-a6e7-31d31674de25",
"value":{
"stateId":{
"string":"cdfc5d94-3f1e-404a-ba27-632b3616dbfd"
},
"sessionId":{
"string":"13937ec1-82e1-4071-a6e7-31d31674de25"
},
"timestamp":126.98802185058594,
"task":0,
"subtask":{
"string":"0"
},
"lx":0.6099972724914551,
"ly":2.122999906539917,
"lz":-0.22100147604942322,
"rx":0.9099972248077393,
"ry":2.122999906539917,
"rz":-0.22099855542182922,
"hx":0.8208140134811401,
"hy":2.4463324546813965,
"hz":0.6848392486572266,
"rItemHeld":null
"lItemHeld":null,
"partition":2,
"offset":15
}
}
State Schema: Example

12. 1. Maven Plugin to generate Java Classes for Avro Schema: org.apache.avro.avro-maven-plugin
2. Parse STOMP messages as Generic Avro Record.
Inspired by stackoverflow answer from @lucapette here
eventSchemaPath = IOUtils.toString(classLoader.getResourceAsStream("event.avsc"));
Schema eventSchemaParser =parser.parse(eventSchemaPath);
@MessageMapping("/event")
public void pushEvents(Eventevent){
String key=event.getSessionId();
GenericRecord eventAvroRecord=mapObjectToRecord(event, eventSchemaParser);
ProducerRecord<Object,Object>record =new ProducerRecord<>(config.getEventTopicName(), event.getSessionId(),
eventAvroRecord);
}
public GenericData.Record mapObjectToRecord(Objectobject,Schema schema) {
final GenericData.Record record =new GenericData.Record(schema);
schema.getFields().forEach(r->record.put(r.name(),PropertyAccessorFactory.forDirectFieldAccess(object).getPropertyValue(r.name())));
return record;
}
KAFKA Producer: Generic Avro Record

13. Using GlobalKTable to maintain user's active status
String[] state = {"GameStart","GameEnd"};
valueGenericAvroSerde.configure(serdeConfig, false); // `false` for record values
StreamsBuilder builder = new StreamsBuilder();
builder.stream("event-topic", Consumed.with(keySerde,valueGenericAvroSerde))
.map((key,value) -> KeyValue.pair(value.get("sessionId").toString(),value.get("name").toString()))
.filter((key,value) -> Arrays.stream(state).anyMatch(value::equalsIgnoreCase))
.to("user-status");
GlobalKTable<String, String> userStatusTable=builder.globalTable("user-status",Consumed.with(Serdes.String(),
Serdes.String()),Materialized.as(”active-user-status"));
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);
streams.cleanUp();
streams.start();
Active User Status: GlobalKTable

14. ML 101: Finding Patterns
Mining Association Rules Based on
Apriori Algorithm
Problems
• One hot Encoding Numeric Data
• Updating the model needs to update the application.
• Computational Complexity Problem.
Prediction Pattern:
• Understanding Past User Behavior
• Personalization of current User

15. Events:
• No. of Frames
• No. of Tasks
• Grip Right
• Grip Left
• Trigger Right
• Trigger Left
• Both Right
• Both Left
• Only Uses One Hand
• No. of Drops
• No. of Incorrect Objects Picked Up
• No. of times looking at trainer
Derived Events:
• No. of times not looking at object of interest
• Hands moving towards/away from object of interest:
• This is not an once-off event, but runs over time through the task.
• Hands jittering
• Picking up the wrong item
• Touching the wrong item
• Dropping an object
• Hitting an object (including surfaces or hood)
• Facing away from Objects of interest
• Task Start/End

16. For creating association rules, algorithm needs a
boolean matrix as a parameter. Because of this user /
event count data should be converted a boolean
matrix.
Data Encoding Challenge
PossibleApproaches
• Thresholding
• Ranging
• Bucketing
• Smart Bucketing
Colour Red Yellow Green
Red
Red
Yellow
Yellow
Green
1 0 0
1 0 0
0 1 0
0 0 1

17. Predicting Responses:
eventStream.join(activeUserGlobalKTable,
(session, event) -> event.getSessionId(),
(event, userStatus) -> model.predict(event, userStatus))
.filter((key , response) -> response.getMessageText()!=null)
.to("response-topic");
{
"type": "record",
"namespace": "com.accenture",
"name": "Response",
"version": "1",
"fields": [
{ "name": "responseId", "type": "string", "doc": "Unique Id per record" },
{ "name": "sessionId", "type": "string", "doc": "session Id per record" },
{ "name": "action", "type": "string", "doc": "NA" },
{ "name": "severity", "type": "int", "doc" : "NA" },
{ "name": "messageText", "type": "string", "doc": "NA" },
{ "name": "messageType", "type": "string", "doc": "predefined message types"},
{ "name": "messageObject", "type": "string" , "doc": "description of event"}
]
}
Response Schema

18. Low Latency Predictions: Kafka Producers in Python
oarp = OnlineAssociationRulePrediction()
try:
oarp.initialize_model(application_id, Config.S3_ADDRESS,Config.S3_KEY,Config.S3_SECRET)
except:
print("no model")
execute = False
while (execute):
event_msg = event_consumer.poll(600000)
state_msg = state_consumer.poll(600000)
eventRecord = decode_message(event_msg, event_schema)
stateRecord = decode_message(state_msg, state_schema)
append_windowed_df(state_msg)
analytics_response= oarp.predict(state_data=state_df, event_data=event_df)
# Return responseto Unity
key = {"name": eventRecord["sessionId"]}
for responseinanalytics_response:
value = {"responseId": str(uuid.uuid4()), "sessionId": eventRecord["sessionId"], "action": response["action"],"severity":
response["severity"], "messageText": response["messageText"], "messageObject":
response["messageObject"], "messageType": response["messageType"] }
avro_producer.produce(topic=Config.RESPONSE_TOPIC, value=value, key=key)

19. public void run(String... strings)
{
final Properties streamsConfiguration =newProperties();
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,Serdes.String().getClass());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
KStream<String, GenericRecord>responseStream =builder.stream(“response-topic”,
Consumed.with(keySerde,valueGenericAvroSerde));
responseStream.foreach((key,value)->publishToVR(key,value.toString()));
final KafkaStreams streams =newKafkaStreams(builder.build(), streamsConfiguration);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
public void publishToVR(String sessionId,String message)
{
brokerMessagingTemplate.convertAndSendToUser(sessionId,"/topics/feedback",JSON.toJSON(message));
}
Stream Response: Generic Avro Record

20. Challenges and Learnings
• Handling Multi-tenancy in streaming
• Pushing new models in streaming app.
• Handling offline devices could be challenging.
• Featuredetection

21. Key Takeaways
• Extended Reality needs streaming.
• Transfer learning between environments.
• Using Avro map with Schema Registry is powerful.
• Everything is an event except state.
• Node based deployments in Kubernetes for robustcluster.
• VR data is biometric, so ask for Consent.
• STOMP compatibility with Kafka
(Simple TextOrientated Messaging Protocol)

22. XR Insights Demo

23. Thank You
accenture.com/thedock
Navdeep Sharma | Data & AI Engineer
navdeep.a.sharma@accenture.com
/showcase/accenture-the-dock @AccentureDock