The document discusses the integration of Kafka Streams and machine learning in the context of extended reality (XR) at Accenture's R&D center. It highlights the architecture for a generic streaming platform that captures XR insights, focusing on applications in various industries such as healthcare and education, while detailing challenges in machine learning model deployment. Key takeaways emphasize the importance of real-time data streaming, the power of schema management, and necessary consent for biometric data usage.

1. Closing the Loop in Extended
Reality with Kafka Streams and
Machine Learning
Rob Drysdale | Senior Principal | robert.drysdale@accenture.com
Sarah Healy | Software Engineer | s.healy@accenture.com

2. The Dock is Accenture’s flagship
R&D and Global Innovation
Centre
300+ multi-disciplinary team
42% female
35+ nationalities
142 patents filed
Based in Silicon Docks, Dublin.
A hub for many global tech companies

3. • Extended reality in mainstream
• XR Insights : Usecase
• Building Generic Streaming Platform: Architectureand Schema
• Deep dive into our implementation
• Machine Learning 101 : Finding Patterns
• Learnings & Takeaways
• Demo
Agenda

4. XR Industry
• Exponential Growth
• Industry Adoption:
• Healthcare
• Aviation
• Education & Training
• Retail & Advertising
• Training: VR
• Safer Healthcare
• Cheaper
• Monitorable
• On-the-job: AR
• Future Worker
• Hands-free Manuals
• Remote Guidance
• Automated Guidance
• Prediction/Prevention

5. A Streaming platform with capability to
plug & play VR/AR devices and ML models.
XR Insights
Data Stream Capture:
• State Stream
• Event Stream
Data Analyses:
• Data-Driven User Assessment:
o Improving Accuracy of Assessment
o Removing Subjectivity of Assessment
• Insights on Performance and behaviour
• Personalise Training
• Improve Work/Training Process
• Predict/Guide User behaviour

6. TrainingData
WebSocket
Server
(STOMP)
/topic/feedback
XR Insights
/app/state
/app/event
Kafka
Producer
Kafka
Consumer
Kafka Stream Data Transformations
Kafka Connect
Master DB
BuildingML Model
Model & Features
S3 Bucket
Production ML Model
Personalization
Real Time Reporting
XR Devices

7. • Speed of Data Transmation
• Consumer Groups
• Easy Embedding of Kafka clients in existing servicedeployments
• Avro Schema support
• Clients for Java & Python
• Eventdriven & supportto query on real-time data
Why Kafka Streams?

8. Kafka
Producer
Kafka
Consumer
Schema Registry
XR Devices
Schema Registry: Single point of Success

9. • Kubernetes 1.13 & Docker (Statefulsets)
• Spring boot with WebSocket + STOMP
• Unity & C#
• Kafka 2.1.0, Kafka Streams DSL
• Schema Registry, Avro, ConnectCluster
• Couchbase, Grafana, Elastic search
Technologies

10. {
"type": "record",
"namespace": "com.accenture",
"name": "Event",
"version": "1",
"fields": [
{ "name": "eventId", "type": "string", "doc": "Unique Id per record" },
{ "name": "sessionId", "type":"string", "doc": "session Id per record" },
{ "name": "timestamp", "type": "float", "doc": "Timestamp of Event" },
{ "name": "type", "type": "string", "doc" : "Type of Event" },
{ "name": "name", "type": "string", "doc": "Name of Event" },
{ "name": "desc", "type": {"type": "map", "values": "string"} , "doc":
"description of event"}
]
}
{
"topic":"event-topic",
"key":"d433b1db-4f89-4128-896f-95ac204b8681",
"value":{
"eventId":"d433b1db-4f89-4128-896f-95ac204b8681",
"sessionId":"13937ec1-82e1-4071-a6e7-31d31674de25",
"timestamp":148.45999,
"type":"Error",
"name":"Drop",
"desc":{
"collided":"Floor",
"Task id":"0",
"point of contact":"(1.1 1.0 -1.1)",
"lhand":"NA",
"collider":"spray_bottle",
"rhand":"NA"
}
},
"partition":2,
"offset":10
}
Event Schema: Example

11. {
"type": "record",
"namespace": "com.accenture",
"name": "State",
"version": "1",
"fields": [
{ "name": "stateId", "type":["null", "string"], doc:”NA” },
{ "name": "sessionId", "type":["null", "string"], "doc": ”NA" },
{ "name": "timestamp", "type": "double", "doc": ”NA" },
{ "name": "task", "type": "long", "doc": ”NA" },
{ "name": "subtask", "type":["null", "string"], "doc": ”NA" },
{ "name": "lx", "type": "double", "doc": "NA" },
{ "name": "ly", "type": "double", "doc": "NA" },
{ "name": "lz", "type": "double", "doc": "NA" },
{ "name": "rx", "type": "double", "doc": "NA" },
{ "name": "ry", "type": "double", "doc": "NA" },
{ "name": "rz", "type": "double", "doc": "NA" },
{ "name": "hx", "type": "double", "doc": "NA" },
{ "name": "hy", "type": "double", "doc": "NA" },
{ "name": "hz", "type": "double", "doc": "NA" },
{ "name": "rItemHeld", "type":["null", "string"], "doc": "NA"}
{ "name": "lItemHeld", "type":["null", "string"], "doc": "NA"},
{ "name": "ltrigger", "type":["null", "boolean"], "doc": "NA"},
{ "name": "rtrigger", "type":["null", "boolean"], "doc": "NA"} ]
}
{
"topic":"state-topic",
"key":"13937ec1-82e1-4071-a6e7-31d31674de25",
"value":{
"stateId":{
"string":"cdfc5d94-3f1e-404a-ba27-632b3616dbfd"
},
"sessionId":{
"string":"13937ec1-82e1-4071-a6e7-31d31674de25"
},
"timestamp":126.98802185058594,
"task":0,
"subtask":{
"string":"0"
},
"lx":0.6099972724914551,
"ly":2.122999906539917,
"lz":-0.22100147604942322,
"rx":0.9099972248077393,
"ry":2.122999906539917,
"rz":-0.22099855542182922,
"hx":0.8208140134811401,
"hy":2.4463324546813965,
"hz":0.6848392486572266,
"rItemHeld":null
"lItemHeld":null,
"partition":2,
"offset":15
}
}
State Schema: Example

12. 1. Maven Plugin to generate Java Classes for Avro Schema: org.apache.avro.avro-maven-plugin
2. Parse STOMP messages as Generic Avro Record.
Inspired by stackoverflow answer from @lucapette here
eventSchemaPath = IOUtils.toString(classLoader.getResourceAsStream("event.avsc"));
Schema eventSchemaParser =parser.parse(eventSchemaPath);
@MessageMapping("/event")
public void pushEvents(Eventevent){
String key=event.getSessionId();
GenericRecord eventAvroRecord=mapObjectToRecord(event, eventSchemaParser);
ProducerRecord<Object,Object>record =new ProducerRecord<>(config.getEventTopicName(), event.getSessionId(),
eventAvroRecord);
}
public GenericData.Record mapObjectToRecord(Objectobject,Schema schema) {
final GenericData.Record record =new GenericData.Record(schema);
schema.getFields().forEach(r->record.put(r.name(),PropertyAccessorFactory.forDirectFieldAccess(object).getPropertyValue(r.name())));
return record;
}
KAFKA Producer: Generic Avro Record

13. Using GlobalKTable to maintain user's active status
String[] state = {"GameStart","GameEnd"};
valueGenericAvroSerde.configure(serdeConfig, false); // `false` for record values
StreamsBuilder builder = new StreamsBuilder();
builder.stream("event-topic", Consumed.with(keySerde,valueGenericAvroSerde))
.map((key,value) -> KeyValue.pair(value.get("sessionId").toString(),value.get("name").toString()))
.filter((key,value) -> Arrays.stream(state).anyMatch(value::equalsIgnoreCase))
.to("user-status");
GlobalKTable<String, String> userStatusTable=builder.globalTable("user-status",Consumed.with(Serdes.String(),
Serdes.String()),Materialized.as(”active-user-status"));
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);
streams.cleanUp();
streams.start();
Active User Status: GlobalKTable

14. ML 101: Finding Patterns
Mining Association Rules Based on
Apriori Algorithm
Problems
• One hot Encoding Numeric Data
• Updating the model needs to update the application.
• Computational Complexity Problem.
Prediction Pattern:
• Understanding Past User Behavior
• Personalization of current User

15. Events:
• No. of Frames
• No. of Tasks
• Grip Right
• Grip Left
• Trigger Right
• Trigger Left
• Both Right
• Both Left
• Only Uses One Hand
• No. of Drops
• No. of Incorrect Objects Picked Up
• No. of times looking at trainer
Derived Events:
• No. of times not looking at object of interest
• Hands moving towards/away from object of interest:
• This is not an once-off event, but runs over time through the task.
• Hands jittering
• Picking up the wrong item
• Touching the wrong item
• Dropping an object
• Hitting an object (including surfaces or hood)
• Facing away from Objects of interest
• Task Start/End

16. For creating association rules, algorithm needs a
boolean matrix as a parameter. Because of this user /
event count data should be converted a boolean
matrix.
Data Encoding Challenge
PossibleApproaches
• Thresholding
• Ranging
• Bucketing
• Smart Bucketing
Colour Red Yellow Green
Red
Red
Yellow
Yellow
Green
1 0 0
1 0 0
0 1 0
0 0 1

17. Predicting Responses:
eventStream.join(activeUserGlobalKTable,
(session, event) -> event.getSessionId(),
(event, userStatus) -> model.predict(event, userStatus))
.filter((key , response) -> response.getMessageText()!=null)
.to("response-topic");
{
"type": "record",
"namespace": "com.accenture",
"name": "Response",
"version": "1",
"fields": [
{ "name": "responseId", "type": "string", "doc": "Unique Id per record" },
{ "name": "sessionId", "type": "string", "doc": "session Id per record" },
{ "name": "action", "type": "string", "doc": "NA" },
{ "name": "severity", "type": "int", "doc" : "NA" },
{ "name": "messageText", "type": "string", "doc": "NA" },
{ "name": "messageType", "type": "string", "doc": "predefined message types"},
{ "name": "messageObject", "type": "string" , "doc": "description of event"}
]
}
Response Schema

18. Low Latency Predictions: Kafka Producers in Python
oarp = OnlineAssociationRulePrediction()
try:
oarp.initialize_model(application_id, Config.S3_ADDRESS,Config.S3_KEY,Config.S3_SECRET)
except:
print("no model")
execute = False
while (execute):
event_msg = event_consumer.poll(600000)
state_msg = state_consumer.poll(600000)
eventRecord = decode_message(event_msg, event_schema)
stateRecord = decode_message(state_msg, state_schema)
append_windowed_df(state_msg)
analytics_response= oarp.predict(state_data=state_df, event_data=event_df)
# Return responseto Unity
key = {"name": eventRecord["sessionId"]}
for responseinanalytics_response:
value = {"responseId": str(uuid.uuid4()), "sessionId": eventRecord["sessionId"], "action": response["action"],"severity":
response["severity"], "messageText": response["messageText"], "messageObject":
response["messageObject"], "messageType": response["messageType"] }
avro_producer.produce(topic=Config.RESPONSE_TOPIC, value=value, key=key)

19. public void run(String... strings)
{
final Properties streamsConfiguration =newProperties();
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,Serdes.String().getClass());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
KStream<String, GenericRecord>responseStream =builder.stream(“response-topic”,
Consumed.with(keySerde,valueGenericAvroSerde));
responseStream.foreach((key,value)->publishToVR(key,value.toString()));
final KafkaStreams streams =newKafkaStreams(builder.build(), streamsConfiguration);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
public void publishToVR(String sessionId,String message)
{
brokerMessagingTemplate.convertAndSendToUser(sessionId,"/topics/feedback",JSON.toJSON(message));
}
Stream Response: Generic Avro Record

20. Challenges and Learnings
• Handling Multi-tenancy in streaming
• Pushing new models in streaming app.
• Handling offline devices could be challenging.
• Featuredetection

21. Key Takeaways
• Extended Reality needs streaming.
• Transfer learning between environments.
• Using Avro map with Schema Registry is powerful.
• Everything is an event except state.
• Node based deployments in Kubernetes for robustcluster.
• VR data is biometric, so ask for Consent.
• STOMP compatibility with Kafka
(Simple TextOrientated Messaging Protocol)

22. XR Insights Demo

23. Thank You
accenture.com/thedock
Navdeep Sharma | Data & AI Engineer
navdeep.a.sharma@accenture.com
/showcase/accenture-the-dock @AccentureDock