A Fusion of Machine Learning and Graph Analysis for Free-Form Data Entry Clustering

1. © 2022 Neo4j, Inc. All rights reserved.
© 2022 Neo4j, Inc. All rights reserved.
A Fusion of Machine Learning
and Graph Analysis for Free-
Form Data Entry Clustering
Dr. Andrew Flinders
Joel Linford
Data Scientists
Northrop Grumman Corporation – Space Sector

2. © 2022 Neo4j, Inc. All rights reserved.
Repair Narratives
Building Clusters
BERT Embeddings [1]
Constelations
2

3. © 2022 Neo4j, Inc. All rights reserved.
3
Motivation
Problem
• Maintenance Records
• Need to identify patterns and structures present in free form text
• Finding general topics can be challenging
Hypothesis
• We hypothesized that the combination of large language models (deep
learning), clustering techniques (shallow learning), and graph databases
(graph algorithms) could be used to map and retain these patterns.

4. © 2022 Neo4j, Inc. All rights reserved.
Narratives – free form text with vital info
• 10 REPLACED ALL PISTONS
• 11 CLEANED HUBCAPS
• 12 COMPLETED DRIVE SHAFT CO
• 13 REPLACED WATER PUMP
• 14 SCHEDULED MAINTENANCE
• 15 CLEANED INJECTORS
• 16 CLEANED FLOOR MATS
• 17 PATCHED WIRING IN CAB
• 18 LABELED SEATING ASSIGNMENTS
• 19 NO FOUND ON SPARK PLUGS
4
This technique will
work for any free
form text where
there is a reason to
believe that there are
patterns or trends.
Here are a couple of
examples of the text
we are working with.

5. © 2022 Neo4j, Inc. All rights reserved.
BERT Embeddings [1]
BERT is a language model which embeds text into semantically sensitive
vectors (as opposed to a Bag of Words model, which is mostly semantically
insensitive.)
These vectors are extremely effective at allowing text to be used for machine
learning.
BERT is a Deep Neural Net (bringing Deep learning and Transfer Learning to
play.)
How BERT was trained (and why it did not use twitter)
5

6. © 2022 Neo4j, Inc. All rights reserved.
Using BERT [1] embeddings for clustering
6 Image from “Attention Is All You Need.” [1]
“We fixed the thing” –[BERT]-> [0.124, 0.432, 0.4523, ….. , 1.2432]

7. © 2022 Neo4j, Inc. All rights reserved.
Clustering Algorithms
We tested several clustering algorithms. My favorite is the OPTICS [2]
clustering algorithm as implemented by Sci-Kit Learn [3]. We also tested the
DBScan [4] Method and the KNN [5] method.
7
The optics algorithm detects
dense groupings in the data and
designates those as cluster
cores. It then allows the cluster
to grow to a certain point and
excludes outliers. This can be
helpful for identifying unique
entries. Image from https://scikit-
learn.org/stable/auto_examples/
cluster/plot_optics.html#sphx-glr-
auto-examples-cluster-plot-
optics-py [3]

8. © 2022 Neo4j, Inc. All rights reserved.
8
Cluster
1
Cluster
2
Cluster
3
Cluster
4
WATER PUMP
SEIZEDREQS WATER
PUMP REPLACE
REPLACE WATER
PUMP
REPLACE WATER
PUMP
REPLACE WATER
PUMP REQS CO
REPLACE WATER
PUMP
REPLACED THE
WATER PUMP ALL
CODES CLEARED
REPLACED THE
WATER PUMP ALL
CODES CLEARED
REPLACED WATER
PUMP CODES
CLEARED
REPLACED WATER
PUMP ALL CODES
CLEARED
REPLACED WATER
PUMP ALL CODES
CLEARED
TEAM CLEANED
PLUGS ALL CODES
CLEARED
TEAM CLEANED
PLUGS ALL CODES
CLEARED
TEAM CLEANED
PLUGS ALL CODES
CLEARED
TEAM CLEANED
PLUGS ALL CODES
CLEARED
AIR CONDITIONER
LEAKING
REFRIGERANT
REPLACED
AIR CONDITIONER
WAS LEAKING
REFRIGERANT AIR
CONDITIONER
CHILLER ALL CODES
CLEARED
REPLACED AIR
CONDITIONER FOR
LOW REFRIGERANT

9. © 2022 Neo4j, Inc. All rights reserved.
9
Similarity within Clusters
1
Performed
Corrosion
Control
Performed
Corrosion
Control
Performed
Corrosion
Control
Corrosion
Control
Performed
Corosion
Control
Performed
CC
[in]
[in]
[in]
[in]
[in]
[in]
Key:
Narrative
Cluster

10. © 2022 Neo4j, Inc. All rights reserved.
10
Dissimilarity for Un-clustered Entries
Perf MaintCOntrol
Sandwich
found in
pump
31542.1240
• Really unique problems
• Misspelled entries
• Data entered incorrectly
CND

11. © 2022 Neo4j, Inc. All rights reserved.
11
1. Average embeddings saved
on each cluster
2. Euclidean distance calculated
between each cluster center
Cluster Linking
Change colors
Key:
Narrative
Cluster

12. © 2022 Neo4j, Inc. All rights reserved.
12
Current/Future Work: Graph Algorithms
• Centrality – what are the most important nodes?
• Pathfinding
• Similarity
• Community Detection
• What graph algorithms have you guys used and had
success with? We are looking to try some soon.

13. © 2022 Neo4j, Inc. All rights reserved.
13
Text
Language
Model
Clusteri
ng
Graph
Building
Graph
Algos
Fine
tuning
Clustering
Algos
Graph
Design
LABEL EDGES
[input]
[then] [then]
[then]
[adjustable] [adjustable] [adjustable]

14. © 2022 Neo4j, Inc. All rights reserved.
14
Current/Future Work: Fine tuning BERT [1]
• Fine tuning a language model will improve efficacy with our dataset
(probably helpful in almost every application.)
• Considering the training set for language models, it is probable that they
will struggle with slang and modern connotations. (Someone should study
this.)
• We have been looking into fine tuning, and we think we have it working,
but we have not tested it yet.

15. © 2022 Neo4j, Inc. All rights reserved.
15
Current/Future Work: Summary Stats
• Clearly the clusters have meaning
◦ But summary statistics seem uninteresting
• This is likely due to over-simplification
◦ I.e. reduction of 500+ degree vectors to one Euclidean
distance just lost too much information.
• Averaging seems to be… ok but not amazing.

16. © 2022 Neo4j, Inc. All rights reserved.
16
Patterns interlocking systems
Fluid
Change
Valve
Gasket
Piston
Engine Transmition
Sched.
Maint.
[subset_of]
[subset_of]
[subset_of]
[subset_of]
[subset_of]
[subset_of]
[subset_of]
Key:
Broad Clustering
Narrow Clustering

17. © 2022 Neo4j, Inc. All rights reserved.
17

18. © 2022 Neo4j, Inc. All rights reserved.
© 2022 Neo4j, Inc. All rights reserved.
18
Thank you!