Scaling into Billions of Nodes and Relationships with Neo4j Graph Data Science

© 2022 Neo4j, Inc. All rights reserved.
Scaling into Billions of Nodes
and Relationships with Neo4j
Graph Data Science
Martin Junghanns
Senior Software Engineer
Neo4j - Product Engineering

Outline
• The Neo4j Graph Data Science (GDS) Platform
◦ Overview
◦ A generic GDS workflow
• Scalability Challenges
• Neo4j GDS under the microscope
◦ Graph Projection
◦ “Huge” Data Structures
◦ Algorithm Execution
◦ Arrow Data Import and Export
• Summary and Outlook
2

Before we start …
This is not an introduction to Neo4j Graph Data Science or Data Science in
general.
See the talks of my colleagues for more information:
Alicia: “Get Started with the Most Advanced Edition Yet of Neo4j Graph Data Science”
Zach: “Connecting Neo4j Graph Data Science into Your Data Ecosystem and Workflows”
Luke: “New! Neo4j AuraDS: The Fastest Way to Get Started”
Dave: “Achieve Blazing-Fast Ingest Speeds with Apache Arrow”
3

Neo4j Graph Data Science
4

Overview
• Main features
◦ in-memory property graph optimized for executing algorithms
◦ graph catalog for managing multiple named in-memory graphs
◦ vast collection of graph and machine learning algorithms
◦ accessible via Cypher procedures and GDS Python Client
• Community and Enterprise versions
◦ Enterprise contains improved data structures for graph representation
◦ Enterprise has additional features (e.g. Apache Arrow, Clustering support, …)
◦ Community is entirely open source; Enterprise code is closed source
5

A generic GDS workflow
1 Graph Projection
2 Algorithm Execution
3 Data Export
6

1 Graph Projection
2 Algorithm Execution
3 Data Export
In GDS 2.1
In GDS 2.1
7

CALL gds.graph.project
CALL gds.wcc.mutate
CALL gds.pageRank.mutate
CALL gds.louvain.mutate
...
CALL gds.graph.[stream,write]NodeProperties
CALL gds.graph.[stream,write]RelationshipProperties
...
doACTION(CREATE_GRAPH, ...)
doPUT(NODE_STREAM)
doPUT(RELATIONSHIP_STREAM)
doGET(NODE_PROPERTIES)
doGET(RELATIONSHIP_PROPERTIES)
...
8

Scalability Challenges
9

Scalability challenges - Customer facing
• Data size
◦ Customers with multiple billion nodes and relationships
◦ Not only topology, but also node and relationship properties
◦ Algorithm results often need to be stored in-memory
◦ Requires a compact in-memory graph representation
• Data import
◦ Customers want their graphs projected within minutes / hours, not days
◦ Requires parallel, non-blocking graph construction
◦ Customers may want their data to be projected from sources other than Neo4j
◦ Requires parallel, structured, compressed data streaming
10

Scalability challenges - Customer facing
• Algorithm execution
◦ Customers want their results computed as fast as possible (within minutes)
◦ Customer graphs vary in their topology (power-law vs uniform distributions)
◦ Requires parallel, topology-aware algorithm execution
• Data export
◦ Customers often export algorithm results for downstream analysis
11

Scalability challenges - Developer facing
• Java is a memory-managed, garbage-collected language
◦ Garbage Collector / Object overhead becomes non-negligible at scale
◦ Requires usage of primitive data types instead of objects
• Java has no generic data structures for handling primitive data types
◦ List<int> is not supported in Java
◦ long[] is limited to ~2.1 bn entries
◦ Requires custom data structure implementations
12

Neo4j GDS under the microscope
Huge Data Structures
13

• “Java has no generic data structures for handling primitive data types”
◦ int (4 Byte) vs. Integer (16 Byte), long (8 Byte) vs. Long (24 Byte)
◦ List<long> vs. List<Long> vs. long[]
• GDS uses High Performance Primitive Collections (hppc)
◦ non-thread-safe, primitive lists, sets and maps
• GDS provides “huge” collections for more than 2 bn elements
◦ Huge[Byte,Int,Long,Float,Double]Array
◦ HugeSparse[Byte,Int,Long,Float,Double]Array
◦ HugeSparse[Byte,Int,Long,Float,Double]ArrayList
◦ HugeAtomic[Byte,Int,Long,Float,Double]Array
◦ implemented to achieve equivalent performance as primitive arrays
https://github.com/carrotsearch/hppc
14

index 0 1 2 3 4 5 6 7 .. n
value 34 55 89 144 233 377 610 987 .. 42
HugeLongArray
index 0 1 2 3 4 5 6 7 .. n
value 34 55 89 144 null .. 42
HugeSparseLongArray
page (long[])
15

Graph Projection
16

● “Customers with multiple billion nodes and relationships”
○ Requires a compact in-memory graph representation
The in-memory graph
Node projection Relationship projection
17

● Nodes := Id Mapping + Node Labels + Node Properties
The in-memory graph - Nodes
id origId
0 42
1 1337
2 1984
..
n 0
id
0 1 0 0 1
1 0 1 0 0
2 0 1 0 1
..
n 1 0 0 0
:= + +
id foo bar
0 73301 [13, 37]
1 78717 [98]
2 78717 [1, 3, 4]
..
n 78751 [12, 13]
18

● Nodes := Id Mapping + Node Labels + Node Properties
The in-memory graph - Nodes
id origId
0 42
1 1337
2 1984
..
n 0
id
0 1 0 0 1
1 0 1 0 0
2 0 1 0 1
..
n 1 0 0 0
:= + +
id foo bar
0 73301 [13, 37]
1 78717 [98]
2 78717 [1, 3, 4]
..
n 78751 [12, 13]
19

● Node id space in the original data is often sparse
○ Neo4j: store fragmentation and/or subgraph projection
○ Arrow: customer data contains arbitrary 64 Bit long ids
● GDS uses a consecutive node id space internally: [0..nodeCount)
○ favors use of array-based data structures
○ simplifies algorithm implementation for (long id = 0; id < nc; id++)
○ favors relationship compression
● Id Mapping has two main methods
○ toInternalId(long originalId) -> long
○ toOriginalId(long internalId) -> long
● Different implementations for Community and Enterprise
The in-memory graph - Nodes - Id Mapping
20

● Community Edition - ArrayIdMap
id origId
0 42
1 1337
2 1984
...
n ..
class ArrayIdMap {
HugeLongArray originalIds; // maps from internal id to original id
HugeSparseLongArray internalIds; // maps from original id to internal id
long toOriginalId(long internalId) { return originalIds.get(internalId); }
long toInternalId(long originalId) { return internalIds.get(originalId); }
}
0 1 2 3 4 5 6 7 .. n
42 1337 1984 1985 1986 43 41 1987 .. ..
originalIds
0 1 2 3 4 5 6 7 .. 40 41 42 43 .. m
8 -1 -1 9 null .. -1 6 0 5 .. 7
internalIds
That’s 2*8 = 16 Byte for a
single id-to-id mapping.
21

● Enterprise Edition - BitIdMap
id origId
0 42
1 1337
2 1984
...
n ..
class BitIdMap {
LongLongBitMap map; // maps between both id spaces
long toOriginalId(long internalId) { return map.getKey(internalId); }
long toInternalId(long originalId) { return map.getValue(originalId); }
}
page 0
[0,63]
.. page 20
[1280,1343]
.. page 31
[1984,2047]
..
0x40000000000 .. 0x200000000000000 .. 0x1 ..
pages
long[]
Each page represents
64 original ids in a single
long value.
That’s 1 Bit for a single
id-to-id mapping.
pages[0] = 00000000 00000000 00000100 00000000 00000000 00000000 00000000 00000000
22

node
count
max original
node id
ArrayIdMap BitIdMap Reduction per
Mapping
1 bn 100 bn [15 GiB ... 752 GiB]
[~16 … 807] Byte / Mapping
12 GiB
~12 Byte / Mapping
~ 32 x
100 bn 100 bn 1490 GiB
~16 Byte / Mapping
12 GiB
~1.03 Bit / Mapping
~ 128 x
BitIdMap: 12386 MiB
|-- this.instance: 32 Bytes
|-- Identifier mapping: 12386 MiB
|-- pages: 11920 MiB
|-- block offsets: 186 MiB
|-- sorted block offsets: 186 MiB
|-- block mapping: 93 MiB
|-- Node Label BitSets: 0 Bytes
ArrayIdMap: [15356 MiB ... 752 GiB]
|-- original identifiers: 7630 MiB
|-- internal identifiers: [7726 MiB ... 745 GiB]
|-- Node Label BitSets: 0 Bytes
Community Edition Enterprise Edition
23

• BitSet per label
◦ constant lookup and efficient for label filters (union)
◦ memory consumption depends on node count: ~ nodeCount Bit
◦ 1 bn nodes ~ 120 MiB
The in-memory graph - Nodes - Node Labels
id origId
0 42
1 1337
2 1984
...
n 0
id
0 1 0 0 1
1 0 1 0 0
2 0 1 0 1
...
n 1 0 0 0
:= + +
id foo bar
0 73301 [13, 37]
1 78717 [98]
2 78717 [1, 3, 4]
...
n 78751 [12, 13]
24

• HugeSparse*Array per property
◦ constant lookup
◦ memory consumption depends on node count: ~ nodeCount * sizeOf(type)
◦ 1 bn long properties ~ 7634 MiB
The in-memory graph - Nodes - Node Properties
id origId
0 42
1 1337
2 1984
...
n 0
id
0 1 0 0 1
1 0 1 0 0
2 0 1 0 1
...
n 1 0 0 0
:= + +
id foo bar
0 73301 [13, 37]
1 78717 [98]
2 78717 [1, 3, 4]
...
n 78751 [12, 13]
25

● Relationships := AdjacencyList + Properties
The in-memory graph - Relationships
source target 0 target 1 target 2 target 3
1337 2001 1984 2005
foo 2.1 1.5 4.3
2003 3456 3442 1992 2005
foo 2.5 24.6 4.2 5.7
:=
source target 0 target 1 target 2
1234 3323 3323 4346
bar 4.2 2.1 8.4
26

● Accessing the adjacency of a node is the most important operation
○ adjacency must be accessible in constant time
○ adjacency must be stored “compact” in memory for cache efficiency
○ adjacency must be optimized for graph traversal
● Real-world graphs are sparse
○ each node is only connected to a subset of other nodes
○ relationship distributions vary depending on the use case
● Adjacency List (AL) has two main methods
○ AL::forEachRelationship(long nodeId, (src, tgt) -> {})
○ AL::forEachRelationship(long nodeId, (src, tgt, prop) -> {})
● Same implementation for Community and Enterprise
The in-memory graph - Relationships
Main API for
algorithm
implementers
27

Storing targets
uncompressed consumes
relationship count * 8 Byte
100 bn rels ~ 745 GiB
The in-memory graph - Relationships - CSR
• Adjacency List implementation: Compressed Sparse Row Variant
0 ... 1337 ... 2003 ... n
0 ... 4096 ... 5432 ... ...
offsets
HugeLongArray
0 ... 4096 - 4098 5432 - 5435
0 ... 1984 2001 2005 ... 1992 2005 3442 3456 ...
targets
long[][]
0 ... 1337 ... 2003 ... n
0 ... 3 ... 4 ... ...
degrees
HugeIntArray
https://en.wikipedia.org/wiki/Sparse_matrix
28

◦ Target compression is applied during graph projection
1984 2001 2005
sorting
delta encoding
1992 2005 3442 3456
source node 1337
1984 17 4 1992 13 1547 14
source node 2003
variable-length
encoding
2001 1984 2005
initial target lists 3456 3442 1992 2005
0x
40
0x
8F
0x
91
0x
84
0x
48
0x
8F
0x
8D
0x
1D
0x
8B
0x
84
32 Byte
6 Byte
24 Byte
4 Byte
29

0 ... 1337 ... 2003 ... n
0 ... 2100 ... 3205 ... ...
offsets
HugeLongArray
targets
byte[][]
0 ... 1337 ... 2003 ... n
0 ... 3 ... 4 ... ...
degrees
HugeIntArray
0 .. 2100 - 2103 .. 3205 - 3210 ...
0 .. 0x
40
0x
8F
0x
91
0x
84
.. 0x
48
0x
8F
0x
8D
0x
1D
0x
8B
0x
84
...
30

The in-memory graph - Relationships - Properties
• Relationship properties are also stored in CSR representation
◦ in the exact same order as the corresponding target node ids
◦ allows lock-step iteration of target ids and relationship values
targets
byte[][]
0 .. 6132 - 6134 .. 8192 - 8195 ..
0 .. 1.5 2.1 4.3 .. 4.2 5.7 24.6 2.5 ..
properties
long[][]
0 .. 2100 - 2103 .. 3205 - 3210 ..
0 .. 0x
40
0x
8F
0x
91
0x
84
.. 0x
48
0x
8F
0x
8D
0x
1D
0x
8B
0x
84
..
1984 2001 2005 1992 2005 3442 3456
31

The in-memory graph - Relationships - Memory
node count relationship count uncompressed values compressed values
1 bn 100 bn ~ 756 GiB
~8.2 Byte / Value
[106 GiB ... 382 GiB]
~1.4 Byte / Value
default for topology
CompressedAdjacencyList: [106 GiB ... 382 GiB]
|-- pages: [95 GiB ... 371 GiB]
|-- degrees: 3815 MiB
|-- offsets: 7630 MiB
UncompressedAdjacencyList: 756 GiB
|-- pages: 745 GiB
|-- degrees: 3815 MiB
|-- offsets: 7630 MiB
used for rel properties
32

The in-memory graph - Native Graph Projection
● Nodes and relationships are read via sequential scans of record stores
○ scan is IO-friendly since we touch a page only once (no random reads)
○ allows us to use a small page cache (~ page size * 100 * concurrency)
○ scales nearly linear with number of threads
● Nodes are read from record store
○ can also make use of node label index and property index if part of projection
○ threads read chunks of node records and forward them to thread-safe Id
Mapping-, Label- and Property-Builders
● Relationships are read from relationship record store
○ threads read chunks of relationships records, group them by source or target,
pre-compress and store partial adjacency lists in intermediate list
○ multi-threaded compression of partially compressed adjacency lists into CSR
33

Native Graph Projection - Scale Testing
• Imported Graph500* artificial data sets into Neo4j
◦ power-law relationship distribution, no properties
• Azure M416ms, 416 vCPU, 11TB RAM
◦ Neo4j with ~9 TB Heap and ~1 TB PageCache
◦ gds.graph.project with readConcurrency: 416
*Graph 500 Benchmark specification https://graph500.org/
scale node count (2^scale) relationship count (2^(scale + 4)) Neo4j disk size [GB]
30 1,073,741,824 17,179,869,184 1113
31 2,147,483,648 34,359,738,368 2226
32 4,294,967,296 68,719,476,736 4452
33 8,589,934,592 137,438,953,472 9272
34

Known problem. GDS
2.2 (fall ‘22) will improve
projection performance
for large scale graphs.
35

36

Algorithm Execution
37

Algorithm Execution - Compute template
• “Customers want their results computed as fast as possible.”
◦ Requires thread-safe, efficient data structures
Most GDS algorithms follow the same “compute pattern” for scalability:
1. Procedure call, e.g. CALL gds.wcc.mutate(...)
2. Input validation (meta data and algo configuration)
3. Graph Catalog read (potentially label / type filtered)
4. Compute
a. initialize algorithm specific data structures (e.g. Huge[Atomic]*Array)
b. partition node id space based on concurrency (range or degree partitioned)
c. create a task for each partition (task captures the algorithm logic)
d. run tasks in parallel (optionally repeat until termination criteria is reached)
5. Process result according to execution mode (stream, mutate, write, stats)
38

Algorithm Execution - Compute template
• Step 4 “Compute” can be implemented with one of two internal APIs:
• The Graph API (internal)
◦ Requires knowledge about GDS internals
◦ Most expressive API for manual optimization
◦ Often faster, but also more complex
◦ Developer needs to take care of partitioning, parallelism, thread-safety etc.
• The Pregel* API (external)
◦ Implemented on top of Graph API
◦ Requires less knowledge about GDS internals
◦ Less expressive API tailored for iterative, message-passing-based algorithms
◦ Automatic parallelization and load balancing using Fork-Join execution
*Malewicz et al: “Pregel: A System for Large-Scale Graph Processing”, Google
39

Algorithm Execution - Weakly Connected Components
40

Wcc::compute(Graph g, Config c) -> Components {
dss = init(g.nodeCount()) // DisjointSetStruct
tasks = rangePartition(g.nodeCount()).map(p -> new WccTask(g, p, dss))
executeParallel(tasks, c.concurrency())
return Components.of(dss)
}
class WccTask(Graph g, Partition p, DisjointSetStruct dss) {
run() {
p.forEachNode(node ->
g.forEachRelationship(node, (s, t) -> dss.union(s, t))
);
}
}
41 Implementation: https://tinyurl.com/mr2ayp75

dss = init(g.nodeCount()) // HugeAtomicDisjointSetStruct
}
run() {
);
}
}
// also called Union-Find
class DisjointSetStruct {
union(long idA, long idB);
setIdOf(long id) -> long;
}
Input: {{0}, {42}, {3}, {6}, {54}, {1337}}
dss.union(0, 42)
dss.union(3, 6)
dss.union(42, 1337)
Output: {{0, 42, 1337}, {3, 6}, {54}}
dss.setIdOf(42) // 0
dss.setIdOf(1337)// 0
dss.setIdOf(6) // 3
dss.setIdOf(54) // 54
42

© 2022 Neo4j, Inc. All rights reserved. Implementation: https://tinyurl.com/mr2ayp75
dss = init(g.nodeCount()) // DisjointSetStruct
}
run() {
);
}
}
Thread-safe,
wait-free
implementation
Graph uses thread-local
decompressing cursors
43

Algorithm Execution - WCC - Scale Testing
44

Algorithm Execution - Page Rank
45

Algorithm Execution - Page Rank
• PageRank is implemented using the Pregel API
◦ initialization, partitioning and task execution is handled by Pregel
◦ provides tailored queue implementation for message passing
◦ developer can focus on the algorithm logic in a vertex-centric computation
PageRank::compute(Messages messages, Context c) {
rank = c.getValue(PAGE_RANK)
if (c.superstep() > 0) {
sum = messages.sum()
rank = ((1 - dampingFactor) / c.nodeCount()) + dampingFactor * sum
c.setValue(PAGE_RANK, rank)
}
c.sendToNeighbors(rank)
}
Implementation: https://tinyurl.com/dpyhmmkw
Called in parallel
for each node in
each superstep
Messages sent by
other nodes in the
previous superstep
Page Rank
specific
logic
Send new
message to
neighbors
46

Algorithm Execution - Page Rank - Scale Testing
47

Arrow Data Import and Export
48

Apache Arrow
• “Customers may want their data to be projected from sources other than Neo4j”
• “Customers often export algorithm results for downstream analysis”
• Requires parallel, structured, compressed data streaming
• Arrow is a language-independent columnar
memory format for flat and hierarchical data:
◦ Data is organized for efficient analytic
operations on CPU and GPU
◦ Arrow Flight is used to move Arrow data
efficiently between processes and machines
◦ Written in C++ with wrappers for Java,
Rust, Python, C/C#, Matlab, R, etc.
Check out Dave’s talk:
“Achieve Blazing-Fast Ingest Speeds with Apache Arrow”
https://arrow.apache.org/
49

Graph Import via Apache Arrow
• GDS Arrow Flight Server is started /
stopped with the DBMS
• Arrow Flight Client must follow
protocol for importing data
• Import phases are indicated by
messages
• Record batches can be sent in
parallel by using multiple clients
Arrow Flight
Client
GDS Arrow
Flight Server
CREATE_GRAPH “my_graph”
ACK
Send Nodes Flight stream
NODE_LOAD_DONE “my_graph”
ACK
Send Relationships Flight stream
RELATIONSHIP_LOAD_DONE “my_graph”
ACK In-memory
graph
1
2
3
50 https://neo4j.com/docs/graph-data-science/current/graph-project-apache-arrow/

• GET actions mirror stream node and
relationship property procedures
• Output streams are produced in
parallel and can be consumed in
parallel
Property Export via Apache Arrow
GET Property Message
{
graph_name: “my_graph”,
procedure: “gds.graph.streamNodeProperty”,
config: {
node_labels: [“A”, “B”],
node_property: “foobar”
}
}
1
Send Node Property Flight stream
In-memory
graph
Arrow Flight
Client
GDS Arrow
Flight Server
51 https://neo4j.com/docs/graph-data-science/current/graph-catalog-apache-arrow-ops/

Arrow Data Import - Scale testing
• Graph 500 data set scale 32 (~4 bn nodes, ~68 bn relationships)
◦ 1 node property, 3 relationship properties
◦ Data stored in Google Storage
◦ Node files: 476 parquet files 101.7 MB each
◦ Relationship files: 50,001 parquet files 41.1 MB each
• Server: GCP m1-ultramem-160 (160vCPU - 3844GB RAM)
◦ Neo4j with 3.3 TB Heap and 12 GiB PageCache
• Ingest via Beam/Dataflow Configuration
◦ 68 dataflow workers (n2-standard-2)
◦ 128 concurrenct server-side graph creation threads
52

Arrow Data Import - Scale testing
● GDS Graph Load in ~3.5 hours
○ Nodes: 4,294,967,296 at a rate of ~6.2 M nodes/s
○ Relationships: 68,719,476,736 at a rate of ~8.0 M relationships/s
○ Overall throughput: 5.5M objects/s
Nodes
(11.5m)
Edges (2h 23m)
Graph
Creation
(55m)
53

Summary and Outlook
54

Summary - Scalability challenges
• Data size
◦ Requires a compact in-memory graph representation
◦ Compact Node Id Mappings and Compressed CSR data structure
◦ Low-level Graph API to achieve the optimal performance on a JVM
◦ Mid-level APIs to simplify the development of algorithms
• Data import / export
◦ Requires non-blocking graph construction
◦ Parallel construction of node and relationship data structures
◦ Apache Arrow provides a fast mechanism for data import / export
55

Outlook
• Property compression
◦ Node and relationship properties are currently stored uncompressed
◦ Mutate leads to many properties stored in-memory
◦ Alternative algorithm execution frameworks
◦ Off-load Machine Learning workloads to native libraries (e.g. tensorflow)
• Graph representation
◦ Allow to only load parts of the graph into memory to enable large-scale
computation with limited ressources
• Integration
◦ Apache Arrow opens a whole world of systems to interact with
◦ Access to this world needs to be made simple for customers
56

Thank you
martin.junghanns@neo4j.com
https://github.com/s1ck
@kc1s
Special thanks to
Kent Stroker, Sr. Sales Engineer at Neo4j
for setting up and running the GDS project and algorithm scale tests and
Dave Voutila, Sales Engineering Manager at Neo4j
for setting up and running the GDS Arrow scale tests

Scaling into Billions of Nodes and Relationships with Neo4j Graph Data Science

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scaling into Billions of Nodes and Relationships with Neo4j Graph Data Science

Similar to Scaling into Billions of Nodes and Relationships with Neo4j Graph Data Science (20)

More from Neo4j

More from Neo4j (20)

Recently uploaded

Recently uploaded (20)

Scaling into Billions of Nodes and Relationships with Neo4j Graph Data Science