Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

1. How a distributed graph
analytics platform uses
Apache Kafka for data
ingestion in real time
Rayees Pasha & Duc Le
Kafka Summit US - Sep 2021
1

2. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Agenda
● Overview of Graph analytics and TigerGraph
● Overview of Data ingestion into TigerGraph
● Use of Kafka Connect Framework and Benefits
● TigerGraph Data Ingestion Deep dive
● Demo - Data Ingestion using Kafka on TG Cloud
2

3. © 2020. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Rayees Pasha
Product Lead,
TigerGraph
● Responsible for
TigerGraph Database
Engine, Language and
Platform areas of the
product.
● Prior Lead PM and ENG
positions at Workday,
Hitachi and HP
● Expertise in Database
Management and Big
Data Technologies
Session Presenters
3
Duc Le
Engineering Manager,
TigerGraph
● Lead Developer for
TigerGraph Cloud
● Master in Management
Information Systems from
Carnegie Mellon University
● Areas of specialty:
Full-stack Development,
Cloud, Containers and
Connectors

4. Overview of Graph
Analytics and TigerGraph
4

5. © 2020. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Why Graph; Why Now?
Businesses want to ask business logic
questions of their data
Blending data from multiple sources,
multiple business units, and
increasingly external data
Larger and more varied datasets mean
more variables to analyze and
connections to explore and test
Importance of Graph in Today’s World
5

6. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM |
6
6
Who is TigerGraph?
We provide advanced analytics and machine learning on connected data
○ The only scalable graph database for the enterprise: 40-300x faster than
competition
○ Foundational for AI and ML solutions
○ Designed for efficient concurrent OLTP and OLAP workloads
○ SQL-like query language (GSQL) accelerates time to solution
○ Available on-premise & on: Google GCP, Microsoft Azure,
Our customers include:
○ The largest companies in financial services, healthcare, telecom, media, utilities
and innovative startups in cybersecurity, ecommerce and retail
Founded in 2012, HQ in Redwood City, California
Corporate Overview Video

7. © 2020. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Advanced Analytics and Machine Learning on Connected Data
Advanced
Analytics
LEARN FROM CONNECTED DATA
AI-based Customer 360 for entity resolution,
recommendation engine, fraud detection
In-Database
Machine Learning
Distributed
Graph DB
Friction-free scale up from GB to TB to
Petabyte with lowest cost of ownership
.
CONNECT ALL DATASETS
AND PIPELINES
Customer 360 connecting 200+
datasets and pipelines
Item 360 for eCommerce across 100+
datasets
Fortune 50 Retailer
7 out of top 10 global banks
Real-time fraud detection and credit risk
assessment
10-100X faster than current solutions
ANALYZE CONNECTED DATA
Automotive Manufacturer
Supply chain planning accelerated
from 3 weeks to 45 minutes
Leading Healthcare Provider
7
Leading FinTech Company

8. Overview of Data Ingestion
into TigerGraph
8

9. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
TigerGraph Architecture

10. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Modes of Data Ingestion supported
Bulk Data
• Bulk data loads
using native File
loader
File Loader
Low-latency
● JDBC Type 4 driver for
Java, Python
● Spark can be used for
parallel loads
Real-time
● Streaming Data
Applications
● High-frequency Data
Apps
Bulk Data
Bulk data loads
using
•Native File loader,
•Kafka loader
Low-latency
● JDBC Type 4
driver for Java,
Python
● Spark can be
used for parallel
loads
Real-time
● Streaming Data
Apps
● High-frequency
Data Apps
Native File
Loader

11. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Data Ingestion Into TigerGraph Using Kafka loader
11
Step 3
Each GPE consumes the
partial data updates,
processes it and puts it on
disk.
Loading Jobs and POST use
UPSERT semantics:
● If vertex/edge doesn't
yet exist, create it.
● If vertex/edge already
exists, update it.
● Idempotent
Step 1
Loaders take in user source
data.
● Bulk load of data files or
a Kafka stream in CSV or
JSON format
● HTTP POSTs via REST
services (JSON)
● GSQL Insert commands
Step 2
Dispatcher takes in the data
ingestion requests in the form of
updates to the database.
1. Query IDS to get internal
IDs
2. Convert data to internal
format
3. Send data to one or more
corresponding GPEs

12. Use of Kafka Connect
Framework and Benefits
12

13. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Data
Source 1
Data
Source 2
Data
Source 3
TigerGraph Connector Framework Using Kafka Connect
TigerGraph
Cluster
Kafka Connect
Kafka (Can be customer-hosted)
Loader
(Available 2021Q4)

14. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
TigerGraph Connector Framework - Benefits
● Full control of data ingestion process
○ Throttle intake based on capacity
○ Pause as needed
○ Resume and restart data ingestion jobs as needed.
● Flexibility of system deployment
○ Works with natively deployed Kafka in the TigerGraph cluster
○ Allows customers to leverage existing TigerGraph with drop-in
integration with external Kafka cluster
● Push down ETL capabilities
○ Users can use data transformation with loader support for UDF
functions

15. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Kafka Loader
Easy integration of data sources
Kafka Connect
+
Data source
connector

16. Current Data Ingestion
Architecture Deep Dive
16

17. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Current Use of TigerGraph Connector Framework
AWS S3
TigerGraph
Cluster
Kafka Connect
Kafka
User Input
Language
Server
GraphStudio
(browser)
Kafka
Stream
GSQL CLI

18. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Define the Data Source:
● CREATE DATA_SOURCE S3 s = "/path/to/s3.config"
● s3.config
S3 Loading Job through GSQL
{
"file.reader.settings.fs.s3a.access.key": "AKIAJ****4YGHQ",
"file.reader.settings.fs.s3a.secret.key": "R8bli****p+dT4"
}

19. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Create a Loading Job
● loading_job.gsql
● files.config
S3 Loading Job through GSQL
{
"file.uris": "s3://my-bucket/data.csv"
}
CREATE LOADING JOB job1 FOR GRAPH my_graph {
DEFINE FILENAME f = "$s:/path/to/files.config";
LOAD f TO VERTEX v1 VALUES ($0, $1, $2);
}

20. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Run the Loading Job
● RUN LOADING JOB job1
S3 Loading Job through GSQL

21. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Define the Data Source:
S3 Loading Job through GraphStudio

22. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Map Data Files to Vertex type or Edge type
S3 Loading Job through GraphStudio

23. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Map Data columns to Vertex or Edge attributes
S3 Loading Job through GraphStudio

24. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Run the Loading Job
S3 Loading Job through GraphStudio

25. Demo using TigerGraph
GraphStudio Application
25

26. Thanks
26