Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.

1. Comparing three data ingestion
approaches where Apache Kafka
integrates with a distributed
graph database in real time
Benyue (Emma) Liu
Product Manager, TigerGraph
April, 2021

2. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Today's Speakers
Benyue (Emma) Liu
Senior Product Manager, TigerGraph
● BS in Engineering from Harvey Mudd College, MS in
Engineering Systems from MIT
● Prior work experience at Oracle and MarkLogic
● Focus - Cloud, Containers, Enterprise Infra, Monitoring,
Management, Connectors, Developer Tools,
Applications
2

3. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
2
TigerGraph Data Ingestion System
Architecture: Internal Kakfa Component
TigerGraph Cloud and Kafka Data
Pipeline Use Cases
Built-in Kafka Loader in TigerGraph
Today’s Outline
4
3
3
1 Graph Analytics Overview

4. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Graph Analytics
Overview
4

5. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
5
By 2025, graph technologies will be used
in 80% of data and analytics innovations,
up from 10% in 2021, facilitating rapid
decision making across the enterprise.1
“To Graph or Not to Graph? That Is Not
the Question — You Will Graph.”2
Mark Beyer, Distinguished VP Analyst
1Gartner, Top Trends in Data and Analytics for 2021, 16 February 2021
2Gartner, Graph Steps Onto the Main Stage of Data and Analytics: A Gartner Trend Insight Report, 14, December 2020

6. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM |
Why Graph Analytics?
Source: Gartner - Top 10 data and analytics Trends for 2019
Graph deployments are going deeper, wider and operational:
Need to make it accessible to non-technical users
6
● Definition - Graph analytics is a set of analytic techniques that
allows for the exploration of relationships between entities of
interest such as organizations, people and transactions.
● Forecasted growth - 100% annually through 2022
● What’s driving the growth
○ Need to ask complex questions across complex data, which is
not always practical or even possible at scale using SQL
queries. (RDBMS requires time-consuming & expensive table
joins!)
● What’s needed for broad adoption of graph data stores
○ Graph data stores can efficiently model, explore and query data
with complex interrelationships across data silos, but the need for
specialized skills has limited their adoption to date.
Customer
Supplier
Location 2
Product
Payment
PURCHASED
Location 1
Order

7. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM
Who is TigerGraph?
Corporate Overview Video
We provide advanced analytics on connected data
○ The only scalable graph database for the enterprise
○ HTAP graph database, foundational for AI and ML solutions
○ SQL-like query language (GSQL) accelerates time to solution
○ Cloud Neutral: Google GCP, Microsoft Azure,
Our customers include:
○ The largest companies in financial, healthcare, telecoms, media,
utilities and innovative startups in cybersecurity, ecommerce and
finserv
Founded in 2012, HQ in Redwood City, California
7

8. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM
How Our Customers Use TigerGraph?
8
Find most influential
users/customers
Find similar users/customers Who are the patients that are going through a particular type of
journey that results in an adverse health outcome?
Is the
Uncover hidden connections Is the new credit card applicant or transaction connected to
known fraudsters?
Recommend next best action Can I run a real-time credit score algorithm and recommend an
offer based on the customer’s credit profile & need?
Which users are driving higher usage or adoption of my product or
service?
Detect connected users
(communities)
What is average spend over time across a community of connected
users (fin. services, airlines, healthcare, retail..)?

9. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
TigerGraph
System Architecture
Overview
(Kafka as a Key Component)
9

10. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
The TigerGraph Difference
Feature Design Difference Benefit
Real-Time Deep-Link Querying ● Native Graph design
● C++ engine, for high performance
● Storage Architecture
● Uncovers hard-to-find patterns
● Operational, real-time
● HTAP: Transactions+Analytics
Handling Massive Scale ● Distributed DB architecture
● Massively parallel processing
● Compressed storage reduces
footprint and messaging
● Integrates all your data
● Automatic partitioning
● Elastic scaling of resource usage
In-Database Analytics ● GSQL: High-level yet Turing-
complete language
● User-extensible graph algorithm
library, runs in-DB
● ACID (OLTP) and Accumulators
(OLAP)
● Avoids transferring data
● Richer graph context
● In-DB machine learning
5 to 10+ hops deep
10

11. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
TigerGraph Architecture - Kafka as a Key Component
11

12. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Data Ingestion Steps Inside of TigerGraph
12
Step 3
Each GPE consumes the
partial data updates,
processes it and puts it on
disk.
Loading Jobs and POST use
UPSERT semantics:
● If vertex/edge doesn't
yet exist, create it.
● If vertex/edge already
exists, update it.
Step 1
Data integration through the
following ways to ingest in
user source data.
● Bulk load of data files or
a Kafka stream in CSV or
JSON format
● HTTP POSTs via REST
services (JSON)
● GSQL Insert commands
Step 2
Dispatcher takes in the data
ingestion requests in the form of
updates to the database.
1. Query IDS to get internal
IDs
2. Convert data to internal
format
3. Send data to one or more
corresponding GPEs

13. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Data Ingestion (Internal)
13
Incremental
Data
Nginx Restpp
GPE GPE GPE
Disk Disk Disk
CSV/JSON Insert/Update/Delete
Vertices and Edges
Listen to
corresponding
topic for new
messages
Acknowledge
Response
Incoming
Outgoing
Synchronize
data to disk
GSE(IDS)
ID Translation
Kafka Kafka Kafka
Server 1 Server 2 Server 3
Kafka Cluster
In-memory
copy of data

14. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Kafka and TigerGraph
-Native Kafka Loader
14
External
Kafka
Cluster

15. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Kafka and TigerGraph Data Pipeline
Static
Data
Sources
Streaming
Data
Sources
Kafka
Loader
15

16. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Kafka Loader - Speed to Value from Real-time
Streaming Data
• Reduce Data Availability Gap and Accelerate Time to Value
• Native Integration with Real-time Streaming Data and Batch
Data
• Enables Real-time Graph Feature Updates with Streaming
Data in Machine Learning Use Cases
• Decrease Learning Curve With Familiar Syntax
• GSQL Support with Consistent Data Loading Syntax
• Maintain Separation of Control for Data Loading
• Designed with Built-in MultiGraph Support
16

17. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Kafka Loader : Three Steps
Consistent with GSQL Data Loading Steps
Step 1: Define the Data Source
Step 2: Create a Loading Job
Step 3: Run the Loading Job
17

18. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Kafka Loader High Level Architecture
● Connect to External Kafka Cluster
● User Commands Through GSQL Server
● Configuration Settings:
○ Config 1: Kakfa Cluster Configuration
○ Config 2: Topic/Partition/Offset Info
18

19. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
DEMO
19

20. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Kafka and tgcloud.io
Data Loading Use Cases
20

21. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
What is TigerGraph Cloud (tgcloud.io)?
● TigerGraph Cloud is a distributed graph Database-as-a-Service
● Cloud Infrastructure Included
● Out-of-box graph starter kits - industry use case library
● Complete operational support by TigerGraph
○ Upgrades
○ Patches
○ Maintenance
○ Status Monitoring
● Flexible and hourly billing rates by credit card and through prepaid bulk
“cloud credits”
● Multiple versions and multiple clusters supported
● Easy provisioning for distributed clusters
● Built in Encryption and Security
21

22. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
TigerGraph Cloud Architecture
TigerGraph Cloud Portal
GraphStudio Admin Portal
GSQL
Web Shell
TigerGraph
Database
+
Graph Starter Kit
GraphStudio Admin Portal
GSQL
Web Shell
TigerGraph
Database
+
Graph Starter Kit
…...
…...
22

23. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Azure Blob + Kafka + tgcloud.io
23
Kafka
Loader
Kafka Connect
Azure Blob Storage
Source Connector
Azure Blob Storage
https://docs.confluent.io/kafka-connect-azure-blob-storage-source/current/index.html

24. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Google Cloud Storage + Kafka + tgcloud.io
24
Kafka
Loader
Kafka Connect
Google Cloud
Storage Source
Connector
https://docs.confluent.io/kafka-connect-gcs-source/current/overview.html

25. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Amazon Kinesis + Kafka + tgcloud.io
25
Kafka
Loader
Kafka Connect
Amazon Kinesis
Source Connector
https://docs.confluent.io/kafka-connect-kinesis/current/index.html

26. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
DEMO
26

27. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Summary
27

28. © 2021. ALL RIGHTS RESERVED. | TIGERGRAPH.COM | CONFIDENTIAL INFORMATION |
Summary
● Graph Analytics Overview
● TigerGraph Data Ingestion System Architecture: Kafka as
an Internal Component
● Built-in Kafka Loader in TigerGraph
● TigerGraph Cloud (tgcloud.io) and Kafka Data Pipeline Use
Cases
28

29. Get Started for Free
● Try TigerGraph Cloud ( tgcloud.io )
● Download TigerGraph’s Developer Edition
● Take a Test Drive - Online Demo
● Get TigerGraph Certified
● Join the Community
@TigerGraphDB /tigergraph /TigerGraphDB /company/TigerGraph
29