Neptune webinar AWS

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
Gianluca Nieri (Solutions Architect)
Amazon Neptune

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
About myself
Gianluca Nieri, Solutions Architect @ AWS
20+ years as solutions architect, data architect;
15+ years in TLC & FSI fields
gianx@amazon.com
@gianx

Agenda
• Introduction to Graph DB concepts;
• Amazon Neptune;
• Working with Neptune;
• Launch and access a Neptune cluster;
• Working with graphs;
• Conclusions;

Introduction to Graph DB
concepts

Introduction - Type of data stores
Amazon Aurora
Amazon RDS
Amazon
DynamoDB
Amazon
ElastiCache
Amazon
Neptune
Amazon
Redshift
Relational NoSql Columnar Key/Value Graph
Tables/Rows JSON Tables/Cols Keys Graphs

WHAT IS A GRAPH DATABASE?
In computing, a graph database is a database that
uses graph structures for semantic
queries with nodes, edges and properties to represent
and store data.
(https://en.wikipedia.org/wiki/Graph_database)

HIGHLY CONNECTED DATA
Social networking Recommendation Fraud detection

OPEN OF GRAPH STANDARDS
( S U P P O R T E D B Y A M A Z O N N E P T U N E )
- Property graph data model and Apache Thinkerpop Gremlin query
language;
- Resource description framework (RDF) data model and SPARQL query
language;
• RDF documents are written in XML in a language called RDF/XML;
• Used in W3C's Semantic Web;
• Use triples: subject – predicate – object;
- In Amazon Neptune label-property graph and RDF data can be stored on
the same cluster but they are separated, and any data that is loaded or
stored with one query language cannot be queried by the other.

PROPERTY GRAPH
( S I M P L I F I C A T I O N )
PERSON
Id: 1
Name: John
Country: UK
Node
PERSON
Id: 2
Name: Mike
Country: UK
PERSON
Id: 3
Name: Luca
Country: IT
LANGUAGE
Id: 101
Name: java
LANGUAGE
Id: 102
Name: c++
LANGUAGE
Id: 103
Name: python
LANGUAGE
Id: 104
Name: php
KNOWS
Id: 201
Level: 3
KNOWS
Id: 202
Level: 2
KNOWS
Id: 203
Level: 4
KNOWS
Id: 204
Level: 2
KNOWS
Id: 205
Level: 5
LEARNING
Id: 206
Label
Properties
Edge

PROPERTY GRAPH – RELATIONAL EQUIVALENT
Person
Id
Name
Country
Language
Id
Name
PersonLanguage
Id
IdP
IdL
Type
Level
Id Name Country
1 John UK
2 Mike UK
3 Luca IT
Id IdL IdP Type Level
201 1 101 knows 3
202 1 102 knows 2
203 1 103 knows 4
204 2 103 knows 2
205 3 103 knows 5
206 3 104 learning
Id Name
101 java
102 c++
103 python
104 php

COMPARISON WITH SQL DATABASES
SELECT language.name
FROM personlanguage JOIN person ON personlanguage.idp = person.id
JOIN language ON personlanguage.idl = language.id
WHERE person.name = 'John';
SQ
L
g.V().hasLabel('person').has('name','John').outE().inV().values('name')
GREMLI
N
Walking,
not
joining!

WHEN TO USE A GRAPH DATABASE
• High values to relationships;
• Many to many relationships;
• Want to find hidden relationships (i.e.: circularity)
• Low latency at large scale;

WHEN NOT TO USE A GRAPH DATABASE
• Operational use case: not efficient for managing high volume of
transactions;
• Datawarehousing;

Amazon Neptune

WHAT IS AMAZON NEPTUNE?
Amazon Neptune is a fast, reliable, fully-managed graph database service
that makes it easy to build and run applications that work with highly connected
datasets.
The core of Amazon Neptune is a purpose-built, high-performance graph
database engine optimized for storing billions of relationships and querying
the graph with milliseconds latency.

FULLY MANAGED
- AWS Management Console / AWS Command Line Interface / AWS APIs;
- Easy to launch standard configurations;
- Database parameter group provide granular control for fine tuning;
- Operational metrics integrated in Amazon CloudWatch;
- Automatic database patching;
- Automatic database backup;
- Automatic database event notification;
- Fast database cloning;
- Fast database bulk load;

PERFORMANCE AND SCALABILITY
- Amazon Neptune is a purpose-built, high-performance graph database;
- Scale-up in memory optimized architecture;
- Automatic storage scaling;
- Up to 64 TB data volumes;
- Low latency read replicas (up to 15);

AVAILABILITY AND DURABILITY
- Neptune always stores two copies of each single data block (10 GB each)
on three Availability Zone in a single AWS Region (total six copies);
- The primary DB instance is synchronously replicated across Availability
Zones to Neptune Replicas;
- If a DB cluster is in a single Availability Zone, you can make it a Multi-AZ
DB cluster adding a Neptune replica in a different Availability Zone (up to
15);
- When failing over, Amazon Neptune promotes a replica to a new primary. If
no replica is available, Amazon Neptune creates a new DB instance.

HIGHLY SECURE
- Fully integrate with IAM for rights management;
- VPC support;
- Encryption at rest with support for KMS;

ENDPOINTS AND CONNECTIONS #1
Cluster endpoint:
- Use the cluster endpoint for all write operations on the DB cluster,
including inserts, updates, deletes, and data definition language (DDL)
changes.;
- If the current primary DB instance of a DB cluster fails, Neptune
automatically switches over to a new primary DB instance;

ENDPOINTS AND CONNECTIONS #2
Reader endpoint:
- A reader endpoint is an endpoint for a Neptune DB cluster that connects to
one of the available Neptune Replicas;
- If there is more than one Neptune Replica, the reader endpoint directs each
connection request to one of the Neptune Replicas;
- Use the reader endpoint for read operations. You can't use the reader
endpoint for write operations.

PRICING
- Instance price (on-demand);
- Database storage (GB/month);
- Database I/O (Per million requests);
- Backup storage (GB/month);
- Data transfer IN and OUT (per GB);
https://aws.amazon.com/neptune/pricing/

Working with Neptune

ACCESSING THE GRAPH(GREMLIN)
- Gremlin console:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-console.html
- Gremlin HTTP and WebSocket API:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-rest.html
- Python:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-python.html
- Java:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-java.html
- .NET
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-dotnet.html
- Node.js
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-node-js.html

ACCESSING THE GRAPH(SPARQL)
- RDF4J Console:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-sparql-rdf4j-console.html
- HTTP REST:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-sparql-http-rest.html
- Java:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-sparql-java.html
- RDF4J Workbench:
https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-sparql-rdf4j-workbench.html

MONITORING NEPTUNE
- Amazon Neptune is integrated with Amazon CloudWatch, so you can
gather and analyze performance metrics using the CloudWatch console, the
AWS Command Line Interface (AWS CLI), or the CloudWatch API;
- Granularity is 5 minutes;
- Some metrics are:
- CPUUtilization, ClusterReplicaLag, EngineUptime, FreeableMemory,
FreeLocaStorage, NetworkThroughput, VolumeBytesUsed, etc.;
- GremlinRequests, GremlinRequestsPerSec, GremlinErrors,
GremlinHttpXXX, etc.;
- SparqlRequests, SparqlPerSec, SparqlErrors, SparqlHttpXXX;
- LoaderRequests, LoaderErrors, etc;
- Amazon Neptune is fully integrated with AWS CloudTrail;

IMPORTING DATA #1
- Different data formats supported (CSV, N-Triplets, N-Quads, RDF/XML and
Turtle);
- All data must be UTF-8 and files could be compressed;
- The process loads data asynchronously from an S3 bucket;
- Neptune needs a IAM role to access S3 bucket;
- Must use loader API;

IMPORTING DATA #2
curl -X POST
-H 'Content-Type: application/json'
http://your-neptune-endpoint:8182/loader -d '
{
"source" : "s3://bucket-name/object-key-name",
"format" : "format",
"iamRoleArn" : "arn:aws:iam::account-id:role/role-name",
"region" : "region",
"failOnError" : "FALSE"
}'

Launch and access a Neptune
Cluster

NOTES
This tutorial is intended for testing
purposes only and not for setting up a
production environment.

EXAMPLE ARCHITECTURE DESIGN
Bastion host
neptuneBastionSG
Neptune Master Neptune Read replica
neptuneClusterSG
Eu-west-1a (public) Eu-west-1a (private) Eu-west-1b (private)
VPC
Eu-west

NETWORK SETUP #1
- You can provision network infrastructure for Neptune using:
- AWS Console;
- AWS API;
- AWS CLI;
- In this tutorial we'll use AWS CLI;
- Install the AWS CLI, refer to the following page:
https://docs.aws.amazon.com/cli/latest/userguide/installing.html

NETWORK SETUP #1
- Create a VPC (write down VpcId):
REGION="eu-west-1"
aws ec2 create-vpc --cidr-block 10.0.0.0/16 --region ${REGION}
aws ec2 create-tags --resources <VpcId> --tags Key=Name,Value="Neptune-webinar"
- Create public and private subnets (write down subnets Ids):
aws ec2 create-subnet --vpc-id <VpcId> --cidr-block 10.0.0.0/20 --availability-zone eu-west-1a
aws ec2 create-subnet --vpc-id <VpcId> --cidr-block 10.0.16.0/20 --availability-zone eu-west-1a
aws ec2 create-subnet --vpc-id <VpcId> --cidr-block 10.0.32.0/20 --availability-zone eu-west-1b

NETWORK SETUP #2
- Create an Internet gateway and a route table to connect to public subnet
(write down InternetGatewayId and RouteTableId)
aws ec2 create-internet-gateway
aws ec2 attach-internet-gateway --vpc-id <VpcId> --internet-gateway-id <InternetGatewayId>
aws ec2 create-route-table --vpc-id <VpcId>
aws ec2 create-route --route-table-id <RouteTableId> --destination-cidr-block 0.0.0.0/0 --gateway-id
<InternetGatewayId>
aws ec2 associate-route-table --subnet-id <PublicSubnetId> --route-table-id <RouteTableId>
aws ec2 modify-subnet-attribute --subnet-id <PublicSubnetId> --map-public-ip-on-launch

NETWORK SETUP #3
- Create a subnet group for Neptune Cluster using previously created subnets
aws rds create-db-subnet-group
--db-subnet-group-name "Neptune Webinar Subnet Group"
--db-subnet-group-description "Private Subnet Group for the Neptune Cluster"
--subnet-ids <PrivateSubnet1Id> <PrivateSubnet2Id>
- Create a security group for bastion host (write down GroupId):
aws ec2 create-security-group --group-name "neptuneBastionSG" --description "Access to Bastion Neptune host on
port TCP 22" --vpc-id <VpcId>
aws ec2 authorize-security-group-ingress --group-id <SG-ID> --protocol tcp --port 22 --cidr 0.0.0.0/0

NETWORK SETUP #4
- Create a security group to connect to Neptune Cluster (write down GroupId)
aws ec2 create-security-group --group-name "neptuneClusterSG" --description "Neptune SG" --vpc-id <VpcID>
aws ec2 authorize-security-group-ingress --group-id <Group2Id> --protocol tcp --port 8182 --source-group
<Group1Id>

LAUNCHING THE CLUSTER #1
- DB engine: choose last version;
- DB instance class: choose the one that best fits your needs;
- Select "Create read replica in different zone");
- DB Instance Identifier: set as you prefer (e.g.: "neptune-webinar-instance");
- NEXT

LAUNCHING THE CLUSTER #2
- Virtual Private Cloud "VPC": choose previously created VPC;
- Subnet Group: choose previously created subnet group;
- Availability zone: choose one of the previously private availability zone (that
are part of the "Subnet Group");
- Choose existing VPC security groups: choose the previously created
security group for Neptune ("neptuneClusterSG");
- DB cluster identifier: set as you prefer (e.g.: "neptune-webinar-cluster);
- Leave other parameters untouched
- CREATE DATABASE

LAUNCHING THE BASTION HOST
- Launch and tag the instance (write down InstanceId):
aws ec2 run-instances --image-id ami-0a5e707736615003c --count 1 --instance-type t2.micro --key-name <Existing
KeyPair> --security-group-ids <neptuneBastionSG id> --subnet-id <public subnet id>
aws ec2 create-tags --resources <InstanceId> --tags Key=Name,Value=neptune-webinar-bastionhost
- Describe the instance and write down PublicIpAddress:
aws ec2 describe-instances --instance-id <InstanceId>
- Connect to the bastion host:
ssh -i <Existing KeyPair file> ec2-user@<PublicIpAddress>

INSTALLING GREMLIN CONSOLE #1
- Connect to the bastion host:
ssh -i <Existing KeyPair file> ec2-user@<PublicIpAddress>
- Update the system and install Java:
sudo yum install java-1.8.0-devel -y
sudo yum install git -y
sudo /usr/sbin/alternatives --config java

- Install Apache Tinkerpop Gremlin console:
cd $HOME/neptune
wget https://archive.apache.org/dist/tinkerpop/3.3.2/apache-tinkerpop-gremlin-console-3.3.2-bin.zip
unzip apache-tinkerpop-gremlin-console-3.3.2-bin.zip
cd apache-tinkerpop-gremlin-console-3.3.2
cat << EOF > conf/neptune-remote.yaml
hosts: [<CLUSTER ENDPOINT>]
port: 8182
serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: {
serializeResultToString: true }}
EOF

- Connecting:
cd $HOME/neptune/apache-tinkerpop-gremlin-console-3.3.2
bin/gremlin.sh
- Connecting to the remote cluster:
gremlin>:remote connect tinkerpop.server conf/neptune-remote.yaml
gremlin>:remote console

Working with data

CONNECTING VIA GREMLIN CONSOLE
- Connecting to the remote cluster:
gremlin>:remote connect tinkerpop.server conf/neptune-remote.yaml
gremlin>:remote console

CREATING NODES
g.addV('person').property(id,'1').property('name','John').property('country','UK')
g.addV('person').property(id,'2').property('name','Mike').property('country','UK')
g.addV('person').property(id,'3').property('name','Luca').property('country','IT')
g.addV('language').property(id,'101').property('name','java')
g.addV('language').property(id,'102').property('name','c++')
g.addV('language').property(id,'103').property('name','python')
g.addV('language').property(id,'104').property('name','php')

CREATING EDGES
g.addE('knows').from(g.V('1')).to(g.V('101')).property(id,'201').property('level', 3)
g.addE('learning').from(g.V('3')).to(g.V('104')).property(id,'206')

SOME EXAMPLE
g.V().label()
g.V().label().dedup()
g.V().values('country')
g.V().values('country').dedup()
g.V().hasLabel('Person')
g.V().has('name','John').values()
g.V().label().groupCount()
g.V().hasLabel('person').has('name','John').outE().hasLabel('knows').inV().hasLabel('language').values('name')

Conclusions

What we learned? #1
- What is a Graph database;
- What is a property graph;
- What are nodes, edges and properties;
- What is a RDF graph;
- What is Apache Tinkerpop Gremlin;

What we learned? #2
- What is Amazon Neptune?
- Fully managed;
- High performance and scalability;
- High availability and durability;
- Pricing.

What we learned? #3
- Example architecture;
- Create network infrastructure;
- Launch a cluster;
- Launch bastion host;
- Install Apache TinkerPop console;
- Connect to the cluster.

Thanks!

Neptune webinar AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Neptune webinar AWS

Similar to Neptune webinar AWS (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Neptune webinar AWS