Building system-of-record applications with Amazon QLDB - SVC218 - New York AWS Summit

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building system-of-record applications
with Amazon QLDB
Michael Labib
Principal SA
AWS
S V C 2 1 8

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
• Blockchain & purpose-built databases at AWS
• Overview of Amazon Quantum Ledger Database (Amazon QLDB) architecture
and features
• Cryptographic verification
• Console demo

What is blockchain?
Consensus algorithms
No intermediaries in
decision process, support
for smart contracts
Immutable, append-only,
cryptographically verifiable
Ledgers Decentralization
Distributed trust and
data replication

Blockchain at AWS
Amazon Managed
Blockchain
Amazon QLDB

Centralized vs. decentralized
• Owned by a single, trusted authority.
• Addresses core need of an immutable and verifiable transactional
log.
• Fast: Doesn’t require consent from members
to commit transactions.
Centralized
• No single owner of the ledger. Joint ownership
by multiple parties.
• Addresses core need of enabling multiple parties to transact
transparently and with trust with each other.
• Removes intermediaries when a group of members needs
to transact. Can make business processes more efficient.
Decentralized

Need for a ledger with centralized trust
TRANSACTIONS WITH DECENTRALIZED
TRUST2
DMV
Track vehicle title history
Manufacturers
Track distribution of a recalled product
HR & payroll
Track changes to an
individual’s profile
Healthcare
Verify and track hospital
equipment inventory
LEDGERS WITH
CENTRALIZED TRUST1

Need for running transactions with decentralized trust
Financial institutions
Peer-to-peer payments
Mortgage
lenders
Process syndicated loans
Supply chain
Transact with suppliers
and distributers
Retail
Streamline customer rewards
TRANSACTIONS WITH DECENTRALIZED
TRUST2
LEDGERS WITH
CENTRALIZED TRUST1

Amazon Managed Blockchain
Easily create and manage scalable blockchain networks
Quickly create blockchain networks that
span multiple AWS accounts. Easily add
or remove members
and monitor the network.
Fully managed Improves reliabilityScalable and secure
Easily scale your blockchain network as
the usage grows. Also, Managed
Blockchain secures your network
certificates with AWS KMS.
Choice of Hyperledger
Fabric or Ethereum
Choose the right framework
for your needs, whether you
are building a permissioned
or public network.
Managed Blockchain improves
the reliability of the “ordering
service,” by replacing the default
technology with Amazon QLDB.
This improves durability.

Purpose-built databases at AWS
Relational
Referential integrity,
ACID transactions,
schema-on-write
Lift and shift, ERP, CRM,
finance
Key-value
High throughput,
low-latency
reads and writes,
endless scale
Real-time bidding,
shopping cart, social,
product catalog,
customer preferences
Document
Store documents
and quickly access
querying on
any attribute
Content management,
personalization, mobile
In-memory
Query by
key with microsecond
latency
Leaderboards,
real-time analytics,
caching
Graph
Quickly and easily
create and navigate
relationships
between data
Fraud detection, social
networking,
recommendation
engine
Time-series
Collect, store, and
process data sequenced
by time
IoT applications, event
tracking
Ledger
Complete, immutable,
and verifiable history
of all changes to
application data
Systems of record,
supply chain,
healthcare,
registrations,
financial

ID Manufacturer Model Year VIN Owner
1 Tesla Model S 2012 123456789 Traci Russell
Traditional database architecture: The log
• Typically an internal implementation
• Used for replicating data
• Difficult, or impossible, to directly access
table
tx1 tx2 tx3 tx4 tx5 tx6 tx7 tx8
log

Amazon QLDB: The journal is the database
• Amazon QLDB’s journal has structural similarity to a database log
• All writes go to the journal—the journal determines state
• Journal handles concurrency, sequencing, cryptographic verifiability, and
availability
• Accessible history of all transactions, document versions, document
metadata
blockAdreess hash data metadata
{strandId:"JpbmngzFZV7
FHjEuuER1Ol",sequence
No:78}
{{XKlKYIzWEyBPRgup1Xf
a/Qp4JE2PEbA8nc0KxlV
Gm8c=}}
{Manufacturer:"Tesla",Model
:"Model
S",Year:"2012",VIN:"123456
789",Owner:"Traci Russell"}
{id:"5PLf8cOOFPoIf7w1NJ
zUXL",version:0,txTime:2
019-06-28,
txId:"3mDCDwAbtYi6vGd
PfUlDGf"}
{strandId:"60bpn7xLtB4
8311uwkihe8",sequence
No:11}
{{ii2h58whRCHk/1zRp4R
LglG9D2SlNDa32rUWZtc
S11E=}}
:"Model
S",Year:"2012",VIN:"1234567
89",Owner:"Traci
Russell",owner:"Ronnie
Nash"}
{id:"Kwo6aQwJ4Dz4D1oy
VqRgxY",version:1,txTime
:2019-07-
04T20:21:22.071Z,txId:"6
BFspx97Mtq4sEid33YkMd
"}
{strandId:"60bpn7xLtB4
8311uwkihe8",sequence
No:13}
{{UdPrq7OTHfiikK9rS8YR
BpjGI0c5Pfl3DreSmQaGrf
c=}}
:"Model
S",Year:"2012",VIN:"1234567
89",Owner:"Traci
Russell",owner:“Robert
Dennison"}
{id:"Kwo6aQwJ4Dz4D1oy
VqRgxY",version:2,txTime
:2019-07-
04T20:24:45.768Z,txId:"2
3khn4h3uvH6i8dwKefLjS"
}
1 Tesla Model S 2012 123456789 Robert Dennison
history() #function to query document history
User #standard user data, the default
tx1 tx2 tx3 tx4 tx5 tx6 tx7 tx8
journal
blockAddress hash data metadata
{strandId:"JpbmngzFZV
7FHjEuuER1Ol",sequenc
eNo:78}
{{XKlKYIzWEyBPRgup1Xf
a/Qp4JE2PEbA8nc0KxlV
Gm8c=}}
{FirstName:“Traci",LastName:
“Russell",DOB:1963-08-
19T00:00:00.000Z,GovId:"LEI
S26LL",GovIdType:"Driver
License"}
{id:"5PLf8cOOFPoIf7w1NJ
zUXL",version:0,txTime:2
019-06-28,
txId:"3mDCDwAbtYi6vGd
PfUlDGf"}
Committed #includes metadata

Amazon QLDB characteristics

WritingReading
Amazon QLDB: The journal is the database
1 Tesla Model S 2012 123456789 Robert Dennison
History/committed
current
INSERT… UPDATE… DELETE… UPDATE… UPDATE… UPDATE…
SEQUENCE
NUMBER: 789
SEQUENCE
NUMBER: 790
SEQUENCE
NUMBER: 791
SEQUENCE
NUMBER: 793
SEQUENCE
NUMBER: 792
SEQUENCE
NUMBER: --
journal
ledger
Application data Amazon QLDB
Writing
ID Version Start Manufacturer Model Year VIN Owner
1 0 7/16/2012 Tesla Model S 2012 123456789 Traci Russell
1 1 8/03/2013 Tesla Model S 2012 123456789 Ronnie Nash
1 2 9/02/2016 Tesla Model S 2012 123456789 Robert Dennison

Transactions (ACID)
Isolation Level Potential Issues
Serializable
Snapshot Isolation
Repeatable read
Read committed
Read uncommitted
-
Potential write skew
Phantom reads
Phantom reads/nonrepeatable reads
Phantom reads/nonrepeatable reads/dirty reads
HIGHESTTOLOWEST
• Amazon QLDB supports the highest level of isolation
• There is no other mode for Amazon QLDB
• There is no risk that you’ll see phantom reads, write skew, dirty reads,
or other issues

Easy to use (SQL)
INSERT INTO cars
{ 'Manufacturer':'Tesla',
'Model':'Model S',
'Year': 2012,
'VIN': 123456789,
'Owner':'Traci Russell'
}
SELECT * FROM cars
UPDATE cars SET owner = 'Ronnie Nash' WHERE VIN = '123456789'

Serverless, scalable, highly available
Region
Availability Zone 1 Availability Zone 2
Journals Journals
Availability Zone 3
Journals

UPDATE… DELETE… UPDATE… UPDATE… UPDATE…
Immutable
INSERT…
SEQUENCE
NUMBER: 789
SEQUENCE
NUMBER: 790
SEQUENCE
NUMBER: 791
SEQUENCE
NUMBER: 793
SEQUENCE
NUMBER: 792
SEQUENCE
NUMBER: --
• The journal is append-only and sequenced
• There is no API or other method to alter committed data
• All operations, including deletes, are written to the journal

Cryptographic verification
Entries
Block
QLDB SQL
Metadata
journal
Record
hash
Digest

Amazon QLDB summary
Journal-first
The journal is the database
ACID transactions
Fully serializable isolation
Easy to use
Familiar SQL operators
Highly scalable
Serverless, highly available
Immutable
Append-only, sequenced
Cryptographically verifiable
Hash-chaining provide data integrity

Deeper look at concurrency control, isolation
Optimistic (CQL)
• Favors short-running transactions
• Encourages “hygiene” by requiring programmer
to carefully consider
read patterns
Thread 1
SELECT
COMMIT.. IF..
Thread 2
SELECT
COMMIT.. IF..
Pessimistic (SQL)
• Favors long-running transactions
• Easier to “over-include” data
in read operations
Thread 1
SELECT FOR
UPDATE
COMMIT
Thread 2
SELECT FOR
UPDATE
COMMIT

Amazon QLDB’s data model: Ion
vehicle = {
‘VIN’ : “KM8SRDHF6EU074761”,
‘MfgDate’ : “2017-03-01”
‘Type’: “Truck”
‘Mfgr’: “Ford”
‘Model’: “F150”
‘Color”: “Black”
‘Specs’: {
‘EngSize’ : 3.3
‘CurbWeight’: 4878
‘HP’: 327
‘BatterySize’: Null
}
}
JSON document
/* Ion supports comments. */
vehicle = {
‘MfgDate’: 2017-03-01T
‘Specs’: {
‘EngSize’ : 3.3 (decimal)
‘CurbWeight’: 4878 (int)
‘HP’: 327 (int)
‘BatterySize’: NULL.int
}
}
Ion document
https://github.com/amzn/ion-java

Amazon QLDB’s data model: Query
vehicle = {
‘MfgDate’ : 2017-03-01T // timestamp
‘Specs’: {
‘EngSize’ : 3.3
‘CurbWeight’: 4,878
‘HP’: 327
‘BatterySize’ : NULL // null values
}
}
SELECT
VIN,
Specs.EngSize,
Specs.HP
FROM vehicles as v
WHERE v.type = ‘Truck’
VIN Specs.EngSize Specs.HP
KM8SRDHF6EU074761 3.3 327
3HGGK5G53FM761765 2.7 285

Assume three tables
Amazon QLDB’s data model: Ecommerce data model using
Ion
ProductsCustomersOrders
CREATE TABLE Orders CREATE TABLE Customers CREATE TABLE Products

ProductsCustomers
How best to model this?
Ledger: Order-system
INSERT INTO customers
{
'customer-id': 1000,
'first-name': 'Mike',
'last-name': 'Labib',
'membership': true,
'address': ‘126 Brampton Lane’
'city': ‘Chicago',
'state': 'IL'
}
INSERT INTO products
{
'product-id': 346211,
'product-description': 'socks',
'product-color': 'blue',
'price': 5.00,
'active': true,
'external-sku': 'Ak3234211'
}
• Flexible document schema leveraging Amazon Ion

INSERT INTO orders
{
'order-id' : 100056,
'customer' : {
'customer-id': 1000,
'first-name' : 'Mike',
'last-name' : 'Labib',
'address' : ‘126 Brampton Lane',
'city' : ‘Chicago',
'state' : 'IL'
},
'order-date' : 2019-04-30T,
'order-details' : {
'item' : {
'product-id' : 346211 ,
'product-description' : '3 pair socks',
'product-color' : 'blue',
'price' : 15.00,
'quantity' : 2
}
},
'total' : 55.00
}
Nested document structure enables
optimal queries and data access
Order
Products
Customers

SELECT o.order-details from orders o
WHERE o.customer.customer-id = 1000
AND o.order-id = 100056
{ item:
{'product-id': 346211,
'product-description': ‘3 pair socks’,
'product-color': ‘blue’,
‘price’: 15.00,
‘quantity’: 2
}
}
Query Result
Nested document query
(customer within orders) Products
Customers Orders

Mapping constructs between RDBMS & Amazon QLDB
Table
Relational
Table
Amazon QLDB
Table row Amazon Ion document
Column
Document
attribute
Index Index
SQL QLDB SQL
Audit logs Journal
Database Ledger

Deeper look at cryptographic verifiability
Four basic steps to seeing how Amazon QLDB’s verifiability works
Proof: A chain of hashes that links
a document to its digest
a4e31e36910d99bd19b7f875f0
a04597dc0ff52c2f164a16a9288
aed9e710fdd
d07fc3d67314905dd065d55988
790070410e87072f27ce2d1cb5
6457c0879bc8
2fc7e994c884bd13d5fd22b742
5328d0e5d5b0cdcba4d285b19
8be612f229ccb
Digest: Periodic hash covering all history
SHA256: Unique signature of a document
a4e31e36910d99bd19b7f875f0
a04597dc0ff52c2f164a16a9288
aed9e710fdd
Merkle trees: Chaining past hashes together
MERKLE ROOT
HABCD
Hash(HAB+HCD)
HAB
Hash(HA+HB)
HCD
Hash(HC+HD)
HA
Hash(TxA)
HB
Hash(TxB)
HC
Hash(TxC)
HD
Hash(TxD)

Cryptographic verifiability: SHA-256
Amazon QLDB uses the SHA-256 algorithm to create unique, fixed-length outputs (hashes).
Change any part, even one character, and the output (hash) is different.
vehicle = {
‘Specs’: {
‘EngSize’ : 3.3
‘HP’: 327
}
}
vehicle = {
‘Specs’: {
‘EngSize’ : 3.3
‘HP’: 327
}
}
SHA-256
SHA-256
a4e31e36910d99bd19b7f875f
0a04597dc0ff52c2f164a16a92
88aed9e710fdd
19318457408920af2d2cbeacd
90c7afe0fbd7f6ff316972c8f65
6c8bbc402dd1

a4e31e36910d99bd19b7f875f
0a04597dc0ff52c2f164a16a92
88aed9e710fdd
vehicle = {
‘Specs’: {
‘EngSize’ : 3.3
‘HP’: 327
}
}
Cryptographic verifiability: SHA-256
SHA-256 is one way. It is unfeasible to compute the input given an output.
SHA-256
SHA-256
19318457408920af2d2cbeacd
90c7afe0fbd7f6ff316972c8f65
6c8bbc402dd1

Cryptographic verifiability: The digest
Thedigestisyourledger’sMerkle
treerootatapointintime
Doc

How it works
cars.history
H
cars
C
J
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Owner: Traci Russell
Metadata: {
Date:07/16/2012
}
H (T1)
INSERT INTO cars <<
'Model':'Model S',
'Year':'2012',
'VIN':'123456789',
'Owner':'Traci Russell' }
>>
journal

How it works
cars.history
H
cars
C
J
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
H (T1)
INSERT INTO cars <<
'Model':'Model S',
'Year':'2012',
'VIN':'123456789',
'Owner':'Traci Russell' }
>>
journal

How it works
cars.history
H
cars
C
J
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
H (T1)
journal
UPDATE cars SET owner = 'Ronnie Nash' WHERE
VIN = '123456789'
UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H (T2)
Ronnie Nash

How it works
cars.history
H
cars
C
J
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
H (T1)
journal
1 Tesla Model S 2012 123456789 Ronnie Nash
UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H (T2)
DELETE FROM cars WHERE VIN = '123456789'
DELETE cars
ID:1
Metadata: {
Date: 09/02/2016
}
H (T3)
1 3 9/02/2016 Deleted

Walk through a hash chain
J
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
H(T1)
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
SHA-256
H(T1) =
2526f16306c819d651af075934170d2430d246d9ab98d975d28a83baded47ca7

Hashing and chaining transactions
J
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
H(T1)
SHA-256
H(T1) = 2526f16306c819d651af075934170d2430d246d9ab98d975d28a83baded47ca7
UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H(T2)
UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H(T2) =
86a90e4166453d9423b84d47dcbd97c0e3099b1a1f0d7cfca6c191d8fd8994ff
H(T1) +

J
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
H(T1) UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H(T2)
H(T2) = 86a90e4166453d9423b84d47dcbd97c0e3099b1a1f0d7cfca6c191d8fd8994ff
DELETE cars
ID:1
Metadata: {
Date: 09/02/2016
}
H(T3)

J
H(T1) H(T2)INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H(T3)DELETE cars
ID:1
Metadata: {
Date: 09/02/2016
}
H(T3) = ae2d64e562ec754ec3194c744eec72c9fdafffc6b559e0414d0e75bf96ca92ad

A digest is a hash value at a point in time
J
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Metadata: {
Date:07/16/2012
}
UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H(T3)DELETE cars
ID:1
Metadata: {
Date: 09/02/2016
}

H(T3) = c6268578a24dbe0c7cfba07bd967411a35462b8c875d42f1991faad02c0ac93c
H(T2) = a90a9898c7e4b1aab19c705b554afd9e0bf6539bb0346df19be362ff63001098
H(T1) =
25d0b44e6e8878151646ffc1fea4eb85c3e4bf4baec212a9fcf67b6d5a81e01a
UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
DELETE cars
ID:1
Metadata: {
Date: 09/02/2016
}
Changing committed data breaks the chain
J
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Owner: Tracy Russell
Metadata: {
Date:07/16/2012
}
H(T3)

Challenges customers face
Building ledgers with traditional databases
Blockchain approaches
Adds unnecessary
complexity
Designed for a different
purpose
Error-prone and
incomplete
Impossible
to verify
Resource
intensive
Difficult to
manage and scale

Why does immutability and verifiability matter?
Reduce risk: Ensures safeguarding of critical system-of-record applications where data loss can be
expensive
Improve data tracking: Helps you or any parties that have access to the system to quickly and
accurately track data’s entire lineage, improving efficiency in tracking the source of issues (e.g.,
manufacturing defects, maintain supply network data hygiene)
Auditability: Helps reduce downtime caused due to audit and compliance issues, saving hundreds
of productivity hours for your team
Reduce implementation effort: Building immutability and verifiability in a traditional way is time
consuming, complex, and expensive

Building system-of-record applications with Amazon QLDB - SVC218 - New York AWS Summit

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building system-of-record applications with Amazon QLDB - SVC218 - New York AWS Summit

Similar to Building system-of-record applications with Amazon QLDB - SVC218 - New York AWS Summit (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Building system-of-record applications with Amazon QLDB - SVC218 - New York AWS Summit