分散DB Apache KuduのアーキテクチャDBの性能と一貫性を両立させる仕組み「HybridTime」とは

2 © Cloudera, Inc. All rights reserved.
• ( ) / takahiko at cloudera.com
•
• Cloudera
•
• Internet & Network
• RDBMS 1
• NoSQL 2
• Hadoop 3 ←Now!

• Apache Kudu
• Kudu OLTP OLAP HTAP
DB #dbts2017 Kudu
• BI/DWH DB Kudu
Google Spanner
https://www.slideshare.net/Cloudera_jp/apache-kududb-dbts2017
• HybridTime
DB HybridTime
Kudu

© Cloudera, Inc. All rights reserved.
Apache Kudu

• 275 3PB
• 1000 PB
• /
• 1 GB/
• DB
• BLOB
•
• 1000
Kudu
1
...

Kudu
Kudu
(Impala)
(Kudu) (S3)(HDFS)
(Impala) (Spark)
(Hive)
(MapReduce)
(ADLS)
SQL
( DB )
HMS

SQL Kudu
Impala + Kudu
(Impala)
(Kudu) (S3)(HDFS)
(Impala) (Spark)
(Hive)
(MapReduce)
(ADLS)
HMS
• Kudu SQL
• Impala SQL

• Impala SQL Impala Kudu
• Impala Kudu predicate push down
• Kudu SCAN Impala aggregation
SQL
SQL Impala
3
90

Spark Kudu
Spark + Kudu
(Impala)
(Kudu) (S3)(HDFS)
(Impala) (Spark)
(Hive)
(MapReduce)
(ADLS)
HMS
• Spark SQL
Kudu API
• SparkSQL

• Kudu 1
Kudu
Tablet
Kudu
TabletServer

• 1 3
• 3
• Raft
•
•
Tablet
TabletServer

•
• INSERT/UPDATE/UPSERT/DELETE
• DECIMAL
•
•
• Kerberos
•
•
•
•
Kudu

HTAP: OLTP OLAP Kudu

• OLTP 1TB RAM
•
OLTP OLAP
• 2
DB DB
insert/update/delete
OLTP OLAP DWH BI
select
ETL

Hadoop
• (PB)
HDFS OLAP
• Impala/Hive SQL
• HDFS OLTP HBase
HBase
Hadoop DB
put / delete OLTP OLAP BIselect
HBase ImpalaHadoop
data ingestion
HDFS
ETL

• OLTP? OLAP?
•
•
• OLTP OLAP
• HTAP(Hybrid Transactional/Analytic Processing)
•
• ...
•
• Kudu HTAP
OLTP OLAP
HTAP
HTAP

• OLTP OLAP 1 DB
•
HTAP DB
Kudu
insert/update/delete
HTAP DWH BI
select
Kudu

(HDFS)
SQL
Impala
(Spark Streaming)
(Flume)
ETL SQL
Hive/Spark
DB DB
(Kudu)
IoT
(Flume)
BI
BI
BI
ETL
MQTT
BrokerIoT
BI
DB DB
/
DB DB
(Kafka)
( )

DB

• DB
•
• ! f
•
• 12:30:00 < 12:30:03
• 2 < 3
• Log Sequence Number LSN
• LSN
• DB
• DB LSN
• DB
DB
12:30:00 12:30:03
2 3
! f

• Physical Clock
•
•
•
•
• Logical Clock)
•
• ...
•
•
•
DB
B
A
A B

•
•
•
• +1
- Lamport Clock
2
3
6
24
16
61
54
69
70
12 24 48423630
8 32 40 48
50 703020

•
• +1
•
•
•
- Vector Clock
2
3
{1,0,0}
{1,1,0}
{1,2,0}
{1,2,1} {1,2,2}
{1,4,2}
{1,3,0}
{2,3,0} {3,3,0}
{3,3,3}
{1,5,2}
{5,5,4}
{5,5,2}{4,5,2}

•
•

•
•
•
12:30:00
12:29:59
A B
B
!
f

• Spanner: Google’s Globally Distributed Database
• DB
ACID
• GPS
• TrueTime API
error bound
Google Spanner

• API
• GPS
•
• TrueTime API TT.now() TTinterval
• TT.now()
• Google DC 1 7ms 4ms
Google Spanner TrueTime API
earliest latest
TT"#$%&'(): %(&)"%+$, )($%+$
TT,now()
--

• commit wait
• TrueTime API
• e f
2"
• External Consistency
• f e T $ < T &
Google Spanner commit-wait
$
" & "
2"
&
$
&
2"
2"
& → $

Technical Report: HybridTime - Accessible Global Consistency
with High Clock Uncertainty

• Technical Report: HybridTime - Accessible Global Consistency with High Clock
Uncertainty
•
• Google DC
• HybridTime NTP DB
• Kudu Kudu
HybridTime
•
• 2014 (
)
Kudu

• Google Spanner DC
• Amazon Dynamo Cassandra DB
Eventual Consistency
•
•
•
[ ] DC DB

• Consistency
•
•
• CAP Consistency ACID Consistency/Isolation
• Consistency
• (Anomaly)
• Lost Update, Dirty Read, Non-Repeatable, Phantom Read, Read Skew, Write Skew, etc...
•
• Lost Update SELECT FOR UPDATE
Consistency

• Lamport Clocks Vector Clocks
•
•
•
• RDB Point-in-Time
• Vector Clocks
[ ]

• Spinnaker Paxos
•
•
• Spanner commit-wait
•
• GPS
•
[ ]

• HybridTime
•
•
• Pint-in-time
• Lamport Clock
• HybridTime
• Vector Clocks Lamport Clocks 2
( commit-wait )
[ ] HybridTime
HTC: { , }

•
• NTP
• NTP
• commit-wait
•
• NTP
• commit-wait
•
•
• Kudu DB
HybridTime

• !"# $ i e
• !"'() $ e
• *# $ i e
• 1:
• 2:
[ ] HybridTime

• HybridTime HTC
• (error)
• Spanner TrueTime API
• HybridTime
• NTP
HybridTime

• ntp_adjtime
• timex
• maxerror
HybridTime

• Kudu macOS
• macOS OS
macOS

Kudu

• 1 2
• 2 1
• !" − !$ ...
•
• %$
• & = !" − !$ − %$
NTP
2
1
100*+
160*+
T1
100ms
160ms 160-100 = 60ms
T2
%$
NG

• !"
• RTT: !
#
$
• ! = !" + !$ = '( − '" − ('+ − '$)
#
$
= !" =
-./-0 /(-1/-2)
$
• 3
• 3 = '$ − '" −
-./-0 / -1/-2
$
• 3 =
$ -2/-0
$
−
-./-0 / -1/-2
$
• 3 =
$-2/$-0/-.4-04-1/-2
$
• 3 =
-2/-0/-.4-1
$
• 3 =
-2/-0 4 -1/-.
$
NTP
NTP
T2 T3
T4T1
NTP
10078
15078 16078
11078
16578
11578 12578
17578
7078 8078 8578 9578
+1078 +1078
+578
!" !$
1) 50ms
2) -30ms
RTT20ms

•
•
•
• ! =
#$#%#&& '(#)$%#$*)
)
= 41./
• 50ms -9ms
• RTT
0
)
• ±
0
)
NTP
T2 T3
T4T1
NTP
100./ 101./ 106./ 125./
+1./ +19./
+5./
8# 8)
150./ 151./ 156./ 175./) 50ms
RTT20ms

•
• 1
•
•
•
• NTP !
•
• ! +
#
$
NTP

• NTP
• DC NTP or Google Public NTP
• AWS Amazon Time Sync Service
• Azure Hyper-V time synchronization Google Public NTP
• GCE Google Public NTP
•
NTP

• UPDATE
• 2
• HybridTime
+1
[ ] HybridTime

• !" #, % !" HTC #, %
[ ] HybridTime

• HTC
• HTC
• ) ! → #
• −%& ! < !(()( # < %* #
[ ] Kudu HybridTime
j
i
#
−%& !
%* #−%& !
!
%& !

• KUDU-146 Deal with leap seconds
• leap second
• Stratum 0 NTP Leap Indicator OS
• 23:59:59 -> 23:59:60 -> 00:00:00
• 23:59:59 -> 23:59:59-> 00:00:00
• 2 1
• HybridTime propagate
• Kudu commit-wait
• wait ms
1
• NTP TIME_INS/TIME_OOP max error
• Leap Smearing
https://issues.apache.org/jira/browse/KUDU-430

• HybridTime
•
• Kudu RDB
• ACID
• Commit-wait
• NTP
• CLIENT_PROPAGETED
• HybridTime
• HybridTime propagate
Kudu

• Kudu MVCC Multi-version Concurrency Control
•
• WAL REDO UNDO
•
• READ_LATEST
•
• READ_AT_SNAPSHOT
• MVCC
•
Repeatable Read
Kudu

Kudu
https://blog.cloudera.co.jp/11c3a749a81b

• YCSB
•
• 3 8
• insert 60%, update 20%, single-row read 20%
•
• GCE: nl-standard-8 x10
• RAM 30GB
• Disk 350GB
• NTP
• GCE
[ ]
NTP

[ ]
HybridTime Commit Wait Commit Wait
Clock Error

• OLTP OLAP 1 DB Kudu
• HybridTime
DB
• HybridTime → P.36
• Kudu
HybridTime Serializable
• OLAP Kudu #dbts2017
• 11 6 Cloudera World Tokyo 2018
Kudu
Kudu

Cloudera World Tokyo 2018
http://www.clouderaworldtokyo.com/

分散DB Apache KuduのアーキテクチャDBの性能と一貫性を両立させる仕組み「HybridTime」とは

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 分散DB Apache KuduのアーキテクチャDBの性能と一貫性を両立させる仕組み「HybridTime」とは

Similar to 分散DB Apache KuduのアーキテクチャDBの性能と一貫性を両立させる仕組み「HybridTime」とは (20)

More from Cloudera Japan

More from Cloudera Japan (20)

Recently uploaded

Recently uploaded (20)