[db tech showcase Tokyo 2014] D36: 次世代分析基盤 "Data Lake" を支えるPivotalのインメモリ+SQL on Hadoopテクノロジー by Pivotalジャパン株式会社市村友寛 & 宵勇樹

ḟୡ௦ศᯒᇶ┙㻌䇿Data Lake” 䜢ᨭ䛘䜛
Pivotal䛾䜲䞁䝯䝰䝸 + SQL on Hadoop䝔䜽䝜䝻䝆䞊
2014ᖺ11᭶13᪥
Pivotal䝆䝱䝟䞁ᰴᘧ఍♫
ᕷᮧ㻌཭ᐶ
ᐘ䚷ຬᶞ
© Copyright 2014 Pivotal. All rights reserved. 1

Pivotal ఍♫ᴫせ
䜶䞁䝍䞊䝥䝷䜲䝈ྥ䛡䛻3rd䝥䝷䝑䝖䝣䜷䞊䝮䜢ᐇ⌧䛩䜛䝋䝣䝖䜴䜵䜰䜢㛤Ⓨ䞉ᥦ౪
䜽䝷䜴䝗(PaaS)䛸䝡䝑䜾䝕䞊䝍䛾ᇶ┙ᢏ⾡䚸ཬ䜃ḟୡ௦䜰䝥䝸㛤Ⓨ䝃䞊䝡䝇䛜୺㍈
CEO 䝫䞊䝹䞉䝬䝸䝑䝒
2013ᖺ4᭶タ❧
(᪥ᮏἲே䠖7᭶)
ᚑᴗဨᩘ
⣙2,000ே
௻ᴗ㢳ᐈ
1,200♫௨ୖ
ฟ㈨௻ᴗ
EMC㐃ྜ䛸GE㐃ᦠ

Pivotal’s Opportunity
§ 䝡䝑䜾䝕䞊䝍
Pivotal HD, Pivotal Greenplum DB
§ 䝣䜯䝇䝖䝕䞊䝍
Pivotal GemFire
§ 䜶䞁䝍䞊䝥䝷䜲䝈㻌PaaS
Pivotal CF
§ 䜰䝆䝱䜲䝹㛤Ⓨᨭ᥼䝃䞊䝡䝇
Pivotal Labs
§ 䝕䞊䝍䝃䜲䜶䞁䝔䜱䝇䝖ཬ䜃
䚷䚷⫱ᡂ䝖䝺䞊䝙䞁䜾
Pivotal Data Science Labs

䝡䝑䜾䝕䞊䝍᫬௦䛻ồ䜑䜙䜜䜛せ⣲ᢏ⾡
吨听吀䞊
吵呉吐
SQL on Hadoop
HAWQ
Impala, Drill, Presto,..
Hadoop
Pivotal HD
CDH, MapR, Horton
䝕䞊䝍㔞
ศᩓᆺRDB
GreenplumDB
PureData, Teradata, ExaData
䜲䞁䝯䝰䝸
GemFire/XD
TimesTen, SAP HANA
RDB
Oracle, DB2, MSSQL Server
MySQL, PostgreSQL

Pivotal Data Lake 䜰䞊䜻䝔䜽䝏䝱
Ÿ 䝕䞊䝍ฎ⌮ᇶ┙䛾ᇶ┙せ⣲䛸䛺䜛Hadoop(HDFS)䛻䝕䞊䝍䜢⵳✚
Ÿ 䛥䜎䛦䜎䛺䝕䞊䝍䞉せ௳䛻ᛂ䛨䛶ฎ⌮䜶䞁䝆䞁䜢౑䛔ศ䛡䜛
Pivotal Data Lake
䜰䝘䝸䝔䜱䝑䜽
䝕䞊䝍䝬䞊䝖
SQLฎ⌮
䜸䝨䝺䞊䝅䝵䝘䝹
䜲䞁䝔䝸䝆䜵䞁䝇
䜲䞁䞉䝯䝰䝸㻌䝕䞊䝍䝧䞊䝇
䝷䞁䞉䝍䜲䝮
䜰䝥䝸䜿䞊䝆䝵䞁
HDFS
䝕䞊䝍
䝇䝔䞊䝆䞁䜾
䝕䞊䝍⟶⌮
䝇䝖䝸䞊䝮
䜲䞁䝆䜵䝇䝏䝵䞁
䝇䝖䝸䞊䝭䞁䜾ฎ⌮
New Data-fabrics
Software-Defined Datacenter
GemFire
XD
...ETC
䜲䞁䞉䝯䝰䝸㻌䜾䝸䝑䝗
GemFire
XD

Pivotal Data Lake 䝸䝣䜯䝺䞁䝇䜰䞊䜻䝔䜽䝏䝱
3. SQL䛻䜒ᑐᛂ䛧䛯㧗䛔㛤Ⓨ⏕⏘ᛶ
S
Q
L
䜲䞁䝯䝰䝸䞊䞉䜶䞁䝆䞁
䝕䞊䝍䜴䜵䜰䝝䜴䝇
䝉䞁䝖䝷䝹DWH 䝕䞊䝍䝬䞊䝖
䝡䝆䝛䝇䚷䚷䚷
䜰䝥䝸䜿䞊䝅䝵䞁
BI
䝡䝆䝛䝇
䜰䝘䝸䝔䜱䜽䝇
䝣䜯䝇䝖䝕䞊䝍
ᵓ㐀໬䝕䞊䝍
຺ᐃ⣔
䝅䝇䝔䝮
᝟ሗ⣔
䝅䝇䝔䝮
࿘㎶
䝅䝇䝔䝮
⤒Ⴀ⪅
⟶⌮⪅
ᴗົ㒊㛛
ศᯒ⪅䞉᝟ሗ
䝅䝇䝔䝮㒊㛛
㠀ᵓ㐀໬䝕䞊䝍
䝕䞊䝍䝺䜲䜽
Hadoop
⏕䝕䞊䝍
ETL
ฎ⌮
䜰䜽䝉䝇䝻䜾䝯䞊䝹䞉㼃㼑㼎㻹㻞㻹 ⏬ീ䞉ᫎീ 㡢ኌ 㻿㻺㻿

㻳㻱䛾䜲䝜䝧䞊䝅䝵䞁㻌Industrial Internet

඲䛶䛾஦ᴗ㒊㛛䜢䜎䛯䛜䛳䛯䝕䞊䝍ศᯒᇶ┙
(Industrial Data Lake)䜢Pivotal♫䛾䝔䜽䝜䝻
䝆䞊䛷ᐇ⌧
2014ᖺ8᭶15᪥㻌᪥⤒⏘ᴗ᪂⪺
25のエアライン
340万フライト
340TBデータ
2,000倍の分析速度向上
10分の1の費用削減
アプリ投入まで7日間

㢼ຊⓎ㟁䝅䝇䝔䝮䛾᭱㐺໬(⡿ᅜ䠅
⥅⥆ⓗ䛺䝸䜰䝹䝍䜲䝮ศᯒ
Fast Data
Big Data
⛊㛫䛒䛯䜚ᩘ༓䝕䞊䝍䝫䜲䞁䝖䛛䜙䛾
䝕䞊䝍ᢞධ
䝏䝳䞊䝙䞁䜾 ᭱㐺໬䚸
⥭ᛴ೵Ṇ
┘ど䛸ไᚚ
ಖᏲసᴗ䛾ຠ⋡໬ ㄢ㔠

ḟୡ௦䝡䝑䜽䝕䞊䝍ᇶ┙䜢ᨭ䛘䜛䝋䝣䝖䜴䜵䜰⩌
䝸䜰䝹䝍䜲䝮ศᯒ Pivotal GemFire Pivotal GemFire XD
䝝䜲䝟䝣䜷䞊䝬䞁䝇
ศᯒ
䝺䜼䝳䝷䞊䝟䝣䜷䞊
䝬䞁䝇䠃䝞䝑䝏
ศᯒ
Pivotal Greenplum DB HAWQ
Pivotal HD
䜲䞁䝯䝰䝸䞊
KVS
㉸୪ิฎ⌮
RDB
Hadoop

Pivotal HD
Data Lake 䝥䝷䝑䝖䝣䜷䞊䝮䛾䝁䜰䝔䜽䝜䝻䝆䞊
• Apache䝧䞊䝇䛾Hadoop䛻䜶䞁䝍䞊䝥䝷䜲䝈ᶵ⬟䜢㏣ຍ䞉䜸䞊䝥䞁䛸ၟ⏝୧᪉䛾䝯䝸䝑䝖䜢ாཷ
• ANSI SQL‽ᣐ䛻䜘䜛㧗㏿SQL䜶䞁䝆䞁䛻䜘䜛᪤Ꮡ㈨⏘㻌(䝥䝻䜾䝷䝮䜔䝇䜻䝹) 䛾ಖㆤ
Pivotal䛾Hadoop䝕䜱䝇䝖䝸䝡䝳䞊䝅䝵䞁
• Apache Hadoop2.2䝧䞊䝇
• ၟ⏝〇ရ䛸䛧䛶䛾㏣ຍᶵ⬟
• Command Center, HVE
• HAWQ, GemFireXD
• 䜸䞊䝥䞁䝋䞊䝇䛸䛾㐃ᦠ䞉⤫ྜ
• Spark, Parquet, GraphLab➼
㧗㏿䛺SQL䜶䞁䝆䞁䛾ᦚ㍕
• ᶆ‽SQLᑐᛂ䛾DB䜶䞁䝆䞁
• HIVEẚᩘ༑ಸࠥᩘⓒಸ䛾㧗ᛶ⬟
VM䞉EMC䝔䜽䝜䝻䝆䞊䛸䛾㐃ᦠ
• VMwareୖ䛷䛾᭱㐺໬ᶵ⬟ HVE
• EMC䛾䝇䜿䞊䝹䜰䜴䝖NAS
䚷㻌”Isilon”䛸䛾㐃ᦠ
HAWQ
䜰䝗䝞䞁䝇䝗䝕䞊䝍䝧䞊䝇䝃䞊䝡䝇
ANSI SQL + 䜰䝘䝸䝔䜱䜽䝇
MADlib
䜹䝍䝻䜾
䝃䞊䝡䝇
䝎䜲䝘䝭䝑䜽䞉䝟䜲䝥䝷䜲䝙䞁䜾
HDFS
HBase
Pig, Hive,
Mahout
Map
Reduce
Xtension
䝣䝺䞊䝮䝽䞊䜽
Hadoop 䝞䞊䝏䝱䝷䜲䝊䞊䝅䝵䞁
(HVE)
䜸䝥䝔䜱䝬䜲
䝄
(Orca)
Spring
Parquet
䈜3
GraphLab,
OpenMPI
䈜3
Sqoop Flume
䝸䝋䞊䝇⟶⌮
䝽䞊䜽䝣䝻
䞊
YARN
ZooKeeper
Oozie
Apache Pivotal HD ㏣ຍᶵ⬟
Pivotal
Command
Center
ᵓᡂ/䝕䝥䝻䜲/
┘ど/⟶⌮
Spark
䈜1
Ambari
䈜2
䈜1. Pivotal HD䛿 Apache Spark 䛸✌ാ䛩䜛䛣䛸䜢ㄆᐃ䛩䜛 “Certified Spark Distribution”䛻Ⓩ㘓䛥䜜䛶䛔䜎䛩䚹(2014/5)
䈜2. Pivotal䛿 Apache Ambari䛾㛤Ⓨ䝥䝻䝆䜵䜽䝖䛻ཧ⏬䛩䜛䛣䛸䜢⾲᫂䛧䛶䛔䜎䛩䚹(2014/7)
䈜3. PivotalHD2.0 䛻䛶GraphLab, OpenMPI, Parquet䛿䝧䞊䝍ᥦ౪䛥䜜䛶䛔䜎䛩䚹

Pivotal GemFire XD
ప㐜ᘏ䚸䝇䜿䞊䝹䜰䜴䝖䜢ᐇ⌧
䝕䞊䝍䜢฼⏝䛩䜛䜶䞁䝍䞊䝥䝷䜲䝈䜰䝥䝸䜿䞊䝅䝵䞁
ಙ㢗ᛶ䛾㧗䛔䜲䝧䞁䝖㏻▱ᶵ⬟ SQL(JDBC/ODBC) ୪ิฎ⌮
㧗䛔䝇䝹䞊䝥䝑䝖 ప㐜ᘏ ඃ䜜䛯䝇䜿䞊䝷䝡䝸䝔䜱 ⥅⥆ⓗ䛺ྍ⏝ᛶ
GemFire XD 䝕䞊䝍䜾䝸䝑䝗
䝕䞊䝍䛾ᣢ⥆ᛶ
WAN ⤒⏤䛾ศᩓ
HDFS 䛭䛾௚䝕䞊䝍䝇䝖䜰
䝣䜯䜲䝹䝅䝇䝔䝮䝕䞊䝍䝧䞊䝇 እ㒊䝕䞊䝍䝇䝖䜰

GemFire XD - ప㐜ᘏ䚸䝇䜿䞊䝹䜰䜴䝖䜢ᐇ⌧

䜲䞁䝯䝰䝸䝕䞊䝍䜾䝸䝑䝗䛾ᴫせ

䜲䞁䝯䝰䝸䝕䞊䝍䜾䝸䝑䝗䛾ᇶᮏⓗ䛺ᴫᛕ
Handle thousands of concurrent connections
Replicated
Region
Synchronous replication for
slow changing data
Partition for large data or highly transactional data
• 䜻䞊䝞䝸䝳䞊/䜸䝤䝆䜵䜽䝖䚸SQL䜰䜽䝉䝇
• 䜽䜶䝸䞊䚸䜲䞁䝕䝑䜽䝇䚸䝖䝷䞁䝄䜽䝅䝵䞁ᑐᛂ
• 䝯䝰䝸ୖ䛷䛾䝺䝥䝸䜿䞊䝅䝵䞁䚸䝟䞊䝔䜱䝅䝵䞁
• 䜽䝷䝇䝍㛫䛷䛾෕㛗䝕䞊䝍䛾䝁䝢䞊
• 䝕䜱䝇䜽䜒䛧䛟䛿RDBMS䜈䛾䝕䞊䝍ᒎ㛤
Redundant copy
Partitioned Region
• ศᩓᆺ䝯䝰䝸ᣦྥ䛾䝕䞊䝍䝇䝖䜰
• 」ᩘ䛾䝇䝖䝺䞊䝆䝰䝕䝹
14
Low latency for
thousands of
clients
• 䜰䝥䝸䜿䞊䝅䝵䞁䝻䝆䝑䜽䛾୪ิ໬
• 」ᩘ䛾㞀ᐖ᳨▱ฎ⌮
• ືⓗ䛺䝯䞁䝞䞊䛾㏣ຍ (elastic)
• 䝧䞁䝎䞊䛾ᕪู໬せ⣲
• SQLᑐᛂ䚸WANᑐᛂ, 䜲䝧䞁䝖ฎ⌮, etc

ᚑ᮶䛾㻾㻰㻮㻹㻿䛻䛚䛡䜛ㄢ㢟
䝞䝑䝣䜯䛿୺䛻䈊
㻵㻛㻻ྥ䛡䛻᭱㐺໬
ḟ䛻䝕䞊䝍䞉䝣䜯䜲䝹䛻䈊
᭩㎸䜐
§ ኱㔞䛾I/O
§ タィ᫬䛾᝿ᐃ䛸䛾䜼䝱䝑䝥
• ACID䛻䝣䜷䞊䜹䝇
• 䝕䜱䝇䜽ྠᮇ䛾䝪䝖䝹䝛䝑䜽
ึ䜑䛻㻸㻻㻳䜢䈊
᭩㎸䜐

GemFire XD/SQLFire䛾䝧䞁䝏䝬䞊䜽⤖ᯝ
㻝㻘㻞㻜㻜㻗ྠ᫬䜰䜽䝉䝇
㻣㻜୓㻗䛾䝇䝹䞊䝥䝑䝖

RDBMS䛸䛾㐃ᦠ㻌䠖㻌DBSynchronizer
• RDBMS䛸䝔䞊䝤䝹ᐃ⩏䛜ྠ䛨ሙྜ䛻᭷ຠ
Flights
Flights
FlightsR
FlightsR
FlightAvailability
FlightAvailability
FlightAvailabilityR
FlightAvailabilityR
Airlines Airlines
java.sql.Connection conn = getConnection();
CallableStatement cs = conn.prepareCall(“CALL
SYS.ADD_ASYNC_EVENT_LISTENER(?,?,?,?,?,?,?,?,?,?,?)”);
cs.setString(1, “SG1”);
cs.setString(2, “MyID”);
cs.setString(3, “com.vmware.sqlfire.callbacks.DBSynchronizer”);
cs.setString(11,“jdbc:oracle:thin:@localhost:1521:XE”);
cs.execute();

䝕䞊䝍䝉䞁䝍䞊㛫䛾䝕䞊䝍䞉䝺䝥䝸䜿䞊䝅䝵䞁
GemFire
1
GemFire2
Standby
Gateway
GemFire4
Gateway
GemFire
3
New York Site
GemFire5
Standby
Gateway
GemFire6
GemFire
7
GemFire8
Gateway
Tokyo Site
Standby Gateway Paths
GemFire9
Gateway
GemFire12
Standby
Gateway
GemFire GemFire 10
11
London Site
Primary Gateway Paths
䜾䝻䞊䝞䝹䛺䝕䞊䝍㓄ಙ
䜰䜽䝔䜱䝤-䜰䜽䝔䜱䝤䛺㻌DR ᑐ⟇
ᶆ‽ᶵ⬟䛸䛧䛶ᥦ౪

‘Shared nothing persistence’䛺䜰䞊䜻䝔䜽䝏䝱䛾฼Ⅼ䛸ㄢ㢟
þ ㏣グ䛾䜏䛾䜸䝨䝺䞊䝅䝵䞁䝻䜾
þ ᏶඲䛺୪ิฎ⌮
þ 䝕䜱䝇䜽䝅䞊䜽䛺䛧
¨ 䝻䜾䛾䝇䜻䝱䞁䛻䛿䜽䝷䝇䝍䝉䝑䝖䛾
෌㉳ື䛜ᚲせ
¨ ኱ᐜ㔞䛾䝪䝸䝳䞊䝮䛻ᑐ䛧䛶䛿ㄪᩚ䛜
ᚲせ
Memory
Tables
Compressor
OS Buffers
Append only
Operation logs
LOG
Record1
Record2
Record3
Record1
Record2
Record3
Memory
Tables
Compressor
OS Buffers
Append only
Operation logs
LOG
Record1
Record2
Record3
Record1
Record2
Record3
䜲䞁䝯䝰䝸䝕䞊䝍䜾䝸䝑䝗䛷䛒䜛
GemFire䛻䛚䛔䛶䜒ㄢ㢟䛜Ꮡᅾ

Hadoop HDFS
Hadoop core(HDFS) for scalable, parallel storage
• ᡂ⇍䛧䛴䛴䛒䜚䚸㐺⏝⠊ᅖ䜒ᣑ኱
• ỗ⏝䝃䞊䝞䛷䜒኱つᶍ䛺䝕䞊䝍䝉䝑䝖䜢ྲྀ䜚ᢅ䛔䛜ྍ⬟
• 㞀ᐖ䜈䛾ᰂ㌾䛺ᑐᛂ
• 䝅䞁䝥䝹䛺୍㈏ᛶ䝰䝕䝹

Hadoop 䜶䝁䝅䝇䝔䝮䛜ᐇ⌧䛩䜛฼Ⅼ
Ÿ ኱つᶍ䛺䝪䝸䝳䞊䝮䝉䝑䝖 ( TB to PB)
Ÿ 㧗ྍ⏝ᛶ, ᅽ⦰ᶵ⬟
Ÿ ୪ิィ⟬䛸䝕䞊䝍ศᯒᇶ┙䛸䛧䛶䛾ᡂ⇍ᗘ䛸䜶䝁䝅䝇䝔䝮䛾
ᒎ㛤
Ÿ 䝇䝖䝺䞊䝆䝅䝇䝔䝮䛻䛚䛔䛶䜒HDFSᑐᛂ䛜ᬑཬ
Ÿ ௬᝿໬⎔ቃ䜈䛾ᑐᛂ䜒ᚎ䚻䛻ᾐ㏱

GemFire XD
䜲䞁䝯䝰䝸䝕䞊䝍䜾䝸䝑䝗䛛䜙䛾᪂䛯䛺ᒎ㛤

GemFire XD – 䝸䜰䝹䝍䜲䝮䝕䞊䝍ศᯒ䜶䞁䝆䞁
䜲䞁䝯䝰䝸䚸䛥䜙䛻䛿ᆅ⌮ⓗ䛻
ศᩓ䛥䜜䛯䝽䞊䜻䞁䜾䝉䝑䝖
SQLFire
㐣ཤ䝕䞊䝍䚸᫬⣔ิ䝕䞊䝍䜢
HDFS䛻᱁⣡
Pivotal HD
GemFire
+
䜽䝷䝇䝍䝸䞁䜾䚸䜲䞁䝯䝰䝸
䝕䞊䝍䝇䝖䜰䚸HA, 䝺䝥䝸䜿䞊
䝅䝵䞁䚸WANᑐᛂ䚸䜲䝧䞁䝖ฎ
⌮䚸ศᩓ䜻䝳䞊…
SQL
Objects,
JSON
SQL 䜶䞁䝆䞁
- 䜸䝥䝔䜱䝬䜲䝄䚸䜲䞁䝯䝰䝸
ୖ䛾䜲䞁䝕䝑䜽䝇సᡂ䚸ศᩓ
䝖䝷䞁䝄䜽䝅䝵䞁䚸RDB䜲䞁䝔
䜾䝺䞊䝅䝵䞁..
䜲䞁䝇䝖䞊䝹䚸ᵓᡂ䚸⟶⌮䛾⤫
ྜ໬䚸┘どᶵ⬟䚸Hadoop䜈䛾
᭱㐺໬

GemFire XD 䜰䞊䜻䝔䜽䝏䝱ᴫせ
JDBC ODBC
HDD HDD HDD HDD HDD
䝕䞊䝍䝜䞊䝗
HDFS
䝕䞊䝍䝜䞊䝗
HDFS
䝕䞊䝍䝜䞊䝗
HDFS
䝕䞊䝍䝜䞊䝗
HDFS
䝕䞊䝍䝜䞊䝗
HDFS
GemFire XD
䜽䝷䝇䝍
Map/Reduce, Hive, HBase GemFire XD PXF Plugin
Hadoop
䜽䝷䝇䝍
䝕䞊䝍䝧䞊䝇
䝕䞊䝍䝉䞁䝍䞊

SQL + IMDG(Objects) + HDFS
䝯䜲䞁䝯䝰䝸䞊䜢䝧䞊䝇䛻㻌ప㐜ᘏ䛷ศᩓ䛧䛯䝡䝑䜾䝕䞊䝍ྥ䛡䛾䝕䞊䝍䝇䝖䜰
ᐇ㝿䛻᧯స䛩䜛
䝕䞊䝍䛾䜏䚸䝯䝰䝸
ୖ䛻ᒎ㛤
Ọ⥆໬䚸䛒䜛䛔䛿
䜰䞊䜹䜲䝤䝕䞊䝍䛿
HDFS䜈

䝺䝥䝸䜿䞊䝅䝵䞁 / 䝟䞊䝔䜱䝅䝵䞁
䝇䝖䝺䞊䝆䝰䝕䝹:
- 䜲䞁䝯䝰䝸
- 䜲䞁䝯䝰䝸䛸䝻䞊䜹䝹䝕䜱䝇䜽
- 䜲䞁䝯䝰䝸䛸HDFS

SQL Engine – OLTP䚸䝖䝷䞁䝄䜽䝅䝵䞁
䛻䜒ᑐᛂ
IMDG 䜻䝱䝑䝅䝳ᶵ
⬟ - readThru,
writeBehind, ➼䚻

HDFS䛸䛾ᐦ䛺⤫ྜ –
䝇䝖䝸䞊䝭䞁䜾䚸R/W䜰䜽䝉䝇䝕䞊䝍ศᯒ䛿䜲䞁䝯䝰䝸䛷䛿䛺䛟HDFS䜢฼⏝䚷
– 䝅䞊䜿䞁䝅䝱䝹䛺䜰䜽䝉䝇䚸䜒䛧䛟䛿䜲䞁䜽
䝸䝯䞁䝍䝹䛺ฎ⌮䜢ᐇ⌧
䝕䞊䝍ᢞධ䛾୪ิ໬䛻䜘䜚䚸䝸䜰䝹䝍䜲䝮
䛻㏆䛔ヲ⣽䛺ศᯒ䜒ྍ⬟

MapReduce䛻䛚䛡
䜛reduceฎ⌮䛻䛶
䜲䞁䝯䝰䝸䜈䛾
䝕䞊䝍཯ᫎ
䝸䜰䝹䝍䜲䝮ฎ⌮䛸ศᯒ䛾㛫䛷䛾䜽䝻䞊䝈䝗䛺䝹䞊䝥䜢ᐇ⌧

GemFire XD䛻䛚䛡䜛䝕䞊䝍⟶⌮
CREATE TABLE FLIGHTS (
FLIGHT_ID CHAR(6) NOT NULL ,
SEGMENT_NUMBER INTEGER NOT NULL ,
…..
PARTITION BY COLUMN (FLIGHT_ID)
PERSISTENT
HDFSSTORE RWStore;
CREATE HDFSSTORE RWStore
NAMENODE hdfs://PHD1:8020
DIR /indexed-tables
BATCHSIZE 10
BATCHTIMEINTERVAL 2000
QUEUEPERSISTENT true;
Replicated Table Table
Replicated Table Replicated Table
Partitioned Table
Partitioned Table
Colocated Partition Colocated Partition Colocated Partition
Redundant Partition
Partitioned Table
Redundant Partition
Redundant Partition

䝺䜲䝔䞁䝅䛾ほⅬ䛷䜏䛯㐺⏝⠊ᅖ
䝬䝅䞁䛻䜘䜛
ᛂ⟅
ே㛫䛻䜘䜛
䜲䞁䝍䝷䜽䝔䜱䝤
䝺䝫䞊䝖䝞䝑䝏ฎ⌮ ᧯స
Milliseconds Seconds Seconds, Minutes Minutes, Hours
Online/OLTP/Operational DBs Analytics, Data Warehousing
GemFire XD PivotalHD HAWQ

Hadoopୖ䛷䛾䝸䜰䝹䝍䜲䝮ฎ⌮䛻䛚䛡䜛㑅ᢥ⫥
䜋䛸䜣䛹䛜䝕䞊䝍ศᯒ䛻ྥ䛔䛯䜲䞁䝍䝷䜽䝔䜱䝤䛺䜽䜶䝸ฎ⌮䛻䝣䜷䞊䜹䝇 Many more….

GemFire XD + Pivotal HD
䝸䜰䝹䝍䜲䝮+䝡䝑䜽䝕䞊䝍䛻䜘䜛Data Lake䜰䞊䜻䝔䜽䝏䝱䛾ᐇ⌧
Online Apps
䝸䜰䝹䝍䜲䝮䝕䞊䝍䛾ᢞධ 䝸䜰䝹䝍䜲䝮ศᯒ 㧗ᗘ䛺䝕䞊䝍ศᯒ
䝕䞊䝍䝰䝕䝹
Sensor Data / Feeds
䝕䞊䝍䝰䝕䝹
᭦᪂ Map-Reduce
Analytic Apps
GemFire XD HAWQ
HDFS
PXF
I/P O/P
Formatter
䝸䜰䝹䝍䜲䝮/䝉䝭䞉䝸䜰䝹䝍䜲䝮䛷䛾
䝕䞊䝍ᢞධ
Shared Data - HFiles
PCC
᭦᪂
ศᯒ䝕䞊䝍䛾ྲྀ䜚㎸䜏
ศᯒ䝕䞊䝍䛾ฎ⌮

PaaS䛻䛚䛡䜛䝃䞊䝡䝇䛸䛧䛶䛾ᥦ౪
Pivotal GemFire XD for Pivotal CF
Ÿ Pivotal CFୖ䛻ᒎ㛤䛥䜜䜛䜰䝥䝸䜿䞊䝅䝵䞁
䛻ᑐ䛧䛶GemFireXD䜢䝃䞊䝡䝇䛸䛧䛶ᥦ౪
– Pivotal CF: 䜸䞊䝥䞁䝋䞊䝇PaaS䛷䛒䜛Cloud
Foundry䛾ၟ⏝䝕䜱䝇䝖䝸䝡䝳䞊䝅䝵䞁
Ÿ ⊂⮬䛾䝃䞊䝡䝇䛸䛧䛶䜒㏣ຍྍ⬟
– Service Broker䛾ᐇ⿦
– 䝴䞊䝄ᐃ⩏䝃䞊䝡䝇

䜶䞁䝍䞊䝥䝷䜲䝈ྥ䛡
SQL on Hadoop䜶䞁䝆䞁

せⅬ
1. HAWQ䛸䛿ఱ䛛䠛
2. HAWQ䛿㏿䛔
3. HAWQ䛿SQL஫᥮

䝞䝑䝏ฎ⌮
http://www.anishsneh.com/2014/07/hadoop-mapreduce-api.html

䜰䝗䝩䝑䜽䜽䜶䝸
Ø SELECT id, name, address FROM foo LIMIT 100;
Ø SELECT a, b, count(c) FROM bar GROUP BY a, b;
…

Hadoopྥ䛡䜽䜶䝸ゝㄒ㻌Hive
Hive :SQL 䛻ఝ䛯HiveQL䛷MapReduce䜢ᐇ⾜
ᑐヰᘧ䛻኱㔞䝕䞊䝍䜢ฎ⌮䛜ྍ⬟䛻

Hive 䛾ၥ㢟Ⅼ
Ÿ ᑠつᶍ䛺䜽䜶䝸䜢ᐇ⾜䛩䜛䛻䜒᫬㛫䛜䛛䛛䜛
– MapReduce䜢౑⏝䛧䛶䛚䜚䚸ẖᅇJavaVM䛾㉳ື-⤊஢ฎ⌮䜢ᐇ⾜䛩䜛Ⅽ
Ÿ BI䝒䞊䝹➼䛷ື䛛䛺䛔ሙྜ䛜ከ䛔
– SQL䛾᏶඲஫᥮䛷䛿↓䛔Ⅽ

Hive䜘䜚䜒㏿䛟䚸SQL஫᥮䛷䚸
Hadoopୖ䛷ື䛟䜶䞁䝆䞁䛜ᚲせ
SQL on Hadoop

SQL on Hadoop “HAWQ”
HAdoop With Query
Ÿ 䜶䞁䝍䞊䝥䝷䜲䝈ྥ䛡䜽䜶䝸䜶䞁䝆䞁
Ÿ HDFSୖ䛻㧗㏿䝕䞊䝍䝧䞊䝇䜶䞁䝆䞁
HAWQ䜢ᦚ㍕
– 䝡䝑䜾䝕䞊䝍䛻ᑐ䛩䜛
㧗㏿䜽䜶䝸ฎ⌮
– ᶆ‽SQL‽ᣐ
– ⤫ィゎᯒ㛵ᩘ㻌MADlib 䛻ᑐᛂ
PivotalHD
MapReduce Pig
HDFS

HAWQ䛿㏿䛔
User intelligence
4.2
198
Sales analysis
8.7
161
Click analysis
2.0
415
Data exploration
2.7
1,285
BI drill down
2.8
1,815
47X
19X
208X
476X
648X
༢఩䠖⛊
User intelligence
4.2
37
Sales analysis
8.7
596
Click analysis
2.0
50
Data exploration
2.7
55
BI drill down
2.8
59
༢఩䠖⛊
9X
69X
25X
20X
21X

HAWQ/Impalaẚ㍑ᛶ⬟᳨ド
㻌㻔ᴗ⏺ᶆ‽ᣦᶆ㼀㻼㻯㻙㻰㻿䜢౑⏝㻕
䝃䝫䞊䝖䜽䜶䝸ᩘẚ㍑
100%
(ẕᩘ111䜽䜶䝸)
䝃䝫䞊䝖䛛䛴᏶஢䛧䛯䜽䜶䝸ᩘẚ㍑
100%
(ẕᩘ111䜽䜶䝸)
㠀䝃䝫䞊䝖䜽䜶䝸
㠀䝃䝫䞊䝖䜽䜶䝸
䜒䛧䛟䛿ฎ⌮୰䛻
␗ᖖ⤊஢䛧䛯䜽䜶䝸
HAWQฎ⌮᫬㛫䜢䠍䛸䛧䛯ሙྜ䛾
Impalaฎ⌮᫬㛫
Impala䛸᏶ẚ஢䛧㍑䛯䛧䜽䛶䜶䝸ᖹ ᆒ6ಸ䛾㧗㏿
28%
18%
ᛶ⬟ẚ
䈜㻚㻌㼀㻼㻯㻙㻰㻿䛸䛿ᴗ⏺ᶆ‽䛾ᛶ⬟ᣦᶆ䜢ᐃ⩏䛩䜛ᴗ⏺ᅋయ㻌㼀㻼㻯㻌㻔㼀㼞㼍㼚㼟㼍㼏㼠㼕㼛㼚㻌㻼㼞㼛㼏㼑㼟㼟㼕㼚㼓㻌㻼㼑㼞㼒㼛㼞㼙㼍㼚㼏㼑㻌㻯㼛㼡㼚㼏㼕㼘㻕㻌䛜ᥦ౪䛩䜛᝟ሗ⣔䝅䝇䝔䝮䛾䛯䜑䛾බᘧ䛺ᛶ⬟ᣦᶆ䛷䛩䚹㻞㻜㻝㻞ᖺ䛛䜙
ᥦ౪䛥䜜䛶䛔䜛㻌㼀㻼㻯㻙㻰㻿㻌䛷䛿䚸ὶ㏻ᴗ䛻䛚䛡䜛඾ᆺⓗ䛺䝕䞊䝍䝰䝕䝹䜢෌⌧䛧䛶䛚䜚䚸㻞㻡䝔䞊䝤䝹䚸㻠㻞㻥䜹䝷䝮䛻ᑐ䛧䛶ᵝ䚻䛺䜽䜶䝸䜢ᐇ⾜䛧䜎䛩䚹
䈜㻚㻌᳨ド䛷䛿㻌㻼㻴㻰㻝㻚㻝㻛㻴㻭㼃㻽㻝㻚㻝㻌䛸㻌㻯㻰㻴㻠㻚㻠㻛㻵㼙㼜㼍㼘㼍㻌㻝㻚㻝㻚㻝㻚䜢౑⏝䛧䜎䛧䛯䚹
ཧ⪃㼁㻾㻸㻦㻌㻻㼞㼏㼍㻦㻌㻭㻌㻹㼛㼐㼡㼘㼍㼞㻌㻽㼡㼑㼞㼥㻌㻻㼜㼠㼕㼙㼕㼦㼑㼞㻌㻭㼞㼏㼔㼕㼠㼑㼏㼠㼡㼞㼑㻌㼒㼛㼞㻌㻮㼕㼓㻌㻰㼍㼠㼍㻌㻌㼔㼠㼠㼜㻦㻛㻛㼣㼣㼣㻚㼓㼛㼜㼕㼢㼛㼠㼍㼘㻚㼏㼛㼙㻛㼟㼕㼠㼑㼟㻛㼐㼑㼒㼍㼡㼘㼠㻛㼒㼕㼘㼑㼟㻛㻿㻵㻳㻹㻻㻰㻹㼍㼥㻞㻜㻝㻠㻴㻭㼃㻽㻭㼐㼢㼍㼚㼠㼍㼓㼑㼟㻚㼜㼐㼒
ཧ⪃㼁㻾㻸㻦㻌㼇㻞㻜㻝㻠㻛㻢㻛㻞㻡㼉㻌㻼㼕㼢㼛㼠㼍㼘㻌㻴㻭㼃㻽㻌㻮㼑㼚㼏㼔㼙㼍㼞㼗㻌㻰㼑㼙㼛㼚㼟㼠㼞㼍㼠㼑㼟㻌㼁㼜㻌㼀㼛㻌㻞㻝㼤㻌㻲㼍㼟㼠㼑㼞㻌㻼㼑㼞㼒㼛㼞㼙㼍㼚㼏㼑㻌㼛㼚㻌㻴㼍㼐㼛㼛㼜㻌㻽㼡㼑㼞㼕㼑㼟㻌㼀㼔㼍㼚㻌㻿㻽㻸㻙㼘㼕㼗㼑㻌㻿㼛㼘㼡㼠㼕㼛㼚㼟㻌㼔㼠㼠㼜㻦㻛㻛㼎㼘㼛㼓㻚㼓㼛㼜㼕㼢㼛㼠㼍㼘㻚㼏㼛㼙㻛㼜㼕㼢㼛㼠㼍㼘㻛㼜㼞㼛㼐㼡㼏㼠㼟㻛㼜㼕㼢㼛㼠㼍㼘㻙㼔㼍㼣㼝㻙
㼎㼑㼚㼏㼔㼙㼍㼞㼗㻙㼐㼑㼙㼛㼚㼟㼠㼞㼍㼠㼑㼟㻙㼡㼜㻙㼠㼛㻙㻞㻝㼤㻙㼒㼍㼟㼠㼑㼞㻙㼜㼑㼞㼒㼛㼞㼙㼍㼚㼏㼑㻙㼛㼚㻙㼔㼍㼐㼛㼛㼜㻙㼝㼡㼑㼞㼕㼑㼟㻙㼠㼔㼍㼚㻙㼟㼝㼘㻙㼘㼕㼗㼑㻙㼟㼛㼘㼡㼠㼕㼛㼚㼟

䝕䞊䝍ฎ⌮䝣䝻䞊ẚ㍑
䜽䜶䝸䛾ᢞධ
䝥䝷䞁సᡂ
䜽䜶䝸ᐇ⾜
⤖ᯝ䛾㏉ಙ
䝹䞊䝹䝧䞊䝇䛾䜸䝥䝔䜱䝬䜲䝄
• 䝔䞊䝤䝹䝕䞊䝍䛾ෆᐜ䛻㛵䜟䜙䛪䜽䜶
䝸䛻䜘䛳䛶ᐇ⾜䝥䝷䞁䜢సᡂ
• MapReduce䝇䜽䝸䝥䝖䜢సᡂ
MapReduceฎ⌮
• Java䝥䝻䝉䝇䛾㉳ື䞉೵Ṇ
୰㛫䝕䞊䝍䛾䝕䜱䝇䜽ฎ⌮
• ㏲ḟ䝕䜱䝇䜽IO䛾Ⓨ⏕
䝁䝇䝖䝧䞊䝇䜸䝥䝔䜱䝬䜲䝄 Orca
• 䝔䞊䝤䝹䝕䞊䝍䛾ෆᐜ(䝕䞊䝍㔞䚸䜹䞊
䝕䜱䝘䝸䝔䜱➼)䜢㋃䜎䛘᭱㐺䛺ᐇ⾜䝥䝷
䞁䜢సᡂ
C䝥䝻䝉䝇ฎ⌮
• ᖖ㥔䝥䝻䝉䝇䛻䜘䜛༶᫬ฎ⌮
୰㛫䝕䞊䝍䛾䜸䞁䝯䝰䝸ฎ⌮
• 䝟䜲䝥䝷䜲䞁ฎ⌮䛻䜘䜛䜸䞁䝯䝰䝸䛾㧗
㏿ฎ⌮
• 䝕䜱䝇䜽IO䜢᤼㝖

䜽䜶䝸䜸䝥䝔䜱䝬䜲䝄䛜ᐇ⌧䛩䜛㧗㏿䝕䞊䝍ฎ⌮
MapReduce䜢௓䛥䛪䝕䞊䝍䜢䝟䜲䝥䝷䜲䞁ฎ⌮
Ÿ 䝁䝇䝖䝧䞊䝇䛾䜸䝥䝔䜱䝬䜲䝄䛜᭱
㐺䛺ᐇ⾜䝥䝷䞁䜢㑅ᢥ
– DBฎ⌮(䝇䜻䝱䞁䚸䝆䝵䜲䞁䚸䝋䞊䝖䚸㞟
ィ➼)䛻ᑐ䛧䛶䝁䝇䝖䜢⟬ฟ
– 䝉䜾䝯䞁䝖㛫㏻ಙ(“䝰䞊䝅䝵䞁”)䜒䜸䝥
䝔䜱䝬䜲䝄䛜ᣦ♧
Ÿ 䝎䜲䝘䝭䝑䜽䝟䜲䝥䝷䜲䞁ฎ⌮
– ୰㛫䝕䞊䝍䛾䜸䞁䝯䝰䝸ฎ⌮
PHYSICAL EXECUTION PLAN
FROM SQL
Gather Motion
4:1(Slice 3)
Sort
HashAggregate
HashJoin
Redistribute Motion
4:4(Slice 1)
HashJoin
Hash
HashJoin
Seq Scan on
customer
Hash Hash
Broadcast Motion
4:4(Slice 2)
Seq Scan on
motion
Seq Scan
on lineitem
Seq Scan on
orders

᭱᪂䜽䜶䝸䜸䝥䝔䜱䝬䜲䝄 Orca䛻䜘䜛ᅽಽⓗᛶ⬟ྥୖ
ᚑ᮶䜽䜶䝸䜸䝥䝔䜱䝬䜲䝄ẚᖹᆒ䠑ಸ䛾ᛶ⬟ྥୖ䜢ᐇ⌧
Ÿ 䝆䝵䜲䞁䜸䞊䝎䞊䝸䞁䜾
– 䜲䞁䝍䝁䝛䜽䝖䜈䛾኱つᶍ䝕䞊䝍㌿㏦䜢㜵䛠䝔䞊䝤䝹⤖ྜ㡰ᗎ䜢㑅ᐃ
Ÿ ┦㛵䝃䝤䜽䜶䝸ฎ⌮
– ཯᚟ⓗ䝃䝤䜽䜶䝸ฎ⌮䜢ᅇ㑊
Ÿ ືⓗ㻌䝟䞊䝔䜱䝅䝵䞁᤼㝖䝇䜻䝱䞁
– 䜽䜶䝸ฎ⌮୰䛾୰㛫䝕䞊䝍䛻䛒䜟䛫䛶䝇䜻䝱䞁ᑐ㇟䝟䞊䝔䜱䝅䝵䞁䜢ື
ⓗ䛻㑅ᐃ䞉᤼㝖

䝆䝵䜲䞁䜸䞊䝎䝸䞁䜾䛾౛
䝴䞊䝄IDẖ䛻㞟ィ䛧䛯䛔ሙྜ
A B
join
ྛ䝜䞊䝗䛻
䝕䞊䝍䜢ศ㓄
A
(100୓௳䚸
䝴䞊䝄id䛷ศᩓ)
B
(100୓௳䚸
᫂⣽id䛷ศᩓ)
ྛ䝜䞊䝗䛻
䝕䞊䝍䜢ศ㓄
join
or
䛹䛱䜙䛜䜘䛔䛛䠛

ศᩓ䜻䞊䛾௳ᩘ䜢䝠䝇䝖䜾䝷䝮໬
A䛾䝜䞊䝗ẖ䛾䝴䞊䝄idᩘ
B䛾䝜䞊䝗ẖ䛾䝴䞊䝄idᩘ
A
(䝴䞊䝄id䛷ศᩓ)
B
(᫂⣽id䛷ศᩓ)
䝔䞊䝤䝹᝟ሗ䛛䜙ศᯒ
䝔䞊䝤䝹᝟ሗ䛛䜙ศᯒ

䝠䝇䝖䜾䝷䝮䛛䜙ᐇ⾜᫬㛫䛾ぢ✚䜒䜚
A䛾䝜䞊䝗ẖ䛾䝴䞊䝄idᩘ
B䛾䝜䞊䝗ẖ䛾䝴䞊䝄idᩘ
A䜢ᅛᐃ䛧䛶B䛾䝕䞊䝍䜢ྛ䝜䞊䝗䛻ศ㓄䛧䛯ሙྜ䛾ᐇ⾜᫬㛫
B䜢ᅛᐃ䛧䛶A䛾䝕䞊䝍䜢ྛ䝜䞊䝗䛻ศ㓄䛧䛯ሙྜ䛾ᐇ⾜᫬㛫

䛣䛱䜙䛾᪉䛜᫬㛫䛜䛛䛛䜙䛺䛔䛾䛷᥇⏝
A B
ศ㓄 ศ㓄
join
A B
join
ẚ㍑

HAWQ䛷䛿⮬ື䛷䝆䝵䜲䞁䜸䞊䝎䝸䞁䜾䜢ᐇ᪋
᥇⏝
A B
ศ㓄 ศ㓄
join
A B
join
ẚ㍑

HAWQ䛿SQL‽ᣐ
SQL ‘92 ’93 2003 OLAPᑐᛂ
䝝䞊䝗/OS RDBMS BI䝒䞊䝹
䝝䞊䝗/OS HDFS BI䝒䞊䝹

HAWQ䜢ᨭ䛘䜛㻌GreenplumDB 10ᖺ䛾ᐇ⦼
㻳㼞㼑㼑㼚㼜㼘㼡㼙㻰㻮䛾୺せ䝔䜽䝜䝻䝆䞊䜢㻌㻴㻭㼃㻽㻌䛷᥇⏝

• ᶆ‽㻌㻿㻽㻸㻌ᑐᛂ
• 䝁䝇䝖䝧䞊䝇䜸䝥䝔䜱䝬䜲䝄
• 䝎䜲䝘䝭䝑䜽䝟䜲䝥䝷䜲䞁ฎ⌮
• 䝻䞊䝇䝖䜰䞉䜹䝷䝮䝇䝖䜰୧᪉䜈䛾ᑐᛂ
• ᅽ⦰㻔㻽㼡㼕㼏㼗㻸㼆㻘㻌㼆㻸㻵㻮㻘㻌㻾㻸㻱㻕
• ศᩓ᱁⣡
• 䝬䝹䝏䝺䝧䝹䝟䞊䝔䜱䝅䝵䝙䞁䜾
• 䝟䝷䝺䝹䞊䝻䞊䝗䞉䜰䞁䝻䞊䝗
• 㧗㏿䝕䞊䝍෌ศᩓ
• ⤫ィゎᯒ㛵ᩘ㻔㻹㻭㻰㼘㼕㼎㻕
• 㻿㻱㻸㻱㻯㼀
• 㻵㻺㻿㻱㻾㼀
• 㻶㻻㻵㻺
• 䝡䝳䞊
• እ㒊⾲
• 䝸䝋䞊䝇䝬䝛䝆䝯䞁䝖
• 䝉䜻䝳䝸䝔䜱
• ㄆド
• ⟶⌮䞉┘ど
• 㻻㻰㻮㻯㻛㻶㻰㻮㻯ᑐᛂ

ศᩓ㻌ᩘ್ィ⟬䞉ᶵᲔᏛ⩦䝷䜲䝤䝷䝸
MADlib䜢ྠᲕ
ண ⓗ䝰䝕䝸䞁䜾䝷䜲䝤䝷䝸
Latest release: MADlib v1.6, URL: madlib.net
ᶵᲔᏛ⩦䜰䝹䝂䝸䝈䝮
• ୺ᡂศศᯒ(PCA)
• 䜰䝋䝅䜶䞊䝅䝵䞁䝹䞊䝹ศᯒ㻌(䜰䝣䜱䝙䝔䜱ศ
ᯒ,䝬䞊䜿䝑䝖䝞䝇䜿䝑䝖ศᯒ)
• 䝖䝢䝑䜽䝰䝕䝸䞁䜾㻌(䝟䝷䝺䝹LDA)
• Ỵᐃᮌ
• 䜰䞁䝃䞁䝤䝹Ꮫ⩦(䝷䞁䝎䝮䝣䜷䝺䝇䝖)
• 䝃䝫䞊䝖䝧䜽䝍䞊䝬䝅䞁
• 䝁䞁䝕䜱䝅䝵䝘䝹䝷䞁䝎䝮䝣䜱䞊䝹䝈(CRF)
• 䜽䝷䝇䝍䝸䞁䜾 (Kᖹᆒἲ)
• 䜽䝻䝇䝞䝸䝕䞊䝅䝵䞁
⥺ᙧ䝅䝇䝔䝮ゎᯒ
• ␯⾜ิ䝋䝹䝞䞊
• ᐦ⾜ิ䝋䝹䝞䞊
୍⯡໬⥺ᙧ䝰䝕䝹
• ⥺ᙧᅇᖐ
• 䝻䝆䝇䝔䜱䝑䜽ᅇᖐ
• ከ㡯䝻䝆䝇䝔䜱䝑䜽ᅇᖐ
• 䝁䝑䜽䝇ẚ౛䝝䝄䞊䝗
• ᅇᖐศᯒ
• 䜶䝷䝇䝔䜱䝑䜽䝛䝑䝖ᆺṇつ໬
• 䝃䞁䝗䜲䝑䝏᥎ᐃ
⾜ิᅉᏊศゎ
• ≉␗್ศゎ㻌(SVD)
• ప䝷䞁䜽㏆ఝ
グ㏙⤫ィ
䝇䜿䝑䝏䝧䞊䝇᥎ᐃ
• CountMin䝇䜿䝑䝏
• Flajolet-Martin䝇䜿䝑䝏
• ᭱㢖್䝇䜿䝑䝏
┦㛵㛵ಀ
⤫ィ್䝃䝬䝸
䝃䝫䞊䝖䝰䝆䝳䞊䝹
㓄ิ₇⟬
␯䝧䜽䝖䝹
䝷䞁䝎䝮䝃䞁䝥䝸䞁䜾
☜⋡㛵ᩘ

MADlib䝃䝫䞊䝖ᶵ⬟䛾౛
K-means 㐺⏝๓
x
y
Ⅼ䛾ሢ䜢኱䜎䛛䛻3䛴
䛻ศ๭䛧䛯䛔

MADlib䝃䝫䞊䝖ᶵ⬟䛾౛
K-means 㐺⏝ᚋ
䜽䝷䝇䝍A 䜽䝷䝇䝍B
x
y
䜽䝷䝇䝍C
Ⅼ䛾ሢ䛛䜙䚸3䛴䛾䜽䝷䝇䝍䛜ᆒ➼
䛻䝞䝷䛡䜛䜘䛖䛻䚸㔜ᚰ●䜢సᡂ
ྛⅬ䛿㔜ᚰ●䛻㏆䛔䜽䝷䝇䝍䛻ᡤᒓ

MADlib䛷䛾K-meansᐇ⾜౛
㔜ᚰ䜢ồ䜑䜛౛
SELECT * FROM madlib.kmeanspp( 'km_sample',
'points',
3,
'madlib.squared_dist_norm2',
'madlib.avg',
20,
0.001
);
ධຊ䝔䞊䝤䝹
madlib䛾㛵ᩘ
ฟຊ
ศ๭ᩘ
centroids | {{13.24,2.59, … ,735},{13.856,…,1078},{14.255,…,1378.75}}
…
3䛴䛾㔜ᚰ䛾఩⨨
ヲ⣽䠖http://doc.madlib.net/latest/group__grp__kmeans.html

䝬䜲䜽䝻䜰䝗ᵝ
Pivotal HD+HAWQ஦౛

Pivotal HD+HAWQ䛻䜘䜚SPSS䛾᪤Ꮡ㈨⏘䛻୍ษᡭ䜢ຍ䛘䜛䛣䛸
䛺䛟ศᯒྍ⬟䛺䝕䞊䝍䛾ᣑ኱䜢ప䝁䝇䝖䛷ᐇ⌧䚹䛚ᐈᵝ䛻䛸䛳䛶䛾
➇தຊ䛾※Ἠ䛷䛒䜛䝕䞊䝍ศᯒ⢭ᗘ䛾ྥୖ䛻㈉⊩
᪤Ꮡ⎔ቃ
䞉 IBM PureData/SPSS䛾ศᯒᇶ┙䜢ᵓ⠏
䞉㻌ศᯒせᮃ䛾㧗ᗘ໬䛻䜘䜚᱁⣡䝕䞊䝍ቑ኱䚸PureData
䛾ᐜ㔞ᯤῬ
䞉㻌䝁䝇䝖䜢ᢚ䛘䜛䛯䜑Hadoop (Cloudera↓ൾ∧)䜢ే⏝
ㄢ㢟
䞉 SPSS䛾䜽䜶䝸䛜Hadoopᶆ‽䝒䞊䝹HIVE䛷䛿
䚷䛂㏻䜙䛺䛔䛃䛂㏵୰䛷䜶䝷䞊䛻䛺䜛䛃䛂ⴭ䛧䛟㐜䛔䛃䛯䜑
䚷ᐇ⏝䛻ሓ䛘䛺䛔
(DWH)
IBM PureData
(Hadoop)
Pivotal
Pivotal HD + HAWQᑟධ⤖ᯝ
䞉 SPSS䛾䜽䜶䝸䛜ኚ᭦↓䛧䛷100%฼⏝ྍ⬟
䞉 HIVE䛸ẚ㍑䛧䛶᭱኱⣙70ಸ㏿䛔ᛶ⬟䜢グ㘓
䞉㻌ỗ⏝IA䝃䞊䝞6ྎ䛷ᐇ⿦
(BA)
IBM
SPSS
ᑡ䛺䛔ᢞ㈨䛷ᗈ⠊ᅖ䛺䝕䞊䝍䛻ᑐ䛧䛶௒䜎䛷䛷
䛝䛺䛛䛳䛯ศᯒ䜢ᐇ᪋ྍ⬟䛻

HAWQ䛾⤖ㄽ
1. HAWQ䛿SQL on Hadoop䛾୍䛴
Hadoop䛷Greenplum䛾䜽䜶䝸䜶䞁䝆䞁䜢ື䛟䜘䛖䛻䛧䛯䜒䛾
2. HAWQ䛿Hive䜘䜚ᩘ༑~ᩘⓒಸ,
Impala䜘䜚ᩘಸ㏿䛔
3. HAWQ䛿SQL஫᥮䛺䛾䛷䚸᪤Ꮡ䛾䝒䞊䝹䛛䜙
౑䛔䜔䛩䛔

䝡䝑䜾䝕䞊䝍᫬௦䛾
௻ᴗኚ㠉䜢ᐇ⌧䛩䜛Pivotal
• 䝕䞊䝍⵳✚䊻ศᯒ䊻䜰䝥䝸䜿䞊䝅
䝵䞁䛾䝃䜲䜽䝹
• 䛒䜙䜖䜛䝕䞊䝍䜢䛸䜙䛘䜛䝡䝑䜾䝕
䞊䝍ᇶ┙䛂䝕䞊䝍䝺䜲䜽䛃ᵓ᝿

䝡䝑䜾䝕䞊䝍᫬௦䛾
௻ᴗኚ㠉䜢ᐇ⌧䛩䜛Pivotal
• 䝕䞊䝍⵳✚䊻ศᯒ䊻䜰䝥䝸䜿䞊䝅䝵䞁䈊
䛾䝃䜲䜽䝹
• 䛒䜙䜖䜛䝕䞊䝍䜢䛸䜙䛘䜛䝡䝑䜾䝕䞊䝍䈊
ᇶ┙䛂䝕䞊䝍䝺䜲䜽䛃㻌ᵓ᝿

Pivotal䛾ᥦ౪䛩䜛䝁䞁䝃䝹䝔䜱䞁䜾䝃䞊䝡䝇
Ÿ 䝡䝑䜾䝕䞊䝍ᇶ┙䜢ᑟධ䛧䛯䛔
– 䛡䛹䝻䜾㌿㏦䛺䛹䛹䛖䛩䜜䜀䞉䞉
Ÿ 䝕䞊䝍ศᯒ䞉ᶵᲔᏛ⩦䜢䛧䛯䛔
䜰䝥䝸䜿䞊䝅䝵䞁
䜸䞊䝥䞁䝋䞊䝇
䜽䝷䜴䝗(PaaS)ᇶ┙
䞉䞉䞉 ᪂䛯䛺
ᇶᖿ䝅䝇䝔䝮㐃ᦠ
䝣䜯䝇䝖䝕䞊䝍
(M2M/䝸䜰䝹䝍䜲䝮)
䝡䝆䝛䝇䝰䝕䝹
䜰䝆䝱䜲䝹㛤Ⓨ
䜰䝘䝸䝔䜱䜽䝇䝕䞊䝍
䝕䞊䝍䝃䜲䜶䞁䝇䝡䝑䜾䝕䞊䝍
(DWH/Hadoop)
Pivotal䝆䝱䝟䞁䛻䛶
ᑟධ䝃䝫䞊䝖䠃䝖䝺䞊䝙䞁䜾ᐇ᪋
᪥ᮏே䛾䝕䞊䝍䝬䜲䝙䞁䜾䜶䞁䝆䝙䜰ཬ䜃
䝕䞊䝍䝃䜲䜶䞁䝔䜱䝇䝖ᅾ⡠

[db tech showcase Tokyo 2014] D36: 次世代分析基盤 "Data Lake" を支えるPivotalのインメモリ+SQL on Hadoopテクノロジー by Pivotalジャパン株式会社市村友寛 & 宵勇樹

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to [db tech showcase Tokyo 2014] D36: 次世代分析基盤 "Data Lake" を支えるPivotalのインメモリ+SQL on Hadoopテクノロジー by Pivotalジャパン株式会社市村友寛 & 宵勇樹

Similar to [db tech showcase Tokyo 2014] D36: 次世代分析基盤 "Data Lake" を支えるPivotalのインメモリ+SQL on Hadoopテクノロジー by Pivotalジャパン株式会社市村友寛 & 宵勇樹 (20)

More from Insight Technology, Inc.

More from Insight Technology, Inc. (20)

Recently uploaded

Recently uploaded (20)