8. 准实时海量数据分析系统架构探究
page_view pv_users
user
pageid age
pageid userid time userid age gender
1 25
1 111 9:08:01 X 111 25 female =
2 111 9:08:13 2 25
222 32 male
1 222 9:08:14 1 32
• SQL:
INSERT INTO TABLE pv_users
SELECT pv.pageid, u.age
FROM page_view pv JOIN user u ON (pv.userid =
u.userid);
Data eXchange Platform | zhouchen.zm
9. 准实时海量数据分析系统架构探究
page_view
pagei userid time key value key value
d
111 <1,1> 111 <1,1>
1 111 9:08:01
111 <1,2> 111 <1,2>
2 111 9:08:13
1 222 9:08:14 222 <1,1> 111 <2,25>
Shuffle
user Map Sort Reduce
userid age gender key value key value
111 25 female 111 <2,25> 222 <1,1>
222 32 male 222 <2,32>
222 <2,32>
Data eXchange Platform | zhouchen.zm
10. 准实时海量数据分析系统架构探究
w
pv_users
time key value key value
111 <1,1> 111 <1,1> pagei age
:08:01 d
111 <1,2> 111 <1,2>
:08:13 1 25
:08:14 222 <1,1> 111 <2,25>
Shuffle 2 25
Map Sort Reduce
ender key value key value
emale 111 <2,25> 222 <1,1> pageid age
1 32
male 222 <2,32>
222 <2,32>
Data eXchange Platform | zhouchen.zm
11. 准实时海量数据分析系统架构探究
Dremel-嵌套列存储
Data eXchange Platform | zhouchen.zm
12. 准实时海量数据分析系统架构探究
Dremel-多级分布式查询树
Data eXchange Platform | zhouchen.zm
13. 准实时海量数据分析系统架构探究
Dremel-近似计算
Data eXchange Platform | zhouchen.zm
24. 准实时海量数据分析系统架构探究
PowerDrill: 双层字典
SELECT search_string, COUNT(*) as c FROM data
WHERE search_string IN ("la redoute", "voyages sncf")
GROUP BY search_string
ORDER BY c DESC LIMIT 10;
Data eXchange Platform | zhouchen.zm
41. 准实时海量数据分析系统架构探究
SQL第一大难题: Join
• Join会带来大量网络间数据交换
• 只支持大表对小表的Join
– Google BigQuery要Join的小表压缩后不能超过8MB
Data eXchange Platform | zhouchen.zm
42. 准实时海量数据分析系统架构探究
SQL第二大难题: distinct aggregation
•distinct aggregation也会带来大量网络间数据交换 , 是除Join的第二难题
•Google BigQuery只支持count(distinct c)
Google面试题：
给你一年的Google搜索日志和一台有限内存的机器，能否只扫描一遍，估计
这一年中不同的独特的搜索(unique queries)的个数
•Hash Table
•Bitmap
•Linear Probabilistic Counter
•HyperLogLog
Big Data Counting: How To Count A Billion Distinct Objects Using Only 1.5KB Of Memory
Data eXchange Platform | zhouchen.zm
43. 准实时海量数据分析系统架构探究
SQL第三大难题: Top K
• Top K是准实时系统的第三个挑战
• Graham Cormode and S. Muthukrishnan. An improved data stream summary:
The Count-Min sketch and its applications. pages 29–38. 2004.
• Cheqing Jin, Weining Qian, Chaofeng Sha, Jeffrey X. Yu, and Aoying Zhou.
Dynamically maintaining frequent items over a data stream. In CIKM ’03:
Proceedings of the twelfth international conference on Information and
knowledge management, pages 287–294, New York, NY, USA, 2003. ACM.
• Ahmed Metwally, Divyakant Agrawal, and Amr Abbadi. Efficient computation
of frequent and top-k elements in data streams. pages 398–412. 2005.
Data eXchange Platform | zhouchen.zm
44. 准实时海量数据分析系统架构探究
Bitmap索引
• 第一个商用系统 Model 204,
P. O’Neil, 1987
• 比创建B-Tree更快, 更容易
• 查询高效: 按位操作
– A < 2 : b0 OR b1
– A > 2 : b3 OR b4 OR b5
• 多维查询高效: bitvector的
与或非
• 求count等aggregation高效
Data eXchange Platform | zhouchen.zm
45. 准实时海量数据分析系统架构探究
Bitmapped Group-set Index
SELECT c.mktsegment, o.order_id
FROM orders o, customers c
WHERE o.cust_id = c.cust_id
GROUP BY c.mktsegment, o.order_id
Data eXchange Platform | zhouchen.zm
46. 准实时海量数据分析系统架构探究
Bitmapped Join Index
SELECT o.customer_id, l.unit_price * l.quantity
FROM lineitems l, orders o
WHERE l.order_id = o.order_id
Data eXchange Platform | zhouchen.zm
48. 准实时海量数据分析系统架构探究
参考文献
• [Dremel] Dremel: Interactive Analysis of Web-Scale Datasets
• [PowerDrill] Processing a Trillion Cells per Mouse Click
• [RCFile] RCFile: A Fast and Space-efficient Data Placement Structure in
MapReduce-based Warehouse Systems
• [InfoBright] BrightHouse A New Database Engine based on Rough Sets
Data eXchange Platform | zhouchen.zm
