20th.陈晓鸣百度海量日志分析架构及处理经验分享

!   46.70.93.94 - - [11/Nov/2011:11:11:11 -1100] "GET /book/
1984.html HTTP/1.1“404 2326 http://www.baidu.com/s?
wd=1984&rsv_bp=0&rsv_spt=3&inputT=947 "Mozilla/
5.0(iPad; U; CPU iPhone OS 3_2 like Mac OS X; en-us)
AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4
Mobile/7B314 Safari/531.21.10 “

——

!   46.70.93.94 - -
!   [11/Nov/2011:11:11:11 -1100]
!   "GET /book/1984.html HTTP/1.1"
!   404
!   2326
!   "http://www.baidu.com/s?
wd=1984&rsv_bp=0&rsv_spt=3&inputT=9
47"
!   "Mozilla/5.0(iPad; U; CPU iPhone OS 3_2
like Mac OS X; en-us) AppleWebKit/
531.21.10 (KHTML, like Gecko) Version/
4.0.4 Mobile/7B314 Safari/531.21.10 “

——

!   46.70.93.94 - -
!   [11/Nov/2011:11:11:11 -1100]
!   404
!   2326
47"
!   " Mozilla/5.0(iPad; U; CPU iPhone OS
3_2 like Mac OS X; en-us) AppleWebKit/
531.21.10 (KHTML, like Gecko) Version/
4.0.4 Mobile/7B314 Safari/531.21.10"

——

!   46.70.93.94 - -
!   [11/Nov/2011:11:11:11 -1100]
!   GET /book/1984.html HTTP/1.1
!   404
!   2326
7"
!   "Mozilla/5.0(iPad; U; CPU iPhone OS
3_2 like Mac OS X; en-us)
AppleWebKit/531.21.10 (KHTML, like
Gecko) Version/4.0.4 Mobile/7B314
Safari/531.21.10 "

——

!   46.70.93.94 - -
!   [11/Nov/2011:11:11:11 -1100]
!   404
!   2326
wd=1984&rsv_bp=0&rsv_spt=3&inputT=947"
!   " Mozilla/5.0(iPad; U; CPU iPhone OS
Safari/531.21.10 "

——

!   46.70.93.94 - -
!   [11/Nov/2011:11:11:11 -1100]
!   404
!   2326
!   " http://www.baidu.com/s?wd=1984&
rsv_bp=0&rsv_spt=3&inputT=947 "
!   "Mozilla/5.0(iPad; U; CPU iPhone OS
Safari/531.21.10 "

•  •  •  • 
•  •  • 
• 
•  Ad$hoc
•  ……

!  …

!   $
!  B*S $
!   $
!   $

!   C++$ $
!  C++ $
!   SQL $
!  PHP$+$C$ $ !   $
!  Schema $
!   $
!   !   $
PHP .so $

DISQL

LOG

LSP

DISQL

! 
!  
!   _Url _Res( )
!   _Url _Site
!  
!  
! 
! 
!  JSON

DQuery
!  
!   _Url _Res( )
!   _Url _S
!  
!  
! 
! 
!  JSON

!  PHP SQL ( )
!  SQL M/R
! 
!  DAG
!  MapReduce
! 
!  SQL
!  PHP
!  C++ + C-Runtime NEW!
!   RAII +
!   Copy On Write
!   schema
!   C++ PHP

! 

! 
!  parser
!  JSON
[
{
"cmd": "load“,
"path": null
"using": "SchemaReader"
"from": 17
"options": {"max_item_in_mem“: 100000}
"include": [25]
}
, {"cmd":"filter"……}, {"cmd":"join"……},…… ……
]

SQL
[
{
"cmd": "load“,
"path": null
"using": "SchemaReader"
"from": 17
"options":
{"max_item_in_mem“:
100000}
"include": [25]
}
, {"cmd":"filter"……},
{"cmd":"join"……},…… ……
]

! 
! 
! 
! 
! 
! 
! 
!  ( )
!  MapReduce
!   Schema
!  schema
!  C++ PHP DOT
!

Group

Unique Shuffle
Map$Phase$
Reduce$
$
$
Limit$1 $
Group
$
$
Combine$
$$
Group Count

Shuffle $
Shuffle
$
Reduce$ $
$ Reduce$
$
Count
$
$
Sum
$
Reduce$Phase

Schema

ﬁeld ID name age ﬁeld ID score
type uint64 string int32 type uint64 double
index 2 5 9 Index 0 1

join

Field ID name age Score
Type Uint64 string int32 double
Index 2 5 9 10

!  
!  
!   Combiner
!   Cached Combiner
!   key Join
!  
!  
!  I/O

!   PHP
!   C++
!   DOT
!   / MapReduce

Processor ——Pipes & Filter
class$
Processor
init()
process(
)
fini()

class$ class$ class$ class$
Selector Filter Counter UserProcessor

init() init() init() init()
process( process( process(
process()
) ) )
fini() fini() fini() fini()

! 
4 1 10 27
3540 4761 1221 +34.5%
DQuery 1153 3359 2206 +191%
1569 2963 1394 +88.9%
!  !   LSP

24% PM 1352 47.4%
}$$67%
DQuery 43% RD 1174 41.2%
33% OP 190 6.66%
136 4.77%
2852 100%

! 
!  ●
! 
!  ● ● ● …
! 
!  LSP
!  ● UI ●
!  DISQL
!  ● ● ● ●
! 
! 
!  (@ ) (chenxiaoming@baidu.com)
!  Hadoop in China 12 2 2 20 DISQL2.0

……
……
chenxiaoming@baidu.com

关注我们：t.baidu-tech.com

资料下载和详细介绍：infoq.com/cn/zones/baidu-salon
“畅想•交流•争鸣•聚会”是百度技术沙龙的宗旨。百度技术沙龙是由百度与InfoQ中文站定期组织的线下技术交流活动。目
的是让中高端技术人员有一个相对自由的思想交流和交友沟通的的平台。主要分讲师分享和OpenSpace两个关键环节，每期
只关注一个焦点话题。

讲师分享和现场Q&A让大家了解百度和其他知名网站技术支持的先进实践经验，OpenSpace环节是百度技术沙龙主题的升华
和展开，提供一个自由交流的平台。针对当期主题，参与者人人都可以发起话题，展开讨论。

InfoQ 策划·组织·实施
关注我们：weibo.com/infoqchina

20th.陈晓鸣百度海量日志分析架构及处理经验分享

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 20th.陈晓鸣百度海量日志分析架构及处理经验分享

Similar to 20th.陈晓鸣百度海量日志分析架构及处理经验分享 (20)

Recently uploaded

Recently uploaded (20)