刘中兵 sohu-tpc
lzbbox@hotmail.com
 大型系统=多台服务器,监控体系=分布式
汇报+统一处理
 Facebook每天log上百亿条信息(每秒log上
百万条)
 Scribe不依赖中心服务器,中心服务器不可
用时log在本地,当中心服务器可用后又会
同步到中心服务器
 S...
 1.Thrift简介
 2.Scribe简介
 3.安装Thrift+Scribe
 4.简单演示样例
 5.部署for log4j
 Facebook的Thrift:基于C++,开源,跨语
言C++、C#、Erlang、Java、Perl、PHP、
Python、Ruby、HTML、XSD、Smalltalk、
Cocoa、OCaml、Haskell
 Google的P...
Thrift传
输
通
信
协
议
服
务
端
代
码
客
户
端
代
码
Java
C++
Python
Erlang
Java
C++
Python
Erlang
 bool true, false
 byte 8位的有符号整数
 i16 16位的有符号整数
 i32 32位的有符号整数
 i64 64位的有符号整数
 double 64位的浮点数
 string UTF-8编码的字符串
...
 定义数据类型
 struct User {
 1: i32 uid,
 2: string name
 }
 定义服务接口
 service UserService {
 void store(1: User user),
...
 导出命令
 thrift –r –gen py service.thrift
 可导出语言配置
 cocoa(Cocoa)、cpp(C++)、csharp(C#)、
erl(Erlang)、hs(Haskell)、html(HTML)...
Central
Server
Client
Client
Local
Server
Local
Server
 enum ResultCode
 {
 OK,
 TRY_LATER
 }
 struct LogEntry
 {
 1: string category,
 2: string message
 }
 service ...
 启动Scribe Server:
 scribed config.conf
 停止Scribe Server:
 scribe_ctrl stop 11810
 监控Scribe Server:
 scribe_ctrl {com...
 配置项:
 port: 端口,默认0
 max_msg_per_second: 默认100,000
 max_queue_size: 默认5,000,000 bytes
 check_interval: 检查每个store间隔,默认...
 file – 写入文件local or nfs.
 network – 发送到Scribe Server
 buffer – 主从存储,primary and a secondary
store
 bucket – 按照消息头分类ha...
autoconf-2.60
automake-1.10.1
libtool-2.2.6b
zlib-1.2.5
libevent-2.0.4
python-2.6.5
boost-1.36.0(依赖python)
erlang-R13B01
t...
 autoconf-2.60(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 autoconf --version
 automake-1.10.1(ftp://ft...
 libtool-2.2.6b(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 libtool --version
 zlib-1.2.5(http://www.zl...
 libevent-2.0.4(http://monkey.org/~provos/libevent/)
 ./configure
 make
 make install
 ll /usr/local/lib/
 python-2....
 boost-1.36.0(http://www.boost.org/)
 ./configure --prefix=/usr/local/boost_1_36_0 --
without-icu
 make install(较长时间)
...
 thrift-0.2.0(http://incubator.apache.org/thrift/)
 ./bootstrap.sh
 ./configure --with-boost=/usr/local/boost_1_36_0
 ...
 scribe-2.2(http://github.com/facebook/scribe)
 vi /etc/profile
▪ export BOOST_ROOT=/usr/local/boost_1_36_0
▪ export LD_...
 启动服务端: /opt/soft/scribe
 scribed examples/example1.conf
 启动客户端: /opt/soft/scribe
 echo "hello world" | ./examples/scr...
Thrift样例:Java、Python通信
Scribe Java客户端:pyserverjavaclient
 安装apache-ant-1.8.0
 export ANT_HOME=/opt/soft/apache-ant-1.8.0
 export PATH=$PATH:$ANT_HOME/bin
 编译libthrift.jar
 an...
 配置文件:/usr/liuzhongbing
 struct User {
 1: i32 uid,
 2: string name
 }
 service UserService {
 void store(1: User u...
 编译:/usr/liuzhongbing/gen-java
 javac -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api-
1.6.0.jar...
 生成:/usr/liuzhongbing
 thrift -r -gen py user.thrift
 启动:/usr/liuzhongbing/gen-py
 服务端:python PythonServer.py 9091
▪ S...
 生成:/opt/soft/scribe/if
 thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen java
scribe.thrift
 thrift -r -I /opt/soft/th...
 启动Python服务端:/opt/soft/scribe
 scribed examples/example1.conf
▪ [Thu May 13 15:39:29 2010] "[test] Creating new category...
scribe-log4j部署结构
部署Scribe Server
引用scribe-log4j
测试Scribe Server
scribe-
log4j.jar
Central
Server:11810
web1
service2
Local
Server:
11811
Local
Server:
11811
service1
 Central Server:192.168.0.1:11810
 启动:/usr/local/scribe/conf/central_start.sh
 停止:/usr/local/scribe/conf/central_stop.s...
 引用scribe-log4j
 <dependency>
 <groupId>${project.groupId}</groupId>
 <artifactId>scribe-log4j</artifactId>
 <version...
 修改host指向Local Server
 log4j.appender.scribeAppender.host=192.168.0.1
 注意:App中必须host=localhost
 运行scribe-log4j测试类
 Te...
 安装并启动服务端: /opt/soft
 ./setup.sh > setup.log
 使用scribe-log4j测试
 Test.java
Thrift+scribe实现分布式日志收集,并与log4j集成
Thrift+scribe实现分布式日志收集,并与log4j集成
Thrift+scribe实现分布式日志收集,并与log4j集成
Upcoming SlideShare
Loading in …5
×

Thrift+scribe实现分布式日志收集,并与log4j集成

2,923 views
2,557 views

Published on

Thrift+scribe实现分布式日志收集,并与log4j集成

Published in: Technology
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,923
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
74
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

Thrift+scribe实现分布式日志收集,并与log4j集成

  1. 1. 刘中兵 sohu-tpc lzbbox@hotmail.com
  2. 2.  大型系统=多台服务器,监控体系=分布式 汇报+统一处理  Facebook每天log上百亿条信息(每秒log上 百万条)  Scribe不依赖中心服务器,中心服务器不可 用时log在本地,当中心服务器可用后又会 同步到中心服务器  Scribe本身由C++实现, RPC通信为Python 实现(基于Thrift)
  3. 3.  1.Thrift简介  2.Scribe简介  3.安装Thrift+Scribe  4.简单演示样例  5.部署for log4j
  4. 4.  Facebook的Thrift:基于C++,开源,跨语 言C++、C#、Erlang、Java、Perl、PHP、 Python、Ruby、HTML、XSD、Smalltalk、 Cocoa、OCaml、Haskell  Google的Protocol Buffers:基于C++,开源, 基于C++, Java, Python  Baidu使用的ICE基于C++,开源  Renren使用ACE基于C++(比ICE底层) Facebook与Google比较:http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers
  5. 5. Thrift传 输 通 信 协 议 服 务 端 代 码 客 户 端 代 码 Java C++ Python Erlang Java C++ Python Erlang
  6. 6.  bool true, false  byte 8位的有符号整数  i16 16位的有符号整数  i32 32位的有符号整数  i64 64位的有符号整数  double 64位的浮点数  string UTF-8编码的字符串  binary 字符数组  struct 结构体  list<type> 有序的元素列表,类似于STL的vector  set<type> 无序的不重复元素集,类似于STL的set  map<type1,type2> key-value型的映射,类似于STL的map  exception 是一个继承于本地语言的exception基类  service 服务。包含多个函数接口(纯虚函数)
  7. 7.  定义数据类型  struct User {  1: i32 uid,  2: string name  }  定义服务接口  service UserService {  void store(1: User user),  User retrieve(1: i32 uid)  }
  8. 8.  导出命令  thrift –r –gen py service.thrift  可导出语言配置  cocoa(Cocoa)、cpp(C++)、csharp(C#)、 erl(Erlang)、hs(Haskell)、html(HTML)、 java(Java)、ocaml(OCaml)、perl(Perl)、 php(PHP)、py(Python)、rb(Ruby)、st(Smalltalk)、 xsd(XSD)
  9. 9. Central Server Client Client Local Server Local Server
  10. 10.  enum ResultCode  {  OK,  TRY_LATER  }  struct LogEntry  {  1: string category,  2: string message  }  service scribe extends fb303.FacebookService  {  ResultCode Log(1: list<LogEntry> messages);  }
  11. 11.  启动Scribe Server:  scribed config.conf  停止Scribe Server:  scribe_ctrl stop 11810  监控Scribe Server:  scribe_ctrl {command} 11810 ▪ status – ALIVE表示运行 ▪ version – 版本 ▪ alive – 服务器启动时的时间(s) ▪ reload – 重载配置文件 ▪ counters – 统计项 ▪ received good, received bad, sent, denied for queue size, denied for rate, retries, requeue, lost, received blank category
  12. 12.  配置项:  port: 端口,默认0  max_msg_per_second: 默认100,000  max_queue_size: 默认5,000,000 bytes  check_interval: 检查每个store间隔,默认5s  new_thread_per_category: 默认yes,为每一个catagory创建一个新 的线程  num_thrift_server_threads: 收取消息的监听线程数,默认1  例如:  port=1463  max_msg_per_second=2000000  max_queue_size=10000000  check_interval=3
  13. 13.  file – 写入文件local or nfs.  network – 发送到Scribe Server  buffer – 主从存储,primary and a secondary store  bucket – 按照消息头分类hash存储  null – 忽略  thriftfile – 类似于file的ThriftTFileTransport  multi – 同时写多个 参考:http://wiki.github.com/facebook/scribe/scribe-configuration
  14. 14. autoconf-2.60 automake-1.10.1 libtool-2.2.6b zlib-1.2.5 libevent-2.0.4 python-2.6.5 boost-1.36.0(依赖python) erlang-R13B01 thrift-0.2.0(依赖automake/libtool/python/erlang) scribe-2.2(依赖libevent/python/boost/thrift) scribe thriftboost python erlang zlib libevent libtoolautomake autoconf 安装包参见:192.168.0.1:/opt/soft
  15. 15.  autoconf-2.60(ftp://ftp.gnu.org/gnu)  ./configure  make  make install  autoconf --version  automake-1.10.1(ftp://ftp.gnu.org/gnu)  ./configure  make  make install  automake --version
  16. 16.  libtool-2.2.6b(ftp://ftp.gnu.org/gnu)  ./configure  make  make install  libtool --version  zlib-1.2.5(http://www.zlib.net/)  ./configure  make  make install  ll /usr/local/include
  17. 17.  libevent-2.0.4(http://monkey.org/~provos/libevent/)  ./configure  make  make install  ll /usr/local/lib/  python-2.6.5(http://python.org/)  ./configure  make(稍长时间,忽略Tcl/Tk错误)  make install  python –V, ctrl+D退出  输出库路径:import sys, print sys.path
  18. 18.  boost-1.36.0(http://www.boost.org/)  ./configure --prefix=/usr/local/boost_1_36_0 -- without-icu  make install(较长时间)  ll /usr/local/boost_1_36_0/  erlang-R13B01(http://erlang.org/)  ./configure  make (较长时间)  make install  erl, ctrl+C
  19. 19.  thrift-0.2.0(http://incubator.apache.org/thrift/)  ./bootstrap.sh  ./configure --with-boost=/usr/local/boost_1_36_0  make (较长时间)  make install  ll /usr/lib/python2.6/site-packages/thrift  fb303(在thrift里:cd contrib/fb303)  ./bootstrap.sh  ./configure --with-boost=/usr/local/boost_1_36_0  make  make install  ll /usr/lib/python2.6/site-packages/fb303 MONO错误需要pkg-config0.20
  20. 20.  scribe-2.2(http://github.com/facebook/scribe)  vi /etc/profile ▪ export BOOST_ROOT=/usr/local/boost_1_36_0 ▪ export LD_LIBRARY_PATH=/usr/local/boost_1_36_0/lib::/usr/lib:/usr/local/lib  . /etc/profile  ./bootstrap.sh  ./configure --with-boost=/usr/local/boost_1_36_0  make  make install  ll /usr/lib/python2.6/site-packages/scribe  拷贝python库  cp -r /usr/lib/python2.6/site-packages/* /usr/local/lib/python2.6/site- packages  cp /usr/local/bin/python /usr/bin/python  验证:python, import thrift, import scribe
  21. 21.  启动服务端: /opt/soft/scribe  scribed examples/example1.conf  启动客户端: /opt/soft/scribe  echo "hello world" | ./examples/scribe_cat test  查看日志  more /tmp/scribetest/test/test_current  hello world
  22. 22. Thrift样例:Java、Python通信 Scribe Java客户端:pyserverjavaclient
  23. 23.  安装apache-ant-1.8.0  export ANT_HOME=/opt/soft/apache-ant-1.8.0  export PATH=$PATH:$ANT_HOME/bin  编译libthrift.jar  ant  安装slf4j-1.6.0
  24. 24.  配置文件:/usr/liuzhongbing  struct User {  1: i32 uid,  2: string name  }  service UserService {  void store(1: User user),  User retrieve(1: i32 uid)  }  生成:/usr/liuzhongbing  thrift -r -gen java user.thrift  产生:User.java、UserService.java
  25. 25.  编译:/usr/liuzhongbing/gen-java  javac -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api- 1.6.0.jar:./ *.java  启动:/usr/liuzhongbing/gen-java  服务端:java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j- 1.6.0/slf4j-api-1.6.0.jar:./ JavaServer 9091 ▪ Starting the server... ▪ store: 1=liuzhongbing ▪ store: 2=guest ▪ retrieve: 1=liuzhongbing  客户端:java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j- 1.6.0/slf4j-api-1.6.0.jar:./ JavaClient localhost 9091 ▪ retrieve: liuzhongbing
  26. 26.  生成:/usr/liuzhongbing  thrift -r -gen py user.thrift  启动:/usr/liuzhongbing/gen-py  服务端:python PythonServer.py 9091 ▪ Starting the server... ▪ store: 1=liuzhongbing ▪ store: 2=guest ▪ retrieve: 1=liuzhongbing  客户端:python PythonClient.py localhost 9091 ▪ retrieve: liuzhongbing
  27. 27.  生成:/opt/soft/scribe/if  thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen java scribe.thrift  thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen py scribe.thrift  编译:/opt/soft/scribe/if/gen-java  javac -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j- api-1.6.0.jar:./ *.java
  28. 28.  启动Python服务端:/opt/soft/scribe  scribed examples/example1.conf ▪ [Thu May 13 15:39:29 2010] "[test] Creating new category from model default" ▪ [Thu May 13 15:39:29 2010] "store thread starting" ▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/scribetest/test/test_00000> for writing" ▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/test/test_00000> for writing" ▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <DISCONNECTED> to <SENDING_BUFFER>" ▪ [Thu May 13 15:39:29 2010] "[test] successfully read <0> entries from file </tmp/test/test_00000>" ▪ [Thu May 13 15:39:29 2010] "[test] No more buffer files to send, switching to streaming mode" ▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <SENDING_BUFFER> to <STREAMING>"  启动Java客户端:/opt/soft/scribe/if/gen-java  java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api- 1.6.0.jar:./ JavaClient localhost 1463
  29. 29. scribe-log4j部署结构 部署Scribe Server 引用scribe-log4j 测试Scribe Server
  30. 30. scribe- log4j.jar Central Server:11810 web1 service2 Local Server: 11811 Local Server: 11811 service1
  31. 31.  Central Server:192.168.0.1:11810  启动:/usr/local/scribe/conf/central_start.sh  停止:/usr/local/scribe/conf/central_stop.sh  查询:/usr/local/scribe/conf/central_status.sh  统计:/usr/local/scribe/conf/central_counters.sh  日志目录:/opt/scribelog/central, 暂存 /opt/scribelog/tmpcentral  Local Server:192.168.0.2:11811(App所在主机)  启动:/usr/local/scribe/conf/local_start.sh  停止:/usr/local/scribe/conf/local_stop.sh  查询:/usr/local/scribe/conf/local_status.sh  统计:/usr/local/scribe/conf/local_counters.sh  远程主机:192.168.0.1:11810,暂存/opt/scribelog/tmplocal 配置文件参见:192.168.0.1:/usr/local/scribe/conf/
  32. 32.  引用scribe-log4j  <dependency>  <groupId>${project.groupId}</groupId>  <artifactId>scribe-log4j</artifactId>  <version>${scribe-log4j.version}</version>  </dependency>  log4j.properties中增加了几个功能配置:  # scribe  log4j.logger.com.mycompany=DEBUG,scribeAppender  log4j.appender.scribeAppender=org.apache.scribe_log4j.ScribeAppender  log4j.appender.scribeAppender.layout=org.apache.log4j.PatternLayout  log4j.appender.scribeAppender.layout.ConversionPattern=[%p] - [%d{yyyy-MM-dd HH:mm:ss}] - [%c.%t(%L)]: %m%n  log4j.appender.scribeAppender.host=localhost  log4j.appender.scribeAppender.port=11811  log4j.appender.scribeAppender.reconnectInterval=10000  log4j.appender.scribeAppender.category=scribe-service  注意:自动重连信息输出在当前App标准stderr.log中
  33. 33.  修改host指向Local Server  log4j.appender.scribeAppender.host=192.168.0.1  注意:App中必须host=localhost  运行scribe-log4j测试类  Test.java  查看服务端收到的日志:  tail -f /opt/scribelog/central/scribe-service/scribe- service_current 在/commons/scribe-log4j下进行
  34. 34.  安装并启动服务端: /opt/soft  ./setup.sh > setup.log  使用scribe-log4j测试  Test.java

×