This document discusses Scribe and Thrift, log aggregation systems developed by Facebook. It provides an overview of Scribe, which is used to collect log data from different servers and applications. Scribe uses Thrift for its RPC communication and does not rely on a central server - if the central server is unavailable, log data will be stored locally and synced once the server is available again. The document also discusses installing and configuring Scribe and Thrift on a Linux system and implementing a simple logging example.

1. 刘中兵 sohu-tpc
lzbbox@hotmail.com

2.  大型系统=多台服务器，监控体系=分布式
汇报+统一处理
 Facebook每天log上百亿条信息(每秒log上
百万条)
 Scribe不依赖中心服务器，中心服务器不可
用时log在本地，当中心服务器可用后又会
同步到中心服务器
 Scribe本身由C++实现， RPC通信为Python
实现(基于Thrift)

3.  1.Thrift简介
 2.Scribe简介
 3.安装Thrift+Scribe
 4.简单演示样例
 5.部署for log4j

5.  Facebook的Thrift：基于C++，开源，跨语
言C++、C#、Erlang、Java、Perl、PHP、
Python、Ruby、HTML、XSD、Smalltalk、
Cocoa、OCaml、Haskell
 Google的Protocol Buffers：基于C++，开源，
基于C++, Java, Python
 Baidu使用的ICE基于C++，开源
 Renren使用ACE基于C++(比ICE底层)
Facebook与Google比较：http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers

6. Thrift传
输
通
信
协
议
服
务
端
代
码
客
户
端
代
码
Java
C++
Python
Erlang
Java
C++
Python
Erlang

7.  bool true, false
 byte 8位的有符号整数
 i16 16位的有符号整数
 i32 32位的有符号整数
 i64 64位的有符号整数
 double 64位的浮点数
 string UTF-8编码的字符串
 binary 字符数组
 struct 结构体
 list<type> 有序的元素列表，类似于STL的vector
 set<type> 无序的不重复元素集，类似于STL的set
 map<type1,type2> key-value型的映射，类似于STL的map
 exception 是一个继承于本地语言的exception基类
 service 服务。包含多个函数接口(纯虚函数)

8.  定义数据类型
 struct User {
 1: i32 uid,
 2: string name
 }
 定义服务接口
 service UserService {
 void store(1: User user),
 User retrieve(1: i32 uid)
 }

9.  导出命令
 thrift –r –gen py service.thrift
 可导出语言配置
 cocoa(Cocoa)、cpp(C++)、csharp(C#)、
erl(Erlang)、hs(Haskell)、html(HTML)、
java(Java)、ocaml(OCaml)、perl(Perl)、
php(PHP)、py(Python)、rb(Ruby)、st(Smalltalk)、
xsd(XSD)

11. Central
Server
Client
Client
Local
Server
Local
Server

12.  enum ResultCode
 {
 OK,
 TRY_LATER
 }
 struct LogEntry
 {
 1: string category,
 2: string message
 }
 service scribe extends fb303.FacebookService
 {
 ResultCode Log(1: list<LogEntry> messages);
 }

13.  启动Scribe Server:
 scribed config.conf
 停止Scribe Server:
 scribe_ctrl stop 11810
 监控Scribe Server:
 scribe_ctrl {command} 11810
▪ status – ALIVE表示运行
▪ version – 版本
▪ alive – 服务器启动时的时间(s)
▪ reload – 重载配置文件
▪ counters – 统计项
▪ received good, received bad, sent, denied for queue size, denied for rate, retries,
requeue, lost, received blank category

14.  配置项：
 port: 端口，默认0
 max_msg_per_second: 默认100,000
 max_queue_size: 默认5,000,000 bytes
 check_interval: 检查每个store间隔，默认5s
 new_thread_per_category: 默认yes，为每一个catagory创建一个新
的线程
 num_thrift_server_threads: 收取消息的监听线程数，默认1
 例如:
 port=1463
 max_msg_per_second=2000000
 max_queue_size=10000000
 check_interval=3

15.  file – 写入文件local or nfs.
 network – 发送到Scribe Server
 buffer – 主从存储，primary and a secondary
store
 bucket – 按照消息头分类hash存储
 null – 忽略
 thriftfile – 类似于file的ThriftTFileTransport
 multi – 同时写多个
参考：http://wiki.github.com/facebook/scribe/scribe-configuration

16. autoconf-2.60
automake-1.10.1
libtool-2.2.6b
zlib-1.2.5
libevent-2.0.4
python-2.6.5
boost-1.36.0(依赖python)
erlang-R13B01
thrift-0.2.0(依赖automake/libtool/python/erlang)
scribe-2.2(依赖libevent/python/boost/thrift)
scribe
thriftboost
python erlang
zlib
libevent
libtoolautomake
autoconf
安装包参见：192.168.0.1:/opt/soft

17.  autoconf-2.60(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 autoconf --version
 automake-1.10.1(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 automake --version

18.  libtool-2.2.6b(ftp://ftp.gnu.org/gnu)
 ./configure
 make
 make install
 libtool --version
 zlib-1.2.5(http://www.zlib.net/)
 ./configure
 make
 make install
 ll /usr/local/include

19.  libevent-2.0.4(http://monkey.org/~provos/libevent/)
 ./configure
 make
 make install
 ll /usr/local/lib/
 python-2.6.5(http://python.org/)
 ./configure
 make(稍长时间，忽略Tcl/Tk错误)
 make install
 python –V, ctrl+D退出
 输出库路径：import sys, print sys.path

20.  boost-1.36.0(http://www.boost.org/)
 ./configure --prefix=/usr/local/boost_1_36_0 --
without-icu
 make install(较长时间)
 ll /usr/local/boost_1_36_0/
 erlang-R13B01(http://erlang.org/)
 ./configure
 make (较长时间)
 make install
 erl, ctrl+C

21.  thrift-0.2.0(http://incubator.apache.org/thrift/)
 ./bootstrap.sh
 ./configure --with-boost=/usr/local/boost_1_36_0
 make (较长时间)
 make install
 ll /usr/lib/python2.6/site-packages/thrift
 fb303(在thrift里：cd contrib/fb303)
 ./bootstrap.sh
 ./configure --with-boost=/usr/local/boost_1_36_0
 make
 make install
 ll /usr/lib/python2.6/site-packages/fb303
MONO错误需要pkg-config0.20

22.  scribe-2.2(http://github.com/facebook/scribe)
 vi /etc/profile
▪ export BOOST_ROOT=/usr/local/boost_1_36_0
▪ export LD_LIBRARY_PATH=/usr/local/boost_1_36_0/lib::/usr/lib:/usr/local/lib
 . /etc/profile
 ./bootstrap.sh
 ./configure --with-boost=/usr/local/boost_1_36_0
 make
 make install
 ll /usr/lib/python2.6/site-packages/scribe
 拷贝python库
 cp -r /usr/lib/python2.6/site-packages/* /usr/local/lib/python2.6/site-
packages
 cp /usr/local/bin/python /usr/bin/python
 验证：python, import thrift, import scribe

23.  启动服务端： /opt/soft/scribe
 scribed examples/example1.conf
 启动客户端： /opt/soft/scribe
 echo "hello world" | ./examples/scribe_cat test
 查看日志
 more /tmp/scribetest/test/test_current
 hello world

24. Thrift样例：Java、Python通信
Scribe Java客户端：pyserverjavaclient

25.  安装apache-ant-1.8.0
 export ANT_HOME=/opt/soft/apache-ant-1.8.0
 export PATH=$PATH:$ANT_HOME/bin
 编译libthrift.jar
 ant
 安装slf4j-1.6.0

26.  配置文件：/usr/liuzhongbing
 struct User {
 1: i32 uid,
 2: string name
 }
 service UserService {
 void store(1: User user),
 User retrieve(1: i32 uid)
 }
 生成：/usr/liuzhongbing
 thrift -r -gen java user.thrift
 产生：User.java、UserService.java

27.  编译：/usr/liuzhongbing/gen-java
 javac -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api-
1.6.0.jar:./ *.java
 启动：/usr/liuzhongbing/gen-java
 服务端：java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-
1.6.0/slf4j-api-1.6.0.jar:./ JavaServer 9091
▪ Starting the server...
▪ store: 1=liuzhongbing
▪ store: 2=guest
▪ retrieve: 1=liuzhongbing
 客户端：java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-
1.6.0/slf4j-api-1.6.0.jar:./ JavaClient localhost 9091
▪ retrieve: liuzhongbing

28.  生成：/usr/liuzhongbing
 thrift -r -gen py user.thrift
 启动：/usr/liuzhongbing/gen-py
 服务端：python PythonServer.py 9091
▪ Starting the server...
▪ store: 1=liuzhongbing
▪ store: 2=guest
▪ retrieve: 1=liuzhongbing
 客户端：python PythonClient.py localhost 9091
▪ retrieve: liuzhongbing

29.  生成：/opt/soft/scribe/if
 thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen java
scribe.thrift
 thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen py
scribe.thrift
 编译：/opt/soft/scribe/if/gen-java
 javac -classpath
/usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-
api-1.6.0.jar:./ *.java

30.  启动Python服务端：/opt/soft/scribe
 scribed examples/example1.conf
▪ [Thu May 13 15:39:29 2010] "[test] Creating new category from model default"
▪ [Thu May 13 15:39:29 2010] "store thread starting"
▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/scribetest/test/test_00000> for
writing"
▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/test/test_00000> for writing"
▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <DISCONNECTED> to
<SENDING_BUFFER>"
▪ [Thu May 13 15:39:29 2010] "[test] successfully read <0> entries from file
</tmp/test/test_00000>"
▪ [Thu May 13 15:39:29 2010] "[test] No more buffer files to send, switching to streaming
mode"
▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <SENDING_BUFFER> to
<STREAMING>"
 启动Java客户端：/opt/soft/scribe/if/gen-java
 java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api-
1.6.0.jar:./ JavaClient localhost 1463

31. scribe-log4j部署结构
部署Scribe Server
引用scribe-log4j
测试Scribe Server

32. scribe-
log4j.jar
Central
Server:11810
web1
service2
Local
Server:
11811
Local
Server:
11811
service1

33.  Central Server：192.168.0.1:11810
 启动：/usr/local/scribe/conf/central_start.sh
 停止：/usr/local/scribe/conf/central_stop.sh
 查询：/usr/local/scribe/conf/central_status.sh
 统计：/usr/local/scribe/conf/central_counters.sh
 日志目录：/opt/scribelog/central, 暂存
/opt/scribelog/tmpcentral
 Local Server：192.168.0.2:11811(App所在主机)
 启动：/usr/local/scribe/conf/local_start.sh
 停止：/usr/local/scribe/conf/local_stop.sh
 查询：/usr/local/scribe/conf/local_status.sh
 统计：/usr/local/scribe/conf/local_counters.sh
 远程主机：192.168.0.1:11810，暂存/opt/scribelog/tmplocal
配置文件参见：192.168.0.1:/usr/local/scribe/conf/

34.  引用scribe-log4j
 <dependency>
 <groupId>${project.groupId}</groupId>
 <artifactId>scribe-log4j</artifactId>
 <version>${scribe-log4j.version}</version>
 </dependency>
 log4j.properties中增加了几个功能配置：
 # scribe
 log4j.logger.com.mycompany=DEBUG,scribeAppender
 log4j.appender.scribeAppender=org.apache.scribe_log4j.ScribeAppender
 log4j.appender.scribeAppender.layout=org.apache.log4j.PatternLayout
 log4j.appender.scribeAppender.layout.ConversionPattern=[%p] - [%d{yyyy-MM-dd
HH:mm:ss}] - [%c.%t(%L)]: %m%n
 log4j.appender.scribeAppender.host=localhost
 log4j.appender.scribeAppender.port=11811
 log4j.appender.scribeAppender.reconnectInterval=10000
 log4j.appender.scribeAppender.category=scribe-service
 注意：自动重连信息输出在当前App标准stderr.log中

35.  修改host指向Local Server
 log4j.appender.scribeAppender.host=192.168.0.1
 注意：App中必须host=localhost
 运行scribe-log4j测试类
 Test.java
 查看服务端收到的日志：
 tail -f /opt/scribelog/central/scribe-service/scribe-
service_current
在/commons/scribe-log4j下进行

36.  安装并启动服务端： /opt/soft
 ./setup.sh > setup.log
 使用scribe-log4j测试
 Test.java