This document discusses Fluentd, an open source data collector. It collects and processes log data from multiple sources and locations. Fluentd handles log collection in JSON format from various systems like user log files, databases, and applications. It then routes the logs to their destinations such as file outputs, databases, or analytics tools through flexible configuration. This allows log data from different sources and formats to be normalized and processed in a unified manner.

1. Fluentd

2. Meetup

3. Group
Introduction
@fluentd
www.fluentd.org
Tuesday, June 25, 13

4. WHAT’S

5. FLUENTD?
•JSON-BASED

6. LOG

7. COLLECTOR
•OPEN

8. SOURCE

9. (APACHE

10. 2.0)
•PRODUCTION

11. READY
Tuesday, June 25, 13

12. JSON-BASED
•“EVENT”

13. IS

14. THE

15. SMALLEST

16. UNIT

17. OF

18. DATA

19.
FOR

20. FLUENTD
•THE

21. FORMAT

22. IS

23. JSON
•INTERNALLY,

24. WE

25. USE

26. MESSAGEPACK

27.
(JSON-LIKE

28. BINARY

29. SERIALIZER)
Tuesday, June 25, 13

30. WHY

31. JSON!?
MAINTAINABLE

32. LOGGING!
Tuesday, June 25, 13

33. HOW

34. LOGGING

35. IS

36. DONE

37. AT
MANY

38. PLACES
Tuesday, June 25, 13

39. HOW

40. LOGGING

41. IS

42. DONE

43. AT
MANY

44. PLACES
LOG

45.
FILES
DATA

46.
USER

47. 1
DATA

48.
USER

49. 2
INTERMED
IATE

50. DB
DATA

51.
USER

52. 3
DATA

53.
USER

54. 2’S

55.
BOSS
DATA

56.
USER

57. 1’S

58.
BOSS
Tuesday, June 25, 13

59. HOW

60. LOGGING

61. IS

62. DONE

63. AT
MANY

64. PLACES
LOG

65.
FILES
DATA

66.
USER

67. 1
DATA

68.
USER

69. 2
INTERMED
IATE

70. DB
DATA

71.
USER

72. 3
DATA

73.
USER

74. 2’S

75.
BOSS
DATA

76.
USER

77. 1’S

78.
BOSSuser 1’s
script
user 2’s
script
DBA’s
script
user 1’s
script 2
user 2’s
script 2
user 3’s
script...
Tuesday, June 25, 13

79. HOW

80. LOGGING

81. IS

82. DONE

83. AT
MANY

84. PLACES
LOG

85.
FILES
DATA

86.
USER

87. 1
DATA

88.
USER

89. 2
INTERMED
IATE

90. DB
DATA

91.
USER

92. 3
DATA

93.
USER

94. 2’S

95.
BOSS
DATA

96.
USER

97. 1’S

98.
BOSSuser 1’s
script
user 2’s
script
DBA’s
script
user 1’s
script 2
user 2’s
script 2
user 3’s
script...
FORMAT
CHANGE!
Tuesday, June 25, 13

99. HOW

100. LOGGING

101. IS

102. DONE

103. AT
MANY

104. PLACES
LOG

105.
FILES
DATA

106.
USER

107. 1
DATA

108.
USER

109. 2
INTERMED
IATE

110. DB
DATA

111.
USER

112. 3
DATA

113.
USER

114. 2’S

115.
BOSS
DATA

116.
USER

117. 1’S

118.
BOSSuser 1’s
script
user 2’s
script
DBA’s
script
user 1’s
script 2
user 2’s
script 2
user 3’s
script...
FORMAT
CHANGE!
Tuesday, June 25, 13

119. HOW

120. LOGGING

121. IS

122. DONE

123. AT
MANY

124. PLACES
LOG

125.
FILES
DATA

126.
USER

127. 1
DATA

128.
USER

129. 2
INTERMED
IATE

130. DB
DATA

131.
USER

132. 3
DATA

133.
USER

134. 2’S

135.
BOSS
DATA

136.
USER

137. 1’S

138.
BOSSuser 1’s
script
user 2’s
script
DBA’s
script
user 1’s
script 2
user 2’s
script 2
user 3’s
script...
FORMAT
CHANGE!
:-(
Tuesday, June 25, 13

139. HOW

140. LOGGING

141. IS

142. DONE

143.
WITH

144. FLUENTD
Tuesday, June 25, 13

145. HOW

146. LOGGING

147. IS

148. DONE

149.
WITH

150. FLUENTD
FLUEND
DATA

151.
USER

152. 1
DATA

153.
USER

154. 2
INTERMED
IATE

155. DB
DATA

156.
USER

157. 3
DATA

158.
USER

159. 2’S

160.
BOSS
DATA

161.
USER

162. 1’S

163.
BOSS
APACHE/
NGINX
SYSLOG
LOGS

164. W/
IN-HOUSE

165.
FORMAT
Tuesday, June 25, 13

166. HOW

167. LOGGING

168. IS

169. DONE

170.
WITH

171. FLUENTD
FLUEND
DATA

172.
USER

173. 1
DATA

174.
USER

175. 2
INTERMED
IATE

176. DB
DATA

177.
USER

178. 3
DATA

179.
USER

180. 2’S

181.
BOSS
DATA

182.
USER

183. 1’S

184.
BOSS
APACHE/
NGINX
SYSLOG
LOGS

185. W/
IN-HOUSE

186.
FORMAT
JSON
JSON
JSON
JSON
JSON
JSON
in_syslog
in_tail
customized
in_tail
Tuesday, June 25, 13

187. HOW

188. LOGGING

189. IS

190. DONE

191.
WITH

192. FLUENTD
FLUEND
DATA

193.
USER

194. 1
DATA

195.
USER

196. 2
INTERMED
IATE

197. DB
DATA

198.
USER

199. 3
DATA

200.
USER

201. 2’S

202.
BOSS
DATA

203.
USER

204. 1’S

205.
BOSS
APACHE/
NGINX
SYSLOG
LOGS

206. W/
IN-HOUSE

207.
FORMAT
JSON
JSON
JSON
JSON
JSON
JSON
in_syslog
in_tail
customized
in_tail
FORMAT
CHANGE!
Tuesday, June 25, 13

208. HOW

209. LOGGING

210. IS

211. DONE

212.
WITH

213. FLUENTD
FLUEND
DATA

214.
USER

215. 1
DATA

216.
USER

217. 2
INTERMED
IATE

218. DB
DATA

219.
USER

220. 3
DATA

221.
USER

222. 2’S

223.
BOSS
DATA

224.
USER

225. 1’S

226.
BOSS
APACHE/
NGINX
SYSLOG
LOGS

227. W/
IN-HOUSE

228.
FORMAT
JSON
JSON
JSON
JSON
JSON
JSON
in_syslog
in_tail
customized
in_tail
FORMAT
CHANGE! MODIFY
JUST

229. HERE!
Tuesday, June 25, 13

230. HOW

231. LOGGING

232. IS

233. DONE

234.
WITH

235. FLUENTD
FLUEND
DATA

236.
USER

237. 1
DATA

238.
USER

239. 2
INTERMED
IATE

240. DB
DATA

241.
USER

242. 3
DATA

243.
USER

244. 2’S

245.
BOSS
DATA

246.
USER

247. 1’S

248.
BOSS
APACHE/
NGINX
SYSLOG
LOGS

249. W/
IN-HOUSE

250.
FORMAT
JSON
JSON
JSON
JSON
JSON
JSON
in_syslog
in_tail
customized
in_tail
FORMAT
CHANGE! MODIFY
JUST

251. HERE!
Tuesday, June 25, 13

252. WHY

253. JSON!?
MAINTAINABLE

254. LOGGING!
Tuesday, June 25, 13

255. OPEN

256. SOURCE!
•APACHE

257. 2.0
•COMMUNITY!

258. (1,100+

259. STARS

260. ON

261. GITHUB)
•160+

262. PLUGINS

263. (MY

264. DEMO

265. WILL

266. USE

267. ONE

268. OF

269.
THEM)
Tuesday, June 25, 13

270. PRODUCTION

271. READY
•USED

272. IN

273. PRODUCTION

274. WITH

275. 4,000+

276. SERVERS
•CAN

277. PROCESS

278. ~15,000

279. MSGS/SEC

280. PER

281. CORE
•GOOD

282. COMMUNITY

283. SUPPORT

284. (ACTIVE

285.
MAILING

286. LIST)
Tuesday, June 25, 13

287. WHAT’S

288. FLUENTD?
•JSON-BASED

289. LOG

290. COLLECTOR
•OPEN

291. SOURCE

292. (APACHE

293. 2.0)
•PRODUCTION

294. READY
Tuesday, June 25, 13

295. DEMO!
Tuesday, June 25, 13

296. DEMO!
Chrome
browser

297.
log
fluentd
your

298.
phone!
customized
in_tail
out_twilio
Tuesday, June 25, 13

299. DEMO!
Chrome
browser

300.
log
fluentd
your

301.
phone!
customized
in_tail
out_twilio
“CALL

302. ME

303. DEFINITELY
IF

304. MY

305. EMPLOYEE

306. IS

307. SURFIN’

308. FACEBOOK.COM”
Tuesday, June 25, 13