Successfully reported this slideshow.
Your SlideShare is downloading. ×

Fluentd and WebHDFS

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 20 Ad
Advertisement

More Related Content

Advertisement

More from SATOSHI TAGOMORI (20)

Advertisement

Fluentd and WebHDFS

  1. 1. Fluentd and WebHDFS & what makes it possible to write out_webhdfs in 30min. TAGOMORI Satoshi (@tagomoris) NHN Japan Fluentd meetup 3 (2012/11/08) 12年11月8日木曜日
  2. 2. @tagomoris NHN Japan Corp (Web Service Division) Fluentd committer, plugin developer fluent-agent-lite, ... 12年11月8日木曜日
  3. 3. Usecase of Fluentd Monitoring, Notification and Visualization growthforecast, notifier, ikachan, .... Real-time aggregation datacounter, numeric-counter, numeric-aggregator, .. Real-time processing parser, exec_filter, .... 12年11月8日木曜日
  4. 4. Log Collection ! 12年11月8日木曜日
  5. 5. Fluentd as log collector Many many output plugins for various storages file, file-alternative mongo, couch, cassandra, redis, s3, .... Hadoooooooooooooooooooooooooooooooooooooop 12年11月8日木曜日
  6. 6. Fluentd with HDFS To write data on HDFS: Java native protocol: HDFSClient.java hadoop fs -put libhdfs and its binding (like scribed) Cloudera Hoop (2011/07-) +WebHDFS (Apache 1.0-), +HttpFs (Apache 2.0-) 12年11月8日木曜日
  7. 7. fluent-plugin-webhdfs Output plugin to write data into HDFS Supports WebHDFS and HttpFs First release: 2012/05/20 by tagomoris v0.1.0 bundled within td-agent v1.1.10 (or later) 12年11月8日木曜日
  8. 8. WebHDFS HTTP REST API of HDFS Clients communicate all of NameNode and DataNodes (like HDFSClient) NameNode DataNode Client DataNode DataNode HTTP 12年11月8日木曜日
  9. 9. HttpFs Proxy server 'httpfs', provides REST API for HDFS Same method set with WebHDFS (not like Hoop) Clients communicate with httpfs server only NameNode DataNode httpfs Client server DataNode HTTP Java Native DataNode 12年11月8日木曜日
  10. 10. WebHDFS or HttpFs WebHDFS: Peer-to-Peer communication Jetty based HTTP server High throughput and stability HttpFs: Proxyed and Centralized communication Tomcat based HTTP server Simple network topology Relatively low performance and SPOF 12年11月8日木曜日
  11. 11. Configuration: WebHDFS Use Apache 1.0.0(or later), CDH3u5 or CDH4(or later) In Namenode/Datanode dfs.webhdfs.enabled=true dfs.support.append=true (only CDH3u5 ?) dfs.support.broken.append=true (only CDH3u5 ?) In fluent-plugin-webhdfs (type webhdfs) host hostname.of.namenode port 50070 path /hdfs/access.%Y%m%d_%H.${hostname}.log 12年11月8日木曜日
  12. 12. WebHDFS in NHN Japan BEFORE: 1400 Timeouts/day with Hoop Tue Aug 14 15:04:34 2012 +0900 "fix to use webhdfs to write into hdfs" "2012-08-14 15:08:18 +0900: starting fluentd-0.10.25" Wed Aug 15 13:11:04 2012 +0900 "fix timeouts for busy AM2-5" AFTER: 130 Timeouts from 08/16 to 11/07 1.2-1.5 TB/day from 10 fluentd nodes 12年11月8日木曜日
  13. 13. CONCLUSION 1 WebHDFS is good enough for: continuous appending into log file daily operations to move/remove/copy/head/tail over client libraries (and your scripts) Fluentd and td-agent is good enough for: log collector before Hadoop/HDFS 12年11月8日木曜日
  14. 14. break 12年11月8日木曜日
  15. 15. fluent-plugin-webhdfs commit log Thu May 17 18:20:15 2012 on 'fluent-plugin-webhdfs' "writing code": in fact, no lines of ruby code.... Sun May 20 19:01:26 2012 on 'xxxxx' (some commits) Sun May 20 19:35:34 2012 on 'fluent-plugin-webhdfs' "fix typo": tagged as v0.0.1 12年11月8日木曜日
  16. 16. 30min!? fluent-plugin-webhdfs 120 lines (including blank line and 'end') 65 lines of configurations very few lines of actual code WebHDFS operations by 'webhdfs' gem Output formatting by 'PlainTextFormatterMixin' 12年11月8日木曜日
  17. 17. webhdfs gem commit log Sun May 20 17:00:57 2012 (15 commits) Sun May 20 19:01:26 2012 "v0.3: add WebHDFS::Client" 12年11月8日木曜日
  18. 18. fluent-mixin-* fluent-mixin-plaintextformatter output text data formatter webhdfs, file-alternative, hoop fluent-mixin-config-placeholders provide placeholders like '${hostname}', '${uuid}' in configurations webhdfs, ping-message 12年11月8日木曜日
  19. 19. CONCLUSION 2 Output plugins have many (complex) problems: communication, formatting, configuration formats, ... We CAN/MUST depends on existing GEMS! We SHOULD write fluent-mixin gems for other plugin developers! many features/codes may be shared by many plugins unified syntax/features over plugins 12年11月8日木曜日
  20. 20. Questions? Thanks! photo: crouton thanks to @kbysmnr 12年11月8日木曜日

×