Downloaded 91 times
Uploaded byShu Ting Tseng
Fluentd - Flexible, Stable, Scalable
Fluentd is an open-source data collector that centralizes data collection and consumption for better usage and understanding. It features a pluggable architecture with various input, output, and filter plugins, enabling integration with systems like Elasticsearch and MongoDB. The document also discusses scalability, stability, and practical use cases for Fluentd within Gogolook's data team.
More Related Content
Similar to Fluentd - Flexible, Stable, Scalable
Fluentd - Flexible, Stable, Scalable
- 1.
- 2. ho am I Suiting (@suitingtseng) Gogolook Inc. Data Team
- 3.
- 4. What is Fluentd? •Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. • Treasure Data: td-agent
- 5. What is Fluentd? •Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. • Treasure Data: td-agent
- 6. What is alog?
- 7. Log definition Time +Tag + Content
- 8.
- 9. How? • Lightweight: C+ Ruby + MessagePack • Pluggable architecture • Built-in Reliability
- 10. Input plugins • forward •tail • AWS Simple Queue Service • AWS CloudWatch
- 11. input: tail $ cat /etc/td-‐agent/conf.d <source> type tail path /var/log/nginx/access.log pos_file /var/log/td-‐agent/httpd-‐access.log.pos tag nginx.access </source> <match nginx.access> blah blah </match>
- 12. input: forward $ cat /etc/td-‐agent/conf.d <source> type forward port 24224 </source> <match flask.index> blah blah </match>
- 13. input: forward $ cat ~/example.py from fluent import sender from fluent import event sender.setup('flask', host='localhost', port=24224) event.Event("index", { "user": "foo", "token": "bar", "action": "POST" })
- 14. Output plugins • forward •copy • Elasticsearch / MongoDB • statsd / influxDB / graphite • S3 / GCS / BigQuery
- 15. output: elasticsearch $ cat /etc/td-‐agent/conf.d <source> foo bar tag nginx.access </source> <match nginx.access> type elasticsearch hosts es-‐host1,es-‐host2 index_name nginx type_name access flush_interval 60s </match>
- 16. output: splunk $ cat /etc/td-‐agent/conf.d <source> foo bar tag nginx.access </source> <match nginx.access> type splunk hosts splunk-‐host1 </match>
- 17. Filter plugins • grok •grep • record-modifier / record-reformer • geoip
- 18.
- 19. Buffer example $ cat /etc/td-‐agent/conf.d <source> foo bar tag nginx.access </source> <match nginx.access> type splunk hosts splunk-‐host1 buffer_chunk_limit 10m buffer_queue_limit 1000 flush_interval 5m </match>
- 21. Scalability • Scale up:multi-process plugin • Scale out: out-forward plugin
- 22. App + Fluentd Fluentd Elastic search Elastic search Elastic search Elastic search App+ Fluentd App + Fluentd
- 23.
- 24. Fluentd Elastic search Elastic search Elastic search Elastic search Fluentd Fluentd Load balance App + Fluentd App+ Fluentd App + Fluentd Auto scaling group
- 25. Stability • Auto retry •Persistent file buffer • At-most-once delivery
- 26. Message Delivery • At-most-once:data may be lost • At-least-once: data may be duplicated • Exactly-once: perfect
- 27. Idempotent • HTTP PUT •Maintain a unique id in application level or • Concatenate (instance-id, time, ….) as id
- 28. Gogolook use cases •MongoDB, nginx log • API, worker log • Monitor • Benchmark
- 29.
- 30.
- 31.
- 32.
- 33. Benchmark? FluentdApp + FluentdDB Local files
- 34. Benchmark? FluentdApp + FluentdDB Local files
- 35.