Download as PDF, PPTX
Uploaded bySadayuki Furuhashi
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container Era discusses how to effectively collect and analyze logs and metrics in microservices-based container environments. It introduces Fluentd as a centralized log collection service that supports pluggable input/output, buffering, and aggregation. Fluentd allows collecting logs from containers and routing them to storage systems like Kafka, HDFS and Elasticsearch. It also supports parsing, filtering and enriching log data through plugins.
More Related Content
![[245] presto 내부구조 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/245presto-150915054242-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Logging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
- 1. Logging for ProductionSystems in The Container Era Sadayuki Furuhashi Founder & Software Architect DOCKER MOUNTAIN VIEW
- 2. A little aboutme… Sadayuki Furuhashi github: @frsyuki A founder of Treasure Data, Inc. located in Silicon Valley. Fluentd - Unifid log collection infrastracture Embulk - Plugin-based ETL tool OSS projects I founded: An open-source hacker.
- 3. It's like JSON. butfast and small. A little about me…
- 4. The Container Era ServerEra Container Era Service Architecture Monolithic Microservices System Image Mutable Immutable Managed By Ops Team DevOps Team Local Data Persistent Ephemeral Log Collection syslogd / rsync ? Metrics Collection Nagios / Zabbix ?
- 5. Server Era ContainerEra Service Architecture Monolithic Microservices System Image Mutable Immutable Managed By Ops Team DevOps Team Local Data Persistent Ephemeral Log Collection syslogd / rsync ? Metrics Collection Nagios / Zabbix ? The Container Era How should log & metrics collection be done in The Container Era?
- 6.
- 7. The traditional logrotate+ rsync on containers Log Server Application Container A File FileFile Hard to analyze!! Complex text parsers Application Container C File FileFile Application Container B File FileFile High latency!! Must wait for a day Ephemeral!! Could be lost at any time
- 8. Server 1 Container A Application ContainerB Application Server 2 Container C Application Container D Application Kafka elasticsearch HDFS Container Container Container Container Small & many containers make storages overloaded Too many connections from micro containers!
- 9. Server 1 Container A Application ContainerB Application Server 2 Container C Application Container D Application Kafka elasticsearch HDFS Container Container Container Container System images are immutable Too many connections from micro containers! Embedding destination IPs in ALL Docker images makes management hard
- 10. Combination explosion withmicroservices requires too many scripts for data integration LOG script to parse data cron job for loading filtering script syslog script Tweet- fetching script aggregation script aggregation script script to parse data rsync server
- 11. A solution: centralizedlog collection service LOG Log Service
- 12. The centralized logcollection service LOG
- 13. The centralized logcollection service LOG We Released! (Apache License)
- 14. What’s Fluentd? Simple core +Variety of plugins Buffering, HA (failover), Secondary output, etc. Like syslogd AN EXTENSIBLE & RELIABLE DATA COLLECTION TOOL
- 15. How to collectlogs from Docker containers
- 16. Text logging with--log-driver=fluentd Server Container App FluentdSTDOUT / STDERR docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 { “container_id”: “ad6d5d32576a”, “container_name”: “myapp”, “source”: stdout }
- 17. Metrics collection withfluent-logger Server Container App Fluentd from fluent import sender from fluent import event sender.setup('app.events', host='localhost') event.Event('purchase', { 'user_id': 21, 'item_id': 321, 'value': '1' }) tag = app.events.purchase { “user_id”: 21, “item_id”: 321 “value”: 1, } fluent-logger library
- 18. Logging methods foreach purpose • Collecting log messages > --log-driver=fluentd • Application metrics > fluent-logger • Access logs, logs from middleware > Shared data volume • System metrics (CPU usage, Disk capacity, etc.) > Fluentd’s input plugins (Fluentd pulls those data periodically)
- 19.
- 20. Server 1 Container A Application ContainerB Application Server 2 Container C Application Container D Application Kafka elasticsearch HDFS Container Container Container Container Primitive deployment… Too many connections from many containers! Embedding destination IPs in ALL Docker images makes management hard
- 21. Server 1 Container A Application ContainerB Application Fluentd Server 2 Container C Application Container D Application Fluentd Kafka elasticsearch HDFS Container Container Container Container destination is always localhost from app’s point of view Source aggregation decouples config from apps
- 22. Server 1 Container A Application ContainerB Application Fluentd Server 2 Container C Application Container D Application Fluentd active / standby / load balancing Destination aggregation makes storages scalable for high traffic Aggregation server(s)
- 23. Aggregation servers • Loggingdirectly from microservices makes log storages overloaded. > Too many RX connections > Too frequent import API calls • Aggregation servers make the logging infrastracture more reliable and scalable. > Connection aggregation > Buffering for less frequent import API calls > Data persistency during downtime > Automatic retry at recovery from downtime
- 24.
- 25. Internal Architecture (simplified) Plugin InputFilter Buffer Output Plugin Plugin Plugin 2012-02-04 01:33:51 myapp.buylog{ “user”:”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } Time Tag Record
- 26. Architecture: Input Plugins HTTP+JSON(in_http) File tail (in_tail) Syslog (in_syslog) … Receive logs Or pull logs from data sources In non-blocking manner Plugin Input
- 27. Filter Architecture: Filter Plugins Transformlogs Filter out unnecessary logs Enrich logs Plugin Encrypt personal data Convert IP to countries Parse User-Agent …
- 28. Buffer Architecture: Buffer Plugins Plugin Improveperformance Provide reliability Provide thread-safety Memory (buf_memory) File (buf_file)
- 29. Architecture: Output Plugins Output Writeor send event logs Plugin File (out_file) Amazon S3 (out_s3) MongoDB (out_mongo) …
- 30. Buffer Architecture: Buffer Plugins Chunk Plugin Improveperformance Provide reliability Provide thread-safety Input Output Chunk Chunk
- 31. Retry Error Retry Batch Stream Error Retry Retry Divide &Conquer for retry
- 32. Divide & Conquerfor recovery Buffer (on-disk or in-memory) Error Overloaded!! recovery recovery + flow control queued chunks
- 33.
- 34. Streaming from Apache/Nginxto Elasticsearch in_tail /var/log/access.log /var/log/fluentd/buffer but_file
- 35. Error Handling andRecovery in_tail /var/log/access.log /var/log/fluentd/buffer but_file Buffering for any outputs Retrying automatically With exponential wait and persistence on a disk and secondary output
- 36. Tailing & parsingfiles Supported built-in formats: Read a log file Custom regexp Custom parser in Ruby • apache • apache_error • apache2 • nginx • json • csv • tsv • ltsv • syslog • multiline • none pos fileevents.log ? (your app)
- 37. Out to MultipleLocations Routing based on tags Copy to multiple storages buffer access.log in_tail
- 38. Example configuration forreal time batch combo
- 39. Data partitioning bytime on HDFS / S3 access.log buffer Custom file formatter Slice files based on time 2016-01-01/01/access.log.gz 2016-01-01/02/access.log.gz 2016-01-01/03/access.log.gz … in_tail
- 40. 3rd party inputplugins dstat df AMQL munin jvmwatcher SQL
- 41. 3rd party outputplugins AMQL Graphite
- 42. Real World UseCases
- 43. Microsoft Operations Management Suiteuses Fluentd: "The core of the agent uses an existing open source data aggregator called Fluentd. Fluentd has hundreds of existing plugins, which will make it really easy for you to add new data sources." Syslog Linux Computer Operating System Apache MySQL Containers omsconfig (DSC) PS DSC Providers OMI Server (CIM Server) omsagent Firewall/proxy OMSService Upload Data (HTTPS) Pull configuration (HTTPS)
- 44. Atlassian "At Atlassian, we'vebeen impressed by Fluentd and have chosen to use it in Atlassian Cloud's logging and analytics pipeline." Kinesis Elasticsearch cluster Ingestion service
- 45. Amazon web services Thearchitecture of Fluentd (Sponsored by Treasure Data) is very similar to Apache Flume or Facebook’s Scribe. Fluentd is easier to install and maintain and has better documentation and support than Flume and Scribe. Types of DataStoreCollect Transactional • Database reads & write (OLTP) • Cache Search • Logs • Streams File • Log files (/val/log) • Log collectors & frameworks Stream • Log records • Sensors & IoT data Web Apps IoTApplicationsLogging Mobile Apps Database Search File Storage Stream Storage
- 46.