Eduardo Silva discusses the importance of unifying events and logs in cloud environments, emphasizing the benefits of effective logging for application status, debugging, and analytics. He introduces Fluentd and Fluent Bit as open-source data collectors that enable high-performance data collection from multiple sources, tailored for IoT and embedded environments. The presentation highlights logging considerations, architectural structures, and the growing relevance of logging in the Internet of Things landscape.

1. Eduardo Silva
eduardo@treasuredata.com
@edsiper
Unifying Events & Logs
into the Cloud
October 06, 2015
StartechConf, Santiago

2. About Me
Eduardo Silva
●
Github & Twitter @edsiper
●
Personal Blog http://edsiper.linuxchile.cl
Treasure Data
●
Open Source Engineer
●
Fluentd / Fluent Bit http://github.com/fluent
Projects
●
Monkey HTTP Server http://monkey-project.com
●
Duda I/O http://duda.io

3. Logging

4. Logging Matters
Pros
●
Application status
●
Debugging
●
General information about anomalies: errors
●
Troubleshooting / Support
●
Local or Remote (network)

5. Logging Matters
From a business point of view
●
Input data Analytics→
●
User interaction / behaviors
●
Improvements

6. Assumptions

7. Logging Matters
Assumptions
●
I have enough disk space
●
I/O operations will not block
●
Log messages are human readable
●
My logging mechanism scale

8. Logging Matters
Assumptions
Basically, yeah.. it should work.

9. Concerns

10. Logging Matters
Concerns
●
Logs increase = data increase
●
Message format get more complex
●
Did the Kernel flush the buffers ? (sync(2))
●
Multi-thread application ?, locking ?
●
Multiple Applications = Multiple Logs

11. Logging Matters
Concerns
If Multiple Applications = Multiple logs
Multiple Hosts x Multiple Applications = ???

12. OK, so:
1. Logging matters
2. It's really beneficial
3. but...

13. It needs to
be done right.

14. Logging
Common sources & inputs
●
Application Logs
●
Apache
●
NginX
●
Syslog (-ng)
●
Custom applications / Languages
●
C, Ruby, Python, PHP, Perl, NodeJS, Java, etc.

15. In a galaxy
not so far away...

17. How to parse/store
multiple data sources ?
note: performance matters!

19. Fluentd is an open
source data collector
It let's you unify the data collection for
a better use and understanding of data.

20. before

21. after

22. Fluentd
Highlights
●
High Performance
●
Built-in Reliability
●
Structured Logs
●
Pluggable Architecture
●
More than 300 plugins! (input/filtering/output)

23. Fluentd
Architecture

24. Fluentd
Internals simplified

25. Fluentd
Input plugins

26. Fluentd
Output plugins

27. Fluentd
Buffer plugins

28. Fluentd
Buffer plugins

30. M x N M + N→

31. Fluentd
Simple Forwarding

32. Fluentd
Simple Forwarding: configuration
# logs from a file # store logs to MongoDB
<source> <match backend.*>
type tail type mongo
path /var/log/httpd.log database fluent
format apache2 collection test
tag backend.apache </match>
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>

33. Fluentd
Less Simple Forwarding

34. Fluentd
Lambda Architecture

35. Fluentd
# logs from a file # store logs to MongoDB
<source> <match *.*>
type tail type copy
path /var/log/httpd.log <store>
format apache2 type elasticsearch
tag backend.apache logstash_format true
</source> </store>
# logs from client libraries <store>
<source> type webhdfs
type forward host 192.x.y.z
port 24224 port 50070
</source> path /path/to/hdfs
</store>
</match>

36. Who uses Fluentd
in production ?

39. We collect
1M events per second !

40. Internet of Things

41. Internet of Things
Facts
●
IoT will grow to many billions of devices over the
next decade.
●
Now it's about device to device connectivity.
●
Different frameworks and protocols are emerging.
●
It needs Logging.

42. Internet of Things
Alliances
Vendors formed alliances to join forces and develop
generic software layers for their products:

43. Internet of Things
Solutions provided
Alliance Framework
→
→

44. IoT & Big Data
Analytics
IoT requires a generic solution to collect events and
data from different sources for further analysis.
Data can come from a specific framework, radio device,
sensor or other. How do we collect and unify data
properly ?

45. @fluentbit

46. Fluent Bit is an open source
data collector
It let's you collect data from IoT/Embedded
devices and transport It to third party
services.

47. Fluent Bit
Targets
●
Services
●
Sensors / Signals / Radios
●
Operating System information
●
Automotive / Telematics

48. Fluent Bit
Requirements
IoT and Embedded environment requires special handling,
specifically on performance and resource utilization:
●
Lightweight
●
Written in C Language
●
Customizable, pluggable architecture
●
Full integration with Fluentd

49. Fluent Bit
Integration

50. Fluent Bit
Direct Output

51. Containers

53. Docker
Logging driver
●
Docker v1.6 released the concept of logging drivers
●
Route container output
●
Fluentd ?

54. Docker

55. Docker v1.8
Fluentd Logging driver!

56. Docker
Data Stream

57. Docker
Data Stream

59. NodeJS
Fluent-Logger (NPM)

60. We Love Data!
Thank you!
●
http://fluentd.org
●
http://fluentbit.io
●
https://docs.docker.com/reference/logging/fluentd/
●
http://github.com/fluent/fluentd