Fluentd v1 provides major improvements over v0.12 including nanosecond event time resolution, multi-core support, Windows support, and new plugin APIs. The new plugin APIs provide well-controlled lifecycles and integrate all output plugins. v1 also introduces a server engine based supervisor, dynamic buffering capabilities, and various plugin helpers. While maintaining compatibility with v0.12 plugins, v1 focuses on ease of use, stability, performance and flexibility.
Keynote on Fluentd Meetup Summer
Related Slide
- Fluentd ServerEngine Integration & Windows Support http://www.slideshare.net/RittaNarita/fluentd-meetup-2016-serverengine-integration-windows-support
- Fluentd v0.14 Plugin API Details http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details
Query Pulsar Streams using Apache FlinkStreamNative
Both Apache Pulsar and Apache Flink share a similar view on how the data and the computation level of an application can be “streaming-first” with batch as a special case streaming. With Apache Pulsar’s Segmented-Stream storage and Apache Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies to provide elastic data processing at massive scale, and build a real streaming warehouse.
In this talk, Sijie Guo from the Apache Pulsar community will share the latest integrations between Apache Pulsar and Apache Flink. He will explain how Apache Flink can integrate and leverage Pulsar’s built-in efficient schemas to allow users of Flink SQL query Pulsar streams in realtime.
Lessons from managing a Pulsar cluster (Nutanix)StreamNative
In this presentation, we will cover:
- How to performance test and optimize a Pulsar cluster. We will present how we load tested Pulsar with locust and, following this, how we tuned our configurations for our use cases.
- Event sourcing pattern with Apache Pulsar. Avro schema usage, compatibility choices and schema evolution on pulsar topics that worked for us.
- Bonus: How we source Apache Flink from apache pulsar and run our workflows.
By attending this webinar, you can expect to come away with:
- How to performance test a Pulsar cluster for your use case.
- How to leverage the highly configurable broker and Bookkeeper to suit your needs.
- Event sourcing patterns on top of Apache Pulsar.
- Avro schema usage, compatibility choices, and evolution.
- Familiarise with pulsar connector for Flink and possible use cases.
Introducing HerdDB - a distributed JVM embeddable database built upon Apache ...StreamNative
We will introduce HerdDB a distributed database written in Java.
We will see how a distributed database can be built using Apache BookKeeper as write-ahead commit log.
Introducing Kafka-on-Pulsar: bring native Kafka protocol support to Apache Pu...StreamNative
Kafka-on-Pulsar has been one of the most anticipated features in the Pulsar ecosystem. The Kafka-on-Pulsar project was initiated by StreamNative and the OVHCloud team quickly joined the project to collaborate on its development. Kafka-on-Pulsar enables Kafka applications to leverage Pulsar’s powerful features, such as streamlined operations with enterprise-grade multi-tenancy, without modifying code.
In this webinar, Sijie Guo, from StreamNative, and Pierre Zemb, from OVHCloud, will introduce KoP and discuss the following:
1. What are the key benefits?
2. What is the protocol handler and how does it work?
3. How KoP is implemented?
4. What are the new use cases it unlocks?
5. Watch a Live Demo!
Integrating Apache Pulsar with Big Data EcosystemStreamNative
In Apache Pulsar Beijing Meetup, Yijieshen gave a presentation of the current state of Apache Pulsar integrating with Big Data Ecosystem. He explains why and how Pulsar fits into current big data computing and query engines, and how Pulsar integrates with Spark, Flink and Presto for unified data processing system.
Both Apache Pulsar and Apache Flink share a similar view on how the data and the computation level of an application can be “streaming-first” with batch as a special case streaming. With Apache Pulsar’s Segmented-Stream storage and Apache Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies to provide elastic data processing at massive scale, and build a real streaming warehouse.
In this talk, Sijie Guo from Apache Pulsar community will given an overview of Apache Pulsar and how it provides the unified data view to fully leverage Apache Flink unified computation runtime for elastic data processing. He will share the latest integrations between Apache Pulsar and Apache Flink, especially around effectively-once processing and schema integration.
Rust is a relatively new programming language promising performance, reliability and productivity. In order to learn Rust Gerard converted a micro service he already wrote in Clojure and Kotlin, both JVM languages, to Rust. The microservice processes two kinds of events, and uses PostgreSQL to keep state. As part of the conversion he wrote a library to use the Confluent Schema Registry with Rust.
What can be said about the promises of Rust compared to the JVM? Gerard will tell about his experiences in using Rust with Kafka, and presenting some benchmarks comparing the different languages. The focus of the benchmarks will be the end to end latency and resource usage. He will conclude with some remarks why it may or may not be a good idea to add some Rust to your Kafka.
In Apache Pulsar Meetup, Jia Zhai from StreamNative presents KoP (Kafka-on-Pulsar) which bring native Kafka protocol support on Pulsar broker. He gave a demo about how to use Kafka clients and Pulsar clients can work seamlessly on same data, and how Kafka Connectors can work on a Pulsar cluster.
Keynote on Fluentd Meetup Summer
Related Slide
- Fluentd ServerEngine Integration & Windows Support http://www.slideshare.net/RittaNarita/fluentd-meetup-2016-serverengine-integration-windows-support
- Fluentd v0.14 Plugin API Details http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details
Query Pulsar Streams using Apache FlinkStreamNative
Both Apache Pulsar and Apache Flink share a similar view on how the data and the computation level of an application can be “streaming-first” with batch as a special case streaming. With Apache Pulsar’s Segmented-Stream storage and Apache Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies to provide elastic data processing at massive scale, and build a real streaming warehouse.
In this talk, Sijie Guo from the Apache Pulsar community will share the latest integrations between Apache Pulsar and Apache Flink. He will explain how Apache Flink can integrate and leverage Pulsar’s built-in efficient schemas to allow users of Flink SQL query Pulsar streams in realtime.
Lessons from managing a Pulsar cluster (Nutanix)StreamNative
In this presentation, we will cover:
- How to performance test and optimize a Pulsar cluster. We will present how we load tested Pulsar with locust and, following this, how we tuned our configurations for our use cases.
- Event sourcing pattern with Apache Pulsar. Avro schema usage, compatibility choices and schema evolution on pulsar topics that worked for us.
- Bonus: How we source Apache Flink from apache pulsar and run our workflows.
By attending this webinar, you can expect to come away with:
- How to performance test a Pulsar cluster for your use case.
- How to leverage the highly configurable broker and Bookkeeper to suit your needs.
- Event sourcing patterns on top of Apache Pulsar.
- Avro schema usage, compatibility choices, and evolution.
- Familiarise with pulsar connector for Flink and possible use cases.
Introducing HerdDB - a distributed JVM embeddable database built upon Apache ...StreamNative
We will introduce HerdDB a distributed database written in Java.
We will see how a distributed database can be built using Apache BookKeeper as write-ahead commit log.
Introducing Kafka-on-Pulsar: bring native Kafka protocol support to Apache Pu...StreamNative
Kafka-on-Pulsar has been one of the most anticipated features in the Pulsar ecosystem. The Kafka-on-Pulsar project was initiated by StreamNative and the OVHCloud team quickly joined the project to collaborate on its development. Kafka-on-Pulsar enables Kafka applications to leverage Pulsar’s powerful features, such as streamlined operations with enterprise-grade multi-tenancy, without modifying code.
In this webinar, Sijie Guo, from StreamNative, and Pierre Zemb, from OVHCloud, will introduce KoP and discuss the following:
1. What are the key benefits?
2. What is the protocol handler and how does it work?
3. How KoP is implemented?
4. What are the new use cases it unlocks?
5. Watch a Live Demo!
Integrating Apache Pulsar with Big Data EcosystemStreamNative
In Apache Pulsar Beijing Meetup, Yijieshen gave a presentation of the current state of Apache Pulsar integrating with Big Data Ecosystem. He explains why and how Pulsar fits into current big data computing and query engines, and how Pulsar integrates with Spark, Flink and Presto for unified data processing system.
Both Apache Pulsar and Apache Flink share a similar view on how the data and the computation level of an application can be “streaming-first” with batch as a special case streaming. With Apache Pulsar’s Segmented-Stream storage and Apache Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies to provide elastic data processing at massive scale, and build a real streaming warehouse.
In this talk, Sijie Guo from Apache Pulsar community will given an overview of Apache Pulsar and how it provides the unified data view to fully leverage Apache Flink unified computation runtime for elastic data processing. He will share the latest integrations between Apache Pulsar and Apache Flink, especially around effectively-once processing and schema integration.
Rust is a relatively new programming language promising performance, reliability and productivity. In order to learn Rust Gerard converted a micro service he already wrote in Clojure and Kotlin, both JVM languages, to Rust. The microservice processes two kinds of events, and uses PostgreSQL to keep state. As part of the conversion he wrote a library to use the Confluent Schema Registry with Rust.
What can be said about the promises of Rust compared to the JVM? Gerard will tell about his experiences in using Rust with Kafka, and presenting some benchmarks comparing the different languages. The focus of the benchmarks will be the end to end latency and resource usage. He will conclude with some remarks why it may or may not be a good idea to add some Rust to your Kafka.
In Apache Pulsar Meetup, Jia Zhai from StreamNative presents KoP (Kafka-on-Pulsar) which bring native Kafka protocol support on Pulsar broker. He gave a demo about how to use Kafka clients and Pulsar clients can work seamlessly on same data, and how Kafka Connectors can work on a Pulsar cluster.
Continuous Integration, Continuous Delivery, workflow automation, more and more tools rely on a vibrant marketplace to share extensions that add new functionality to the systems.
Unfortunately, this means you’re putting trust into a LOT of different people and many of these extensions are built by professionals and enthusiasts in their limited spare time.
GitHub has such a marketplace for Actions and Azure DevOps one for Pipelines. But how do you know whether you can trust these extensions, how do you stay up to date and if you’re building these extensions yourself, how do you keep them secure and your users on the latest version?
Jesse takes us through some of the best practices from the perspective of an extension author, user as well as the administrator of the system that loads these extensions.
Continuous Integration, Continuous Delivery, workflow automation, more and more tools rely on a vibrant marketplace to share extensions that add new functionality to the systems.
Unfortunately, this means you’re putting trust into a LOT of different people and many of these extensions are built by professionals and enthusiasts in their limited spare time.
GitHub has such a marketplace for Actions and Azure DevOps one for Pipelines. But how do you know whether you can trust these extensions, how do you stay up to date and if you’re building these extensions yourself, how do you keep them secure and your users on the latest version?
Jesse takes us through some of the best practices from the perspective of an extension author, user as well as the administrator of the system that loads these extensions.
Using a revision control system that tracks changes in source code with ways to manage your code in separate branches and tag revisions as releases is a bare minimum for developers.
This presentation highlights the importance of using a version control system Subversion.
It is common for enterprises to give a high priority to security and compliance, as these are critical concerns for businesses of all sizes. One way that enterprises can address these concerns is by leveraging agent-based security and compliance tools.
This ppt explains about the solution which helps to centrally deploy agents to all SSM managed workloads that enable a variety of use cases, such as endpoint security, threat intelligence, software asset management, inventory, license management, etc.
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataGetInData
Did you like it? Check out our blog to stay up to date: https://getindata.com/blog
The webinar was organized by GetinData on 2020. During the webinar we explaned the concept of monitoring and observability with focus on data analytics platforms.
Watch more here: https://www.youtube.com/watch?v=qSOlEN5XBQc
Whitepaper - Monitoring ang Observability for Data Platform: https://getindata.com/blog/white-paper-big-data-monitoring-observability-data-platform/
Speaker: Albert Lewandowski
Linkedin: https://www.linkedin.com/in/albert-lewandowski/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
At Tuenti, we do two code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also that proper localization is applied, bundles are generated and files get deployed to hundreds of servers as fast and reliable as possible.
We use opensource tools like Mercurial, MySQL, Jenkins, Selenium, PHPUnit and Rsync among our own in-house ones, and have different development, testing, staging and production environments.
We had to fight with problems like statics bundling and versioning, syntax errors and of course the fact that we have +100 engineers working on the codebase, sometimes merging and releasing more than a dozen branches the same day. We also switched from Subversion to Mercurial to obtain more flexibility and faster branching operations.
With this talk we will explain the process of how code changes in ourcode repository end up in live code, detailing some practices and tips that we apply.
This release aims at addressing the long-awaited JDK 11 compatibility which enables enterprises to migrate their middleware solutions to the LTS JDK version. In addition, this release features new productivity improvements including built-in test framework with comprehensive tooling support from Integration Studio and support for cloud-native integration requirements.
This WSO2 Enterprise Integrator release brings new product components and features specifically targeted to help developers build and deploy container-native integration solutions easily.
Watch the webinar on-demand here: https://wso2.com/library/webinars/2020/01/whats-new-in-wso2-enterprise-integrator-december-2019-release/
Fluentd meetup dive into fluent plugin (outdated)N Masahiro
Fluentd meetup in Japan. I talked about "Dive into Fluent plugin".
Some contents are outdated. See this slide: http://www.slideshare.net/repeatedly/dive-into-fluentd-plugin-v012
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
3. Fluentd v0.12
• Old stable and widely used on production
• Input, Parser, Filter, Formatter, Buffer, Output plugins
• Known issues
• Event time is second unit
• No Windows support
• No multi core support
• Need to improve plugin API to support more various
use cases`
4. Fluentd v0.14
• Development version of v1
• Implemented New features
• New Plugin APIs
• Event Time with Nanosecond resolution
• ServerEngine based Supervisor
• Windows support
• Multicore support
• New Plugin Helpers & Plugin Storage
5. New Plugin APIs
• v1.0 Annoucement at CNCon + KubeCon NA.
• Stable announcement for APIs / features
• No breaking API changes in v1.x
• Compatible with v0.12 and v0.14
• exclude v0 config syntax and detach_process
• Latest version is v1.1.0: Jan 18, 2018
6. New Plugin APIs
• Input/Output plugin APIs w/ well-controlled lifecycle
• stop, shutdown, close, terminate
• Integrate all output plugin into Fluent::Plugin::Output
• New Buffer API for delayed commit and flexible chunking with metadata
• parallel/async "commit" operation for chunks
• For high latency case: forward’s at-least-once, issuing job, etc…
• Users can choose chunk keys by configuration for dynamic parameters
• Compatible w/ v0.12 plugins
• compatibility layer for traditional APIs
• it will be supported between v1.x versions
7. Router
buffer_chunk_limit
enqueue: exceed flush_interval
or buffer_chunk_limit
Key pattern:
- BufferedOutput
empty string or specified key
-ObjectBufferedOutput tag
-TimeSlicedOutput time slice
emit emit
Buffer
Queue
buffer_queue_limit
Output
OutputInput / Filter
Tag Time
Record Chunk
Chunk
Chunk Chunk
Chunk
key:foo
key:bar
key:baz
v0.12 buffer design
9. Buffer keys and placeholders
• Dynamic parameters for table name, object path and more
• We can embed time, tag and any field with placeholder
<match s3.**>
@type s3
aws_key_id "#{ENV['AWS_ACCESS_KEY']}"
aws_sec_key "#{ENV['AWS_SECRETA_KEY']}"
s3_bucket fluent-plugin-s3
path test/%Y/%m/${tag}/${key}/
<buffer time,tag,key>
timekey 3600
</buffer>
</match>
http://docs.fluentd.org/v1.0/articles/buffer-section
time: 2018-02-15 12:00:00 +0700
tag: “test”
record: {“key”:”hello”}
- Event sample
test/2018/2/test/hello/
- Generated “path”
10. Time with nanosecond
• For sub-second systems: Elasticsearch, InfluxData, etc…
• Fluent::EventTime
• behaves as Integer for v0.12’s second unit compatibility
• has methods to get sub-second resolution
• be serialized into msgpack using Ext type
• Fluent::Engine.now now returns EventTime, not Integer
• Fluentd core can handle both of Integer and EventTime as time
• compatible with older versions and software in eco-system
(e.g., fluent-logger, Docker logging driver)
11. ServerEngine based Supervisor
• ServerEngine is a framework for building robust server
• https://github.com/treasure-data/serverengine
• Replacing supervisor process with ServerEngine
• it has SocketManager to share listening sockets between
2 or more worker processes
• Replacing Fluentd's processing model from fork to spawn
• to support Windows environment
• Log rotation support
12. Windows support
• Fluentd and core plugins work on Windows
• Windows service registration is also supported
• http://docs.fluentd.org/v1.0/articles/install-by-msi
• Use HTTP RPC instead of signals
• https://github.com/fluent/fluent-plugin-windows-eventlog
• We can collect windows eventlog :)
13. Symmetric multi core processing
• 2 or more workers share a configuration file
• and share listening sockets via PluginHelper
• under a supervisor process (ServerEngine)
• Multi core scalability for huge traffic
• one input plugin for a tcp port, some filters and one
(or some) output plugin
• buffer paths are managed by Fluentd core. Need
root_dir and @id parameters
16. <worker N> directive
• To execute plugins under one process
• Good for non-multiprocess supported plugins like in_tail
in_tail/out_s3 works under worker 0
in_forward/out_kafka works
under multiprocess environment with
worker 1, worker 2, and worker 3
<worker 0>
<source>
@type in_tail
</source>
<match pattern>
@type s3
</match>
</worker>
<system>
workers 4
</system>
<source>
@type forward
</source>
<match pattern>
@type mongo
</match>
17. TLS/Authn/Authz support for forward plugin
• Support v1 forward protocol spec
• secure-forward is merged into built-in forward
• TLS w/ at-least-one semantics
• Simple authentication/authorization w/o SSL
• Different points
• secure-forward uses keep-alive, but forward doesn’t
• secure-forward uses thread per connection, but
forward uses cool.io, libev based IO.
http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released
18. Plugin Storage & Helpers
• Plugin Storage: new plugin type for plugins
• provides key-value storage to persistent intermediate status
• built-in plugins: in-memory, local file
• pluggable: 3rd party plugin to store data into storage
• storage-redis, storage-memcached
• Plugin Helpers:
• collections of utility methods for plugins
• fully integrated with test drivers to run test code after setup
phase of helpers (e.g. test started after created threads)
19. server helper: before
def start
@loop = Coolio::Loop.new
@handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log,
@delimiter, method(:on_message))
@loop.attach(@handler)
@thread = Thread.new(&method(:run))
end
def shutdown
@loop.watchers.each { |w| w.detach }
@loop.stop
@handler.close
@thread.join
end
def run
@loop.run
rescue => e
log.error "unexpected error", error: e
log.error_backtrace
end
def on_message(msg, addr)
# body
end
20. server helper: after
def start
server_create(:foo_server, @port, bind: @bind) { |data, conn|
# body
}
end
https://docs.fluentd.org/v1.0/articles/api-plugin-helper-server
21. record_accessor helper
• access / delete support for nested field
• e.g. parser’s key_name parameter uses this helper
• Provide two syntax for configuration
• $.field1.field2 == record[“field1”][“field2”]
• $[“field1”][“field2”] == record[“field1”][“field2”]
ra = record_accessor_create(”$.user.name”)
ra.call(record) # access record[”$.user”][”name”]
ra.delete(record) # delete record[”$.user”][”name”]
24. Other helpers
• Timer: one-shot / periodic timer
• Event Loop: Low-layer event loop
• Socket: TCP/UDP/TLS support
• Formatter/Parser: Manage parser/formatter plugins
• Chile Process: Manage process for exec like plugin
• etc…
https://docs.fluentd.org/v1.0/categories/plugin-helpers
25. v1.2.0
• Counter API: store metrics between processes
• Need for limit calculation in multi processes
• https://github.com/fluent/fluentd/pull/1857
• Backup feature for problematic chunks
• Improve retry mechanizm for bad records
• https://github.com/fluent/fluentd/issues/1856
26. Focus
• Easy to use
• Stability
• Performance
• Flexibility
• Avoid fat core
29. Treasure Agent 3 (td-agent 3)
• fluentd v1, Ruby 2.4, systemd support and latest components
• Latest version is 3.1.1: Dec 20, 2017
• 3.2.0 will be released in March
• Environments
• Add msi Windows package, Amazon Linux 2
• Remove CentOS 5, Ubuntu 10.04 support
30. Containers
• Docker
• Alpine and Debian for v0.12 and v1.x
• https://github.com/fluent/fluentd-docker-image
• Kubernetes DaemonSet
• Alpine and Debian for v0.12
• Debian for v1.x (WIP)
• https://github.com/fluent/fluentd-kubernetes-daemonset
• Need other container support?
31. Integrations
• Kafka
• kafka-connect-fluentd for high performance ingestion
• Promethuse
• fluent-plugin-prometheus to push / pull for prometheus
• Integrate internal metrics with monitor_agent
• gRPC?
• Distributed tracing?
32. Benchmark set (WIP)
• Check configuration and performance
• Current fluentd-benchmak is not enough
• Automated test
• Various combo: ruby, fluentd, plugins
• Collect metrics: CPU, Memory, etc…
• Running on: Docker, AWS, etc…
33. fluent-bit
• Lightweight agent written in C
• Running on lots of environment including
embedded systems with small resource
• Pluggable architecute: Input / Parser / Filter /
Buffer / Output
• fluent-bit is useful for forwarders with fluentd
in distributed logging
http://fluentbit.io/
34. Community
• Plugins / Libraries
• Thanks for maintaining the project
• Users
• Experts help new users
• Documentation
• Need feedback!