Elastic gives the flexibility to use a variety of data ingestion methods. Extending Elastic’s ingestion methods by using integrations, is a key feature of Elastic Stack to provide additional functionality. But what tools are provided to the integration developers in order to be more productive regarding the creation of such integrations? And how do we approach the key factors of automation and devops concepts as part of the development? In this session we will cover the following: What is an Elastic Integration Elastic-package tool: How to build a new integration How Elastic Agent leverages the integrations to enable new data inputs Make your builds alive: GitOps example functionality From integrations’ development to advanced template building to match dynamic environments (Kubernetes) needs Speakers: Andreas Gkizas: Principal Software Engineer @ Elastic | Christos Markou: Senior Software Engineer @ Elastic

1. 1
Elastic Meetup Athens | June 2023
Andreas Gkizas| Engineering @Elastic
Christos Markou | Engineering @Elastic
Streamlining Elastic integrations
development

2. About us…
Christos Markou
Elastic Cloud Native Observability
Andreas Gkizas

3. Main Problem
• What I need to do to collect my logs and metrics from my system! I need to
store them (hope in Elastic…) and later retrieve

4. Agenda
• What is an Elastic Integration
• Elastic-package tool: How to build a new integration
• How Elastic Agent leverages the integrations to enable new data inputs
• Make your builds alive: GitOps example functionality to automate
Observability steps

5. Store, Search, &
Analyze
Visualize &
Manage
Ingest
Elastic Stack
Kibana
Elasticsearch
Beats/Elastic-Agent Logstash
Elastic Stack

6. Store, Search, &
Analyze
Visualize &
Manage
Ingest
Elastic Stack
Kibana
Elasticsearch
Beats/Elastic-Agent Logstash
Elastic Stack
Metrics
Logs
Indexing/Mapping
Visualisations
1
2
3
1 Collection
2 Ingestion
3 Visualisation

7. • What is an Elastic Integration

8. • Elastic Integration = a set of definitions: defines how to observe a specific
product with the Elastic Stack.
Defines:
• Configuration for the Elastic Agent
• Assets (such as Kibana dashboards/visualisations)
• Elasticsearch index mappings and templates for the Elastic Stack
• Documentation about the package
• Tests to ensure that it is functioning as expected
Elastic Integration
Elastic Integration = Elastic Package

9. Elastic Integrations
https://docs.elastic.co/en/integrations/redis
https://github.com/elastic/integrations

10. • Elastic Packages have a certain yaml-structure.
• Built Elastic Integrations are stored in the Package Storage repository
• This structure is described by the Package Specification
(https://github.com/elastic/integrations/#package-spec).
Elastic Integration
Elastic Integration = Elastic Package

11. Integration Spec
/data_stream
<name_of_data_stream>
/agent
/fields
/_dev
/kibana
/docs
/img
manifest.yml
changelog.yml
Structure of package folder

12. • Elastic-package tool: How to build a new integration

13. Elastic Package
https://github.com/elastic/elastic-package/

14. Helps to: generate structure, lint, format, test and build your packages.
• ./elastic-package build
• ./elastic-package stack up -d -vvv --version=8.7.1
• ./elastic-package test system --data-streams pod -v
Elastic Package
Go cli tool

15. Demo of elastic-package
🤞

16. • How Elastic Agent leverages the integrations to enable new data
inputs

17. Getting your data into Elastic
You need Elastic Agent + Integrations

18. Getting your data into Elastic
• Similar functionality to Beats for log collection and host monitoring
• Elastic Agent has some distinct advantages over Beats
[+]
• Easier to deploy and manage
• Easier to configure
• Central management
Elastic Agent
What is Elastic Agent
2 running modes:
● Fleet managed
● Standalone

19. Elastic Agent on Infrastructure
One Agent per node
Elastic Agent
Elastic Agent Elastic Agent
node-1 node-2 node-3
Fleet Server

20. Elastic Agent Configuration

21. Ingest configuration
Type of integration
Shipper to use
Metrics endpoint
Integration specific settings

22. Elastic Agent + Integrations
Integration Policy(=configuration) = Integration specific configuration
Elastic Agent Policy(=configuration) = Agent configuration, which consists of multiple
integration policies.
What is Policy → Policy == Configuration

23. Configuring Elastic Agent (managed by user)
• Standalone Elastic Agents are manually
configured and managed locally on the
systems where they are installed.
• They are useful when you are not
interested in centrally managing agents in
Fleet, either due to your company’s
security requirements, or because you
prefer to use another configuration
management system.

24. • Make your builds alive: GitOps example functionality

25. From UI to GitOps This is UI

26. … to GitOps
All config files should
live inside Github
Demonstrate
automation tools
that will replace user
steps

27. Demo for Automated
Deployment 🤞
https://github.com/gizas/automatek8s/blob/TestBranch/INSTALLATION_GUIDE.md

28. Takeaways
Integration + Elastic Package
Elastic Agent
GitOps is possible

29. WE WOULD TO KEEP HEARING FROM
YOU
https://discuss.elastic.co/c/beats
https://github.com/elastic/beats
https://discuss.elastic.co/c/elastic-stack/elastic-agent
https://github.com/elastic/elastic-agent
https://github.com/elastic/integrations

30. Q&A