Elastic Ingest Manager is one of the exciting features, let us master it together before the next release - Beats overview - Elastic-Agent overview - Integrations - Data Streams - Q & A

1. Elastic 101 - Ingest Manager
ENG. ISMAIL ANJRINI
ELASTIC CERTIFIED ENGINEER
COLLECT MORE, SO YOU CAN KNOW MORE
Elastic-Saudi-Arabia 1

2. COLLECT MORE, SO YOU CAN KNOW MORE 2
BEATS

3. COLLECT MORE, SO YOU CAN KNOW MORE 3

4. COLLECT MORE, SO YOU CAN KNOW MORE 4
ELASTIC AGENT - OVERVIEW

5. COLLECT MORE, SO YOU CAN KNOW MORE 5

6. COLLECT MORE, SO YOU CAN KNOW MORE 7
DATA STREAMS

7. COLLECT MORE, SO YOU CAN KNOW MORE 8
Time series
ILM
Templates
.index-01 .index-02 .index-0N
Data stream
@timestamp
append only
ILM is optional
Template is for data stream

8. COLLECT MORE, SO YOU CAN KNOW MORE 9
.index-01 .index-02 .index-0N
Data stream
Search
Search
Async search
Multi search
Field capabilities
EQL search

9. COLLECT MORE, SO YOU CAN KNOW MORE 10
.index-01 .index-02 .index-0N
Data stream
Write POST /ds/_doc/
PUT /ds/_create/<_id>
PUT /ds/_doc/<_id>
PUT /ds/_bulk
Ingest pipeline
Update by query API
Delete by query API
Delete documents in a backing index
Update documents in a backing index
_id
primary term
seq_no
_id

10. COLLECT MORE, SO YOU CAN KNOW MORE 11
mapping
template
ds
new index
all indexes
write_index_only
Add new field

11. COLLECT MORE, SO YOU CAN KNOW MORE 12
dynamic
template
ds
new index
all indexes
Dynamic settings
index.hidden
index.number_of_replicas

12. COLLECT MORE, SO YOU CAN KNOW MORE 13
static
template
ds
new index
all indexes
reindex
Static settings
index.number_of_shards
index.number_of_routing_shards

13. COLLECT MORE, SO YOU CAN KNOW MORE 14
Data stream Alias
Data streams only accept append-only writes Accepts write/delete/update
data_stream property in template
Kibana automatically generates index patterns based on data
streams, and identity the timestamp field
An index can only be part of a single data stream An index can has more than one alias
Index is hidden
PUT /_data_stream/[name]
POST /_aliases
GET /_data_streams/[name] GET [alias_name]
DELETE /_data_stream/[name]
Filtered alias
POST /_aliases
Elasticsearch 7.9 Elasticsearch 0.90

14. COLLECT MORE, SO YOU CAN KNOW MORE 15
AGENT CONFIGURATION

15. COLLECT MORE, SO YOU CAN KNOW MORE 16

16. COLLECT MORE, SO YOU CAN KNOW MORE 17
INTEGRATION

17. COLLECT MORE, SO YOU CAN KNOW MORE 18

18. COLLECT MORE, SO YOU CAN KNOW MORE 19

19. COLLECT MORE, SO YOU CAN KNOW MORE 20

20. COLLECT MORE, SO YOU CAN KNOW MORE 21

21. COLLECT MORE, SO YOU CAN KNOW MORE 22
ADD AGENT

22. COLLECT MORE, SO YOU CAN KNOW MORE 23
Fleet elastic-agent enroll install
.elastic-agent enroll http://localhost:5601 bXNXenhYUUJtcTZCYW9zSW5ib1c6TGJUdUEyNXJTcXFBUUY3a09BaF9OQQ==
.install-service-elastic-agent.ps1
https://ela.st/download-elastic-agent

23. COLLECT MORE, SO YOU CAN KNOW MORE 24
elastic-agent
filebeat
metricbeat
%elastic-agentdatainstall

24. COLLECT MORE, SO YOU CAN KNOW MORE 25
ADD INTEGRATION

25. COLLECT MORE, SO YOU CAN KNOW MORE 26

26. COLLECT MORE, SO YOU CAN KNOW MORE 27

27. COLLECT MORE, SO YOU CAN KNOW MORE 28

28. COLLECT MORE, SO YOU CAN KNOW MORE 29

29. COLLECT MORE, SO YOU CAN KNOW MORE 30

30. COLLECT MORE, SO YOU CAN KNOW MORE 31
DATA STREAM STRUCTURE

31. COLLECT MORE, SO YOU CAN KNOW MORE 32
logs-generic-default
type dataset namespace
metrics-elastic.agent.filebeat-default
.ds-metrics-system.memory-default-000001
.ds ds name generation

32. COLLECT MORE, SO YOU CAN KNOW MORE 33