Get tips directly from the experts at Elastic about planning for, monitoring, and troubleshooting the Elastic Stack at scale. Elastic experts will share the tools, strategies, and architectures that can be used to ensure cluster health and performance. Learn about using tools like automated alerting to identify and remediate issues rapidly. Walk away armed with best practices for how to ensure both cluster and data resiliency.

1. 1
Managing
Elasticsearch at Scale
Michelle Sausa Ravi Kesarwani Chris Roberson
Principal Product Manager Principal Product Manager Principal Software Engineer

2. 2
This presentation and the accompanying oral presentation contain forward-looking statements, including statements
concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future
operations and expected performance. These forward-looking statements are subject to the safe harbor provisions
under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently
available information regarding these matters may not materialize. Actual outcomes and results may differ materially
from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in
circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business
and our customers and partners; our ability to continue to deliver and improve our offerings and successfully
develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and
purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings;
our ability to realize value from investments in the business, including R&D investments; our ability to maintain and
expand our user and customer base; our international expansion strategy; our ability to successfully execute our
go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer
retention and expansion; and general market, political, economic and business conditions.
Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in
our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for
the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any
subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s
website at ir.elastic.co and the SEC’s website at www.sec.gov.
Any features or functions of services or products referenced in this presentation, or in any presentations, press
releases or public statements, which are not currently available or not currently available as a general availability
release, may not be delivered on time or at all. The development, release, and timing of any features or functionality
described for our products remains at our sole discretion. Customers who purchase our products and services
should make the purchase decisions based upon services and product features and functions that are currently
available.
All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not
currently intend to, update any forward-looking statements or statements relating to features or functions of services
or products, except as required by law.
Forward-Looking Statements

3. Agenda
• Planning for scale
• Checking the plan (monitoring)
• Adapt and adjust (troubleshooting)

4. Planning for
scale is easy
 No one, ever

5. 5
Why might you
need to scale?

6. 6
elastic.co/integrations
Expanding data

7. 7
Extending existing use cases

8. 8
Kibana
Elasticsearch
Beats Logstash
Elastic Stack
Elastic Enterprise Search Elastic SecurityElastic Observability
Adding new use cases

9. 9
Architecting for scale

10. 10
One cluster or many?
Elastic Enterprise
Search
Elastic
Security
Elastic
Observability
VS.
Elastic
Security
Elastic
Observability
Elastic Enterprise
Search

11. Where to deploy?
SaaS Orchestration
Elastic Cloud
on Kubernetes
Elastic Cloud Elastic Cloud
Enterprise
Elastic Enterprise Search Elastic SecurityElastic Observability
Kibana
Elasticsearch
Beats Logstash
Powered by
the stack
3 solutions
Deployed
anywhere

12. 12
Minimum architecture (POC
Master
Data
Coordinating
Ingest
Machine Learning

13. 13
Scalable Architecture Master
Data
Coordinating
Ingest
Machine Learning

14. 14
Data at scale

15. 15
E.g. 1 million metrics
per second
Aggregate per minute
Data rollups

16. 16
Schema on write
query performance
Schema on read
flexibility, cost, ingest pace
Extract, Transform, Index
Readiness for immediate query/agg
Load almost raw
Prep per query upon need
Advantages:
● Immediate response time
● Flexibility for new docs
Advantages:
● Flexibility for ingested docs
● Start without data/use knowledge
● Improved ingest rate

17. 17
New and important
security data is
coming in! Keep it
for 2 weeks.
1 2 3
No new data but
data is searched
sometimes. Keep it
for 90 days.
No longer querying
but data should be
retained for 3 years
per company policy.
Hot Warm Cold
Index lifecycle management
4
Delete
Data is no longer
needed. Delete it!

18. 18
Data tiering (Future)
Maybe SSDs Spinning Disks
Node
1
Node
2
Hot Tier
SSD
Warm Tier
Node
3
Node
4
Cold Tier
Node
5

19. 19
Master
Data
Coordinating
Ingest
Machine Learning
Optimized Scalable Architecture
Hot Hot Warm Warm Cold Cold

20. 20
Stack monitoring

21. 21
Turnkey solution focused on the health and monitoring of the Elastic stack
What is Stack Monitoring?
Enterprise
Search
SecurityObservability
➔ Targeted to all solution users persona,
responsible for the wellbeing of the underlying
infrastructure, the Elastic stack
➔ Visualize health and performance
characteristics of all the Elastic stack
components
➔ Preconfigured alerts work for you all the time
and notifies you of any potential issues

22. 22
One click “setup” in Elastic Cloud
• Enable “Monitoring” with a single click
• Consolidate monitoring of multiple production clusters for bird’s eye view

23. 23
Full-stack monitoring
● Elasticsearch
● Kibana
● APM
● Logstash
● Beats
Stack Monitoring
Visualize health and performance characteristics
• Provides curated UI experience with easy grouping and drill
down navigation
• Performance characteristics of key metrics helps spot potential
issues

24. 24
Out of the box Alerts
• Preconfigured alerts work for you all the time
• Notifies you of any potential issues
Alerts
● Cluster Health Status
● Elasticsearch version mismatch
● Kibana version mismatch
● Logstash version mismatch
● Elasticsearch nodes changed
● CPU threshold
Coming soon)
● Disk capacity
● Memory utilization
● Threadpool rejections
● Shard size
● Average search latency
● Shard/segment count
● ...

25. 25
Built in investigative workflows and
next step suggestions
Out of the box Alerts
• Integrated alert views reduces MTTD and MTTR

26. 26
Easily customize in place to meet
specific use cases:
• Modify alert “conditions” for
unique cluster scenarios
• Configure “actions” for
unique notification
preferences
Out of the box Alerts -
Alert customizations

27. 27
Easily extend, if needed, with familiar
Kibana tools
• Kibana Alerts and Actions
– Central place to create, edit
and control all alerts
– Create new alerts on the
monitoring data
• Kibana Visualization
– Create new visualizations and
dashboard with the
monitoring data
Out of the box Alerts -
Alert customizations

28. 28
Demo