Discover the safeguards Kroger uses to understand how to protect your cluster, improve performance, and provide a better end user experience that enables observability at scale.

1. Protecting Your Cluster From
Your Humans
Jaclyn Priede

2. Protecting Your Cluster From
Your Humans
Jaclyn Priede

3. 3
Kroger
• Grocery retailer offering in
store and online shopping
• Founded by Barney Kroger
1883
• 2,700 stores
• Ranked 20 in Fortune 500

4. 4
Echo
• Kroger’s Enterprise Log
Aggregation Solution
• Based on Elasticsearch and
Kibana
• Custom Solutions:
• Ingestion
• Alerting
• Data management
• Rollup
• Clients
• Why Echo?

5. 5
Utilization
Echo Stats
Over a thousand aliases
165 teams
100K Messages/Second
71.7 TB Storage
Use Cases
SOC
Development
SRE
ORE
Alerting

6. 6
Have you ever had a query cripple
your cluster?
Is that acceptable?

7. 7
Data Ingestion
• Problem: Sending too much
data to Echo
• Solution:
• Custom Ingestion Pipeline
• Default quota
• Rates Page

8. 8
Data Ingestion
• Problem: Sending too much
data to Echo
• Solution:
• Custom Ingestion Pipeline
• Default quota
• Rates Page

9. 9
Data Ingestion
• Problem: Sending too much
data to Echo
• Solution:
• Custom Ingestion Pipeline
• Default quota
• Rates Page

10. 10
Sensitive Data
• Problem: Sending sensitive
data to Echo

11. 11
Sensitive Data
• Problem: Sending sensitive
data to Echo
• Solution:
• Terms and Conditions
• Protected Spaces
• Data Masking

12. 12
Batch Failures
• Problem:
• Parsing Failures
• Indexing Failures
• Solution:
• Alerting users

13. 13
Data Storage
• Problem: Storing too much
data in Echo
• Solution:
• Default retention
• Alerting on disk space

14. 14
Indexing Fields
• Problem:
• Fields with too many characters
• Fields with high cardinality data
• Solution:
• Truncating fields
• Kafka & Elasticsearch
• Alerting users

15. 15
Expensive Queries
• Problem: Executing
expensive queries in
Elastic
• Solution:
• Slow Logging
• Slow Indexing

16. 16
Long Running
Tasks
• Problem: Tasks failing to
complete in a timely fashion
• Solution:
• Automated job to kill tasks >5
min

17. 17
Refresh Rates
• Problem: Excessively high
refresh rates in Kibana
• Solution:
• Removing refresh intervals
under 30 seconds from the UI

18. 18
High Search
Volume
• Problem: Excessively high
search volumes against Elastic
• Solution:
• Audit Log
• Alerting users

19. 19
Coaching Humans
• Encourage people to be
developers
• Advocate best practices
for efficient querying
• Structure your data to
make it easier to search

20. 20
Benefits
Improved performance Better end user
experience
Increased observability
throughout the Enterprise

21. Thank you