1. ◦ Shieldx Detailed Alerts Overview
◦ Integration with Elastic Search
◦ Insights about Elastic Search
◦ Demo of the System
◦ Shieldx Alerts CSV File Download

2. Data Accuracy vs Speed
Trade-of

3. Area of Research
 Elastic Search Ingestion Performance
- Going Async
- Better Memory management
- Improving existing Data Structures

4. Test Suite
 Using encoders to encode messages in
ElasticSearch
 Turning off all Indexes and Analysis
 Going Async
 Enabling G1GC collection
 Remodeling the data structure to Nested Type
 Improve Latency Sensitivity, Datastore

5. Data Ingestion Results
TerraBitTest Sync Async/G1GC
(Non Nested)
Async
(Nested)
Striped
Disk
240,000
events/sec = 1
Terrabit
OutOfMem
oryError
30,000
events/sec
120,000
events/sec
120,000
events/se
c

7. Some other improvements
index.refresh_interval = 0
ElasticSearch Routing
ES Heap size = 50% of available disk

8. Future Work
 1) More research on ES Nested Queries
Performance Improvement
 2) Set up optimal ES Cluster with seperate co-
ordinator and data nodes for improved
performance.
 3) Measure latency on SSD