Learn how NCIS data analysis uses Elasticsearch to process evidence in the form of log files, its impact on efficient law enforcement, and some lessons learned along the way. See the video: https://www.elastic.co/elasticon/tour/2019/washington-dc/turning-evidence-into-insights-how-ncis-leverages-elastic-

1. Justin Myers
Lead Computer Scientist
Naval Criminal Investigative Service
Enabling Investigations With
Log Data

2. Log Files and the NCIS Mission
UNCLASSIFIED (Approved for public release)
What we do for the Navy and
Marine Corps:
• Investigate felony crime
• Prevent terrorism
• Protect secrets
Incorporating digital technology is
core to success in every mission
area.

3. Log Files and the NCIS Mission
UNCLASSIFIED (Approved for public release)
NCIS Cyber Capabilities Integration Division
bridges the gap between technical subject
matter experts and field agents/analysts who
are out accomplishing the mission.

4. Log Files and the NCIS Mission
UNCLASSIFIED (Approved for public release)
https://www.researchgate.net/profile/Athena_Vakali/publication/220773991/figure/fig1/AS:393984494063616@1470944631218/A-sample-of-Web-Server-Log-File.png
• Investigations (especially hacking
incidents) generate a lot of log files
• Web server logs
• Firewall logs
• DNS logs
• Netflow
• Syslogs
• Host-based logs
• Proxy/WAF logs
• And more!

5. Log Files and the NCIS Mission
UNCLASSIFIED (Approved for public release)
• Previous solution had limitations:
• Oracle couldn’t handle volume of
data necessary at desired speed
• Only limited types of queries
• Not user-friendly

6. Log Files and the NCIS Mission
UNCLASSIFIED (Approved for public release)
• New requirements:
• Accessible for non-technical users
• Fast
• Allows creative searching across
all available cases
• Handle arbitrarily large data
volumes

7. How (not to?) build an investigative cluster
UNCLASSIFIED (Approved for public release)
Early 2018:
• 2 Dell PowerEdge R930s
• 8 Sockets, 8TB RAM total
• 40TB HD
• 10 separate instances of Elasticsearch each
• Total Nodes: 20 Master/Data nodes
Design: Lessons learned:
• Helpful to finalize design
before buying hardware
• Dedicated hard drives are
better than network storage

8. How (not to?) build an investigative cluster
UNCLASSIFIED (Approved for public release)
Early 2019:
• 3 Dell PowerEdge R930s
• 12 Sockets, 16TB RAM, 60TB HD
• 10 separate instances of Elasticsearch each
• 9 M640 Blades
• 2 CPU each, 64GB RAM each, 4TB SSD each
• Total Nodes: 39 Master/Data nodes
Design: Lessons learned:
• Dedicated hardware = better
performance and stability
• One node per blade > shared
bare metal > virtual
machines
• Ensure front end distributes
work evenly across nodes

9. How (not to?) build an investigative cluster
UNCLASSIFIED (Approved for public release)
Today:
• 24 MX740C Blades
• 2 CPU each, 96GB RAM each, 3.6TB SSD each
• 48 M640 Blades
• 2 CPU each, 64GB RAM each, 4TB SSD each
• Total Nodes: 3 Master/69 Data nodes
Design: Lessons learned:
• Dedicated master nodes
• Reducing garbage collection
impact (jvm.options):
• XX:ParallelGCThreads=<# CPUs on
blade>
• XX:NewRatio=2 (Changes the ratio
of New/Old GC)
• These resulted in our GC times
dropping from ~45ms to ~1-3ms
392 indices
7,754 shards
210 billion documents
138TB on disk
And counting…

10. Results: Enhanced
investigations using data
• Search across investigations
• Lucene query syntax for advanced users
• Enable trend/behavior identification
• Geospatial-aware searching
• Automated alerts and reporting
UNCLASSIFIED (Approved for public release)
https://www.cucocreative.co.uk/wp-content/uploads/2013/01/twitter-peaks.gif

11. Future
• More nodes = more data, better performance
• Separate Beats and logging into separate cluster
• Decrease size of shards/add to shard count
• Focus on user accessibility
• Continued integration of Elastic Stack features
• Machine learning integration
• Integration with investigative processes
UNCLASSIFIED (Approved for public release)

12. Learn more
Justin Myers
Lead Computer Scientist
Naval Criminal Investigative Service
justin.myers@ncis.navy.mil
UNCLASSIFIED (Approved for public release)