The document discusses the importance of normalizing security data using the Elastic Common Schema (ECS), which addresses challenges such as disparate data types and blind spots. It outlines ECS's role in providing a consistent data format for integrated security across various Elastic products and services, alongside available documentation and tools for mapping data to ECS. Key points include the significance of custom fields, integration capabilities, and resources for further community engagement.

1. 1
The importance of
normalizing your security
data
Jamie Hynds - Product Manager - Security Integrations
Mat Martin - Tech Lead - Elastic Common Schema

2. 2
This presentation and the accompanying oral presentation contain forward-looking statements, including statements
concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future
operations and expected performance. These forward-looking statements are subject to the safe harbor provisions
under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently
available information regarding these matters may not materialize. Actual outcomes and results may differ materially
from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in
circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business
and our customers and partners; our ability to continue to deliver and improve our offerings and successfully
develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and
purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings;
our ability to realize value from investments in the business, including R&D investments; our ability to maintain and
expand our user and customer base; our international expansion strategy; our ability to successfully execute our
go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer
retention and expansion; and general market, political, economic and business conditions.
Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in
our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for
the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any
subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s
website at ir.elastic.co and the SEC’s website at www.sec.gov.
Any features or functions of services or products referenced in this presentation, or in any presentations, press
releases or public statements, which are not currently available or not currently available as a general availability
release, may not be delivered on time or at all. The development, release, and timing of any features or functionality
described for our products remains at our sole discretion. Customers who purchase our products and services
should make the purchase decisions based upon services and product features and functions that are currently
available.
All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not
currently intend to, update any forward-looking statements or statements relating to features or functions of services
or products, except as required by law.
Forward-Looking Statements

3. 3
Agenda
• Security Data Challenges
• What is ECS?
• Elastic Security Integrations
• Mapping data to ECS
• Demo
• Get started with ECS

4. Security Data Challenges
● Disparate data types
● Heterogeneous environments
● Countless field names
● Searching & correlating across
multiple data sources
● Blind spots
● Strict schemas

5. 5
ECS - Elastic Common Schema
• Defines a common set of fields
• Works across multiple data sources
• Remember common field names
• Reduces blind spots
• Integrate custom sources to Elastic Security

6. What's Missing?
src: 172.16.137.6 OR
client_ip: 172.16.137.6 OR
apache.access.remote_ip: 172.16.137.6 OR
context.user.ip: 172.16.137.6 OR
src_ip: 172.16.137.6
it's not
OR srcIP: 172.16.137.6
A Common Schema!
source.ip: 172.16.137.6

7. 7
3 solutions powered by 1 stack
(and 1 schema)
Kibana
Elasticsearch
Beats Logstash
Elastic Stack
Elastic Enterprise Search Elastic SecurityElastic Observability

8. 8
Elastic Common Schema
• Re-use analysis content across
multiple data sources ♻
• Leverage content in any
environment, without modification
– Elastic
– Partners
– Community
ECS
Visualizations
Alerts
Reports
Searches
ML jobs
Dashboards
Detection Rules

9. Elastic Integrations
https://www.elastic.co/integrations

10. Building a pipeline
For custom and supported sources
Elastic Agent (Beta)

11. Mapping to ECS  high level
● Documentation
○ ECS elastic.co/guide/en/ecs/current/ecs-reference.html
○ Elastic Security elastic.co/guide/en/security/current/siem-field-reference.html
○ Populate as many fields as you can
● Spreadsheet 🤔

12. Mapping to ECS field names
● Plaintext logs
○ Use ECS field names in your grok or dissect
● Structured events
○ Multiple field renames
○ ecs-mapper: mapping spreadsheet ⇒ 3 starter pipelines
useradd[12507]: new user: name=bob, UID=1002, GID=1003
%{DATA:process.name}[%{POSINT:process.pid}]: new user: name=%{DATA:user.name},
UID=%{NUMBER:user.id}, GID=%{NUMBER:group.id}

13. Types of Enrichment
● Environment information (agent)
○ host, container, cloud
● Lookups
○ Process: PID => executable name, CLI arguments, parent process
○ Geolocation: 42.43.44.45 => Sparks, Nevada, US
● Processors
○ User agent string ⇒ browser name & version, OS
○ FQDN => registered domain (cdn-42f0ab.images.google.co.uk)
○ Calculating community ID

14. Building on your logs
{
"message": "pam_unix(su:session)...",
"agent": {
"type": "filebeat",
"version": "7.9.1"
},
"host": {
"name": "bob-laptop",
"id": "7fa34c7914aa4e4...",
"os": {
"name": "Ubuntu",
"version": "18.04.4 LTS"
},
"ip": [ "10.0.2.15" ]
},
su[9000]: pam_unix(su:session): session opened for user root by vagrant(uid=0)
"process": {
"start": "2020-08-26T17:20:38.430Z",
"pid": 9000,
"name": "su",
"executable": "/bin/su",
"parent": {
"pid": 8999,
"name": "sudo",
"executable": "/usr/bin/sudo"
}
}
}

15. Event Categorization
Designed to work across data sources

16. Event Categorization Fields
● Where’s it from
● What it is
Predetermined values Source values
Hardcoded
values

17. Custom Fields
● ECS is extensible
● Consider labels
● Adding custom fields 👍
○ But avoid conflicts with future versions 🤔
ECS Field Custom Fields
Lowercase
proxy.origin.ip
Capitalization
Proxy.origin
Uses concept names:
proxy.origin.ip
Nest under proper names (vendor, project)
squid.origin
acme.proxy.origin

18. Index Templates
● Index templates are like a database schema
● ECS is your starter index template
○ github.com/elastic/ecs/tree/master/generated/elasticsearch
● Elasticsearch can also guess datatypes, but beware ☝
● Best practice: use an index template
○ Kibana
○ JSON
○ Tools in the ECS project (including your custom fields)
○ github.com/elastic/ecs/blob/master/USAGE.md

19. Demo

20. 20
Implementing a common event
model (ECS) has been a force
multiplier for our team, that
dramatically streamlines
investigations
Wes Connell - Security Engineering Lead, Uber
“

21. Summary
● Security requires a consistent data format
● ECS is used across Elastic solutions
● ECS allows for custom fields
● Elastic Integrations
● Tools available to map your data to ECS

22. Resources
Documentation
ECS
elastic.co/guide/en/ecs/current/index.html
Elastic Security
elastic.co/guide/en/ecs/current/ecs-field-reference.html
Tools
ECS Mapper
github.com/elastic/ecs-mapper
Customizing the ECS Template
github.com/elastic/ecs/blob/master/USAGE.md
Get in touch!
Community Slack
elasticstack.slack.com ⇒ #stack-ecs
Forum
discuss.elastic.co/tag/ecs-elastic-common-schema

23. 23
Try free on Cloud:
ela.st/siem
Take a quick spin:
demo.elastic.co
Connect on Slack:
ela.st/slack
Join the Elastic Security community

24. 24
Thank You!