SlideShare a Scribd company logo
1
Runtime Fields
Gilad Gal
Product Manager, Elasticsearch
2
This presentation and the accompanying oral presentation contain forward-looking statements, including statements
concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future
operations and expected performance. These forward-looking statements are subject to the safe harbor provisions
under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently
available information regarding these matters may not materialize. Actual outcomes and results may differ materially
from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in
circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business
and our customers and partners; our ability to continue to deliver and improve our offerings and successfully
develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and
purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings;
our ability to realize value from investments in the business, including R&D investments; our ability to maintain and
expand our user and customer base; our international expansion strategy; our ability to successfully execute our
go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer
retention and expansion; and general market, political, economic and business conditions.
Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in
our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for
the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any
subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s
website at ir.elastic.co and the SEC’s website at www.sec.gov.
Any features or functions of services or products referenced in this presentation, or in any presentations, press
releases or public statements, which are not currently available or not currently available as a general availability
release, may not be delivered on time or at all. The development, release, and timing of any features or functionality
described for our products remains at our sole discretion. Customers who purchase our products and services
should make the purchase decisions based upon services and product features and functions that are currently
available.
All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not
currently intend to, update any forward-looking statements or statements relating to features or functions of services
or products, except as required by law.
Forward-Looking Statements
3
Runtime fields in a nutshell
• Empowering all users to generate fields upon need
• Flexibility vs. performance at query time
Schema on read
A Runtime field is a field that is associated with instructions for
calculating it at query time (e.g. script). Runtime fields can be
defined in the mapping or introduced in a query. Other than that
runtime fields behave like any other field in Elasticsearch.
Agenda Slide
What are runtime fields?1
How will runtime fields be implemented?3
Why are runtime fields useful?2
Schema on write
query performance
Extract, Transform, Index
Readiness for immediate query/agg
Advantages:
● Immediate response time
● Flexibility for new docs
Schema on read
flexibility, cost, ingest pace
Load almost raw
Prep per query upon need
Advantages:
● Flexibility for ingested docs
● Start without data/use knowledge
● Improved ingest rate
Schema on write
query performance
Extract, Transform, Index
Readiness for immediate query/agg
Advantages:
● Immediate response time
● Flexibility for new docs
Runtime Fields
Elastic’s schema on read
• Instructions for calculating the
field upon need (e.g. script)
• Defined in the mappings or
introduced in a query
• Smaller index and faster ingest
• Lower query performance
• Other than that - like any other
field
Schema on read
flexibility, cost, ingest pace
Load almost raw
Prep per query upon need
Advantages:
● Flexibility for ingested docs
● Start without data/use knowledge
● Improved ingest rate
Add to mapping
PUT /test {
"mappings": {
"properties": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_second"
},
"message": {
"type": "wildcard"
},
"status": {
"type": "runtime",
"runtime_type": "long",
"script": "String m = doc["message"].value; int end = m.lastIndexOf(" "); int start =
m.lastIndexOf(" ", end - 1) + 1; emit(Long.parseLong(m.substring(start, end)));"
}
}
}
POST /test/_doc?refresh
{
"timestamp" : "1998-04-30T14:30:17-05:00" ,
"message" : "40.135.0.0 - -
[1998-04-30T14:30:17-05:00] "GET /images/hm_bg.jpg
HTTP/1.0 " 200 24736"
}
and use like any other field
POST /_async_search
{
"query": {
"bool": {
"must" : [
{ "match": { "status": "200" } },
{
"range" : {
"@timestamp" : { "gte": "1998-05-01T00:00:00Z" , "lt": "1998-05-02T00:00:00Z" }
}
}
]
}
}
}
POST /_async_search
{
"runtime_mappings": {
"ip": {
"type": "runtime",
"runtime_type": "ip",
"script": "String m = doc["message"].value; emit(m.substring(0, m.indexOf(" ")));"
}
},
"query": {
"bool": {
"must": [
{ "range": { "ip": { "gte": "40.135.0.0", "lt": "40.135.255.255" } } },
{ "match": { "status": "200" } },
{ "range": { "@timestamp": { "gte": "1998-05-01T00:00:00Z", "lt": "1998-05-02T00:00:00Z" } } }
]
}
}
}
Query a runtime field defined on the fly
POST /test/_doc?refresh
{
"timestamp" : "1998-04-30T14:30:17-05:00" ,
"message" : "40.135.0.0 - - [1998-04-30T14:30:17-05:00]
"GET /images/hm_bg.jpg HTTP/1.0 " 200 24736"
}
Future enhancements
• Painless script
• Grok patterns
• Query time enrichment
• Source field
Options for defining the function that yields the value in the field
Agenda Slide
Use color to highlight
What are runtime fields?1
How will runtime fields be implemented?3
Why are runtime fields useful?2
Schema on read
Benefits:
– Flexibility in defining the data
– No index footprint (lower TCO
– Improved ingest pace
Extract, transform and index data *only* upon need
Beneficial, but we do have better
mechanisms to help deal with these
Letting analysts define their schema in retrospect
A new field lifecycle
Extract more data
with Runtime fields
Index only @timestamp
The rest as log entry in
_source
Turn frequently
used runtime fields
into indexed fields
Benefits:
● Save time and effort
● Add fields if and when required, without knowing everything in advance
● Only index what you need - save index size - performance and hardware cost
Fix mapping errors
Benefits:
• Fix immediately, without reindexing
• Queries and schema don’t change (performance impacted)
Index Index data for optimal performance
Retrospective
Fix
Identify an error in the ingest instructions and
override the indexed field with runtime field for
indexed documents
Index Index new documents with the revised mapping
Field per context
Query, visualization, or completely ad-hoc
"runtime_mappings": {
"ip": {
"type": "runtime",
"runtime_type": "ip",
"script": "String m =
doc["message"].value;
emit(m.substring(0, m.indexOf(" ")));"
}
Benefits:
• Avoid polluting everyone’s schema with fields that answer a need only for a subset of the users
• Analyze more efficiently with fields designed to answer a specific need
What’s the average size of an article in my index?
I need to know for relevance ranking tuning.
Please don’t add it to everyone’s articles
index… You’re the only one interested in
it, and even you just look at it once a
month.
Autonomy
Anyone is free to create new fields
No collateral
impact
Adding a Runtime field
(not indexed)
Low permission
barrier
Benefits:
● Administrators avoid spending time on creating schema for specific needs
● Employees that are permitted to define their own data structure can achieve
more with fewer resources
Agenda Slide
Use color to highlight
What are runtime fields?1
How will runtime fields be implemented?3
Why are runtime fields useful?2
The complex parts are things we already have
Putting pre-existing mechanisms together
• Calculate a field value per document and do that quickly
– Prefered Painless script over ingest processor adaptation
• Index to rely on for the heavy lifting
• Logic to minimize the cases in which the calculation is performed
• Async search to deal with slow queries
Async Queries
Robustness to
slow queries
Sync search
Query
Results
or or
Query
Partial
Results & ID
Call w.
ID
Complete
Result set
Timeout
Query
Query
Results
Async search
Efficient calculation at query time
• Calculate only upon need
– Aggregations
– Filter only after filtering by indexed fields
– Display fields for top documents per query
• Initial performance tests prove the important of indexed timestamp
23
Matching is done by the query
Only extract and transform are
made with a script
Define a field with the script
PUT /test {
"mappings": {
"properties": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_second"
},
"message": {
"type": "wildcard"
},
"status": {
"type": "runtime",
"runtime_type": "long",
"script": "String m = doc["message"].value; int end = m.lastIndexOf(" "); int start =
m.lastIndexOf(" ", end - 1) + 1; emit(Long.parseLong(m.substring(start, end)));"
}
}
}
POST /test/_doc?refresh
{
"timestamp" : "1998-04-30T14:30:17-05:00" ,
"message" : "40.135.0.0 - -
[1998-04-30T14:30:17-05:00] "GET /images/hm_bg.jpg
HTTP/1.0 " 200 24736"
}
Matching logic is in the query
POST /_async_search
{
"query": {
"bool": {
"must" : [
{ "match": { "status": "200" } },
{
"range" : {
"@timestamp" : { "gte": "1998-05-01T00:00:00Z" , "lt": "1998-05-02T00:00:00Z" }
}
}
]
}
}
}
Summary
• Runtime fields - schema on read in Elasticsearch
• Gaining in flexibility, index size and ingest pace, at a cost to
performance
• Leveraging existing mechanisms, e.g. index, async search, painless,
query optimization
• Facilitating new workflows:
– Field per context (query, visualization, schema, etc.)
– Fixing ingest errors in retrospect
– New field creation and ingest workflow: start working and gradually create the
schema
Runtime fields
Coming soon to an
elasticsearch cluster
near you
27
Thank You!

More Related Content

What's hot

An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
Elasticsearch
 
Dynamic Allocation in Spark
Dynamic Allocation in SparkDynamic Allocation in Spark
Dynamic Allocation in Spark
Databricks
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and Containers
SATOSHI TAGOMORI
 
PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)
PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)
PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)
NTT DATA Technology & Innovation
 
Apache Pinot Meetup Sept02, 2020
Apache Pinot Meetup Sept02, 2020Apache Pinot Meetup Sept02, 2020
Apache Pinot Meetup Sept02, 2020
Mayank Shrivastava
 
Practical Elasticsearch - real world use cases
Practical Elasticsearch - real world use casesPractical Elasticsearch - real world use cases
Practical Elasticsearch - real world use cases
Itamar
 
Linking Metrics to Logs using Loki
Linking Metrics to Logs using LokiLinking Metrics to Logs using Loki
Linking Metrics to Logs using Loki
Knoldus Inc.
 
MLflow MLOps Architecture - Technical Perspective
MLflow MLOps Architecture - Technical PerspectiveMLflow MLOps Architecture - Technical Perspective
MLflow MLOps Architecture - Technical Perspective
amesar0
 
Introduction to elasticsearch
Introduction to elasticsearchIntroduction to elasticsearch
Introduction to elasticsearch
pmanvi
 
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Databricks
 
Introduction to ML with Apache Spark MLlib
Introduction to ML with Apache Spark MLlibIntroduction to ML with Apache Spark MLlib
Introduction to ML with Apache Spark MLlib
Taras Matyashovsky
 
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPUHow We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
Databricks
 
Hive spark-s3acommitter-hbase-nfs
Hive spark-s3acommitter-hbase-nfsHive spark-s3acommitter-hbase-nfs
Hive spark-s3acommitter-hbase-nfs
Yifeng Jiang
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
Ryan Blue
 
Productizing Structured Streaming Jobs
Productizing Structured Streaming JobsProductizing Structured Streaming Jobs
Productizing Structured Streaming Jobs
Databricks
 
Spark DataFrames and ML Pipelines
Spark DataFrames and ML PipelinesSpark DataFrames and ML Pipelines
Spark DataFrames and ML Pipelines
Databricks
 
DocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer Simon
DocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer SimonDocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer Simon
DocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer Simon
lucenerevolution
 
Introduction to KSQL: Streaming SQL for Apache Kafka®
Introduction to KSQL: Streaming SQL for Apache Kafka®Introduction to KSQL: Streaming SQL for Apache Kafka®
Introduction to KSQL: Streaming SQL for Apache Kafka®
confluent
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
confluent
 
In-memory OLTP storage with persistence and transaction support
In-memory OLTP storage with persistence and transaction supportIn-memory OLTP storage with persistence and transaction support
In-memory OLTP storage with persistence and transaction support
Alexander Korotkov
 

What's hot (20)

An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
Dynamic Allocation in Spark
Dynamic Allocation in SparkDynamic Allocation in Spark
Dynamic Allocation in Spark
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and Containers
 
PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)
PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)
PostgreSQLをKubernetes上で活用するためのOperator紹介!(Cloud Native Database Meetup #3 発表資料)
 
Apache Pinot Meetup Sept02, 2020
Apache Pinot Meetup Sept02, 2020Apache Pinot Meetup Sept02, 2020
Apache Pinot Meetup Sept02, 2020
 
Practical Elasticsearch - real world use cases
Practical Elasticsearch - real world use casesPractical Elasticsearch - real world use cases
Practical Elasticsearch - real world use cases
 
Linking Metrics to Logs using Loki
Linking Metrics to Logs using LokiLinking Metrics to Logs using Loki
Linking Metrics to Logs using Loki
 
MLflow MLOps Architecture - Technical Perspective
MLflow MLOps Architecture - Technical PerspectiveMLflow MLOps Architecture - Technical Perspective
MLflow MLOps Architecture - Technical Perspective
 
Introduction to elasticsearch
Introduction to elasticsearchIntroduction to elasticsearch
Introduction to elasticsearch
 
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
 
Introduction to ML with Apache Spark MLlib
Introduction to ML with Apache Spark MLlibIntroduction to ML with Apache Spark MLlib
Introduction to ML with Apache Spark MLlib
 
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPUHow We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
 
Hive spark-s3acommitter-hbase-nfs
Hive spark-s3acommitter-hbase-nfsHive spark-s3acommitter-hbase-nfs
Hive spark-s3acommitter-hbase-nfs
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
 
Productizing Structured Streaming Jobs
Productizing Structured Streaming JobsProductizing Structured Streaming Jobs
Productizing Structured Streaming Jobs
 
Spark DataFrames and ML Pipelines
Spark DataFrames and ML PipelinesSpark DataFrames and ML Pipelines
Spark DataFrames and ML Pipelines
 
DocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer Simon
DocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer SimonDocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer Simon
DocValues aka. Column Stride Fields in Lucene 4.0 - By Willnauer Simon
 
Introduction to KSQL: Streaming SQL for Apache Kafka®
Introduction to KSQL: Streaming SQL for Apache Kafka®Introduction to KSQL: Streaming SQL for Apache Kafka®
Introduction to KSQL: Streaming SQL for Apache Kafka®
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
 
In-memory OLTP storage with persistence and transaction support
In-memory OLTP storage with persistence and transaction supportIn-memory OLTP storage with persistence and transaction support
In-memory OLTP storage with persistence and transaction support
 

Similar to Schema on read with runtime fields

Why you should use Elastic for infrastructure metrics
Why you should use Elastic for infrastructure metricsWhy you should use Elastic for infrastructure metrics
Why you should use Elastic for infrastructure metrics
Elasticsearch
 
Centralized logging in a changing environment at the UK’s DVLA
Centralized logging in a changing environment at the UK’s DVLACentralized logging in a changing environment at the UK’s DVLA
Centralized logging in a changing environment at the UK’s DVLA
Elasticsearch
 
HR management system
HR management systemHR management system
HR management system
aakashmehta1993
 
Case Study - Upgrading to the Next Gen User Interface for Documentum- final
Case Study - Upgrading to the Next Gen User Interface for Documentum- finalCase Study - Upgrading to the Next Gen User Interface for Documentum- final
Case Study - Upgrading to the Next Gen User Interface for Documentum- final
Brian Nace
 
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
CodeScience
 
Unlock your core business assets for the hybrid cloud with addi webinar dec...
Unlock your core business assets for the hybrid cloud with addi   webinar dec...Unlock your core business assets for the hybrid cloud with addi   webinar dec...
Unlock your core business assets for the hybrid cloud with addi webinar dec...
Sherri Hanna
 
How Zebra Technologies delivers business intelligence with Elastic on Google ...
How Zebra Technologies delivers business intelligence with Elastic on Google ...How Zebra Technologies delivers business intelligence with Elastic on Google ...
How Zebra Technologies delivers business intelligence with Elastic on Google ...
Elasticsearch
 
PAC Fast Track Implementation Program
PAC Fast Track Implementation ProgramPAC Fast Track Implementation Program
PAC Fast Track Implementation Program
Performance Analytics Corporation
 
Elastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and actionElastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and action
Elasticsearch
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resume
archana yadav
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resume
archana yadav
 
Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...
Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...
Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...
Mumbai B.Sc.IT Study
 
ApexUnit: Open source test framework for apex
ApexUnit: Open source test framework for apexApexUnit: Open source test framework for apex
ApexUnit: Open source test framework for apex
Vamshidhar Gandham
 
What's New Overview for IBM Streams V4.3
What's New Overview for IBM Streams V4.3 What's New Overview for IBM Streams V4.3
What's New Overview for IBM Streams V4.3
lisanl
 
Abhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _TestingAbhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _Testing
Abhishek Banerjee
 
Abhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _TestingAbhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _Testing
Abhishek Banerjee
 
Elasticsearch: From development to production in 15 minutes
Elasticsearch: From development to production in 15 minutesElasticsearch: From development to production in 15 minutes
Elasticsearch: From development to production in 15 minutes
Elasticsearch
 
Business Technology Brief
Business Technology BriefBusiness Technology Brief
Business Technology Brief
King's College London
 
1) Question Add Targets to Balanced score Card
1) Question  Add Targets to Balanced score Card1) Question  Add Targets to Balanced score Card
1) Question Add Targets to Balanced score Card
MartineMccracken314
 
1) Question Add Targets to Balanced score Card
1) Question  Add Targets to Balanced score Card1) Question  Add Targets to Balanced score Card
1) Question Add Targets to Balanced score Card
AbbyWhyte974
 

Similar to Schema on read with runtime fields (20)

Why you should use Elastic for infrastructure metrics
Why you should use Elastic for infrastructure metricsWhy you should use Elastic for infrastructure metrics
Why you should use Elastic for infrastructure metrics
 
Centralized logging in a changing environment at the UK’s DVLA
Centralized logging in a changing environment at the UK’s DVLACentralized logging in a changing environment at the UK’s DVLA
Centralized logging in a changing environment at the UK’s DVLA
 
HR management system
HR management systemHR management system
HR management system
 
Case Study - Upgrading to the Next Gen User Interface for Documentum- final
Case Study - Upgrading to the Next Gen User Interface for Documentum- finalCase Study - Upgrading to the Next Gen User Interface for Documentum- final
Case Study - Upgrading to the Next Gen User Interface for Documentum- final
 
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
 
Unlock your core business assets for the hybrid cloud with addi webinar dec...
Unlock your core business assets for the hybrid cloud with addi   webinar dec...Unlock your core business assets for the hybrid cloud with addi   webinar dec...
Unlock your core business assets for the hybrid cloud with addi webinar dec...
 
How Zebra Technologies delivers business intelligence with Elastic on Google ...
How Zebra Technologies delivers business intelligence with Elastic on Google ...How Zebra Technologies delivers business intelligence with Elastic on Google ...
How Zebra Technologies delivers business intelligence with Elastic on Google ...
 
PAC Fast Track Implementation Program
PAC Fast Track Implementation ProgramPAC Fast Track Implementation Program
PAC Fast Track Implementation Program
 
Elastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and actionElastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and action
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resume
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resume
 
Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...
Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...
Project Management (Practical Qustion Paper) [CBSGS - 75:25 Pattern] {2013-20...
 
ApexUnit: Open source test framework for apex
ApexUnit: Open source test framework for apexApexUnit: Open source test framework for apex
ApexUnit: Open source test framework for apex
 
What's New Overview for IBM Streams V4.3
What's New Overview for IBM Streams V4.3 What's New Overview for IBM Streams V4.3
What's New Overview for IBM Streams V4.3
 
Abhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _TestingAbhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _Testing
 
Abhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _TestingAbhishek_Banerjee_Functional _Testing
Abhishek_Banerjee_Functional _Testing
 
Elasticsearch: From development to production in 15 minutes
Elasticsearch: From development to production in 15 minutesElasticsearch: From development to production in 15 minutes
Elasticsearch: From development to production in 15 minutes
 
Business Technology Brief
Business Technology BriefBusiness Technology Brief
Business Technology Brief
 
1) Question Add Targets to Balanced score Card
1) Question  Add Targets to Balanced score Card1) Question  Add Targets to Balanced score Card
1) Question Add Targets to Balanced score Card
 
1) Question Add Targets to Balanced score Card
1) Question  Add Targets to Balanced score Card1) Question  Add Targets to Balanced score Card
1) Question Add Targets to Balanced score Card
 

More from Elasticsearch

From MSP to MSSP using Elastic
From MSP to MSSP using ElasticFrom MSP to MSSP using Elastic
From MSP to MSSP using Elastic
Elasticsearch
 
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webCómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios web
Elasticsearch
 
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas
Elasticsearch
 
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudTirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Elasticsearch
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
Elasticsearch
 
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.
Elasticsearch
 
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Elasticsearch
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
Elasticsearch
 
Welcome to a new state of find
Welcome to a new state of findWelcome to a new state of find
Welcome to a new state of find
Elasticsearch
 
Building great website search experiences
Building great website search experiencesBuilding great website search experiences
Building great website search experiences
Elasticsearch
 
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchKeynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified search
Elasticsearch
 
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesCómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisiones
Elasticsearch
 
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud
Elasticsearch
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
Elasticsearch
 
Transforming data into actionable insights
Transforming data into actionable insightsTransforming data into actionable insights
Transforming data into actionable insights
Elasticsearch
 
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?
Elasticsearch
 
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentEmpowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside Government
Elasticsearch
 
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodThe opportunities and challenges of data for public good
The opportunities and challenges of data for public good
Elasticsearch
 
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticEnterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and Elastic
Elasticsearch
 
What's new at Elastic: Update on major initiatives and releases
What's new at Elastic: Update on major initiatives and releasesWhat's new at Elastic: Update on major initiatives and releases
What's new at Elastic: Update on major initiatives and releases
Elasticsearch
 

More from Elasticsearch (20)

From MSP to MSSP using Elastic
From MSP to MSSP using ElasticFrom MSP to MSSP using Elastic
From MSP to MSSP using Elastic
 
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webCómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios web
 
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas
 
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudTirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.
 
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
Welcome to a new state of find
Welcome to a new state of findWelcome to a new state of find
Welcome to a new state of find
 
Building great website search experiences
Building great website search experiencesBuilding great website search experiences
Building great website search experiences
 
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchKeynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified search
 
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesCómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisiones
 
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Transforming data into actionable insights
Transforming data into actionable insightsTransforming data into actionable insights
Transforming data into actionable insights
 
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?
 
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentEmpowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside Government
 
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodThe opportunities and challenges of data for public good
The opportunities and challenges of data for public good
 
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticEnterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and Elastic
 
What's new at Elastic: Update on major initiatives and releases
What's new at Elastic: Update on major initiatives and releasesWhat's new at Elastic: Update on major initiatives and releases
What's new at Elastic: Update on major initiatives and releases
 

Recently uploaded

Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 

Recently uploaded (20)

Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 

Schema on read with runtime fields

  • 1. 1 Runtime Fields Gilad Gal Product Manager, Elasticsearch
  • 2. 2 This presentation and the accompanying oral presentation contain forward-looking statements, including statements concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future operations and expected performance. These forward-looking statements are subject to the safe harbor provisions under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently available information regarding these matters may not materialize. Actual outcomes and results may differ materially from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business and our customers and partners; our ability to continue to deliver and improve our offerings and successfully develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings; our ability to realize value from investments in the business, including R&D investments; our ability to maintain and expand our user and customer base; our international expansion strategy; our ability to successfully execute our go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer retention and expansion; and general market, political, economic and business conditions. Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s website at ir.elastic.co and the SEC’s website at www.sec.gov. Any features or functions of services or products referenced in this presentation, or in any presentations, press releases or public statements, which are not currently available or not currently available as a general availability release, may not be delivered on time or at all. The development, release, and timing of any features or functionality described for our products remains at our sole discretion. Customers who purchase our products and services should make the purchase decisions based upon services and product features and functions that are currently available. All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not currently intend to, update any forward-looking statements or statements relating to features or functions of services or products, except as required by law. Forward-Looking Statements
  • 3. 3 Runtime fields in a nutshell • Empowering all users to generate fields upon need • Flexibility vs. performance at query time Schema on read A Runtime field is a field that is associated with instructions for calculating it at query time (e.g. script). Runtime fields can be defined in the mapping or introduced in a query. Other than that runtime fields behave like any other field in Elasticsearch.
  • 4. Agenda Slide What are runtime fields?1 How will runtime fields be implemented?3 Why are runtime fields useful?2
  • 5. Schema on write query performance Extract, Transform, Index Readiness for immediate query/agg Advantages: ● Immediate response time ● Flexibility for new docs
  • 6. Schema on read flexibility, cost, ingest pace Load almost raw Prep per query upon need Advantages: ● Flexibility for ingested docs ● Start without data/use knowledge ● Improved ingest rate Schema on write query performance Extract, Transform, Index Readiness for immediate query/agg Advantages: ● Immediate response time ● Flexibility for new docs
  • 7. Runtime Fields Elastic’s schema on read • Instructions for calculating the field upon need (e.g. script) • Defined in the mappings or introduced in a query • Smaller index and faster ingest • Lower query performance • Other than that - like any other field Schema on read flexibility, cost, ingest pace Load almost raw Prep per query upon need Advantages: ● Flexibility for ingested docs ● Start without data/use knowledge ● Improved ingest rate
  • 8. Add to mapping PUT /test { "mappings": { "properties": { "@timestamp": { "type": "date", "format": "strict_date_optional_time||epoch_second" }, "message": { "type": "wildcard" }, "status": { "type": "runtime", "runtime_type": "long", "script": "String m = doc["message"].value; int end = m.lastIndexOf(" "); int start = m.lastIndexOf(" ", end - 1) + 1; emit(Long.parseLong(m.substring(start, end)));" } } } POST /test/_doc?refresh { "timestamp" : "1998-04-30T14:30:17-05:00" , "message" : "40.135.0.0 - - [1998-04-30T14:30:17-05:00] "GET /images/hm_bg.jpg HTTP/1.0 " 200 24736" }
  • 9. and use like any other field POST /_async_search { "query": { "bool": { "must" : [ { "match": { "status": "200" } }, { "range" : { "@timestamp" : { "gte": "1998-05-01T00:00:00Z" , "lt": "1998-05-02T00:00:00Z" } } } ] } } }
  • 10. POST /_async_search { "runtime_mappings": { "ip": { "type": "runtime", "runtime_type": "ip", "script": "String m = doc["message"].value; emit(m.substring(0, m.indexOf(" ")));" } }, "query": { "bool": { "must": [ { "range": { "ip": { "gte": "40.135.0.0", "lt": "40.135.255.255" } } }, { "match": { "status": "200" } }, { "range": { "@timestamp": { "gte": "1998-05-01T00:00:00Z", "lt": "1998-05-02T00:00:00Z" } } } ] } } } Query a runtime field defined on the fly POST /test/_doc?refresh { "timestamp" : "1998-04-30T14:30:17-05:00" , "message" : "40.135.0.0 - - [1998-04-30T14:30:17-05:00] "GET /images/hm_bg.jpg HTTP/1.0 " 200 24736" }
  • 11. Future enhancements • Painless script • Grok patterns • Query time enrichment • Source field Options for defining the function that yields the value in the field
  • 12. Agenda Slide Use color to highlight What are runtime fields?1 How will runtime fields be implemented?3 Why are runtime fields useful?2
  • 13. Schema on read Benefits: – Flexibility in defining the data – No index footprint (lower TCO – Improved ingest pace Extract, transform and index data *only* upon need Beneficial, but we do have better mechanisms to help deal with these Letting analysts define their schema in retrospect
  • 14. A new field lifecycle Extract more data with Runtime fields Index only @timestamp The rest as log entry in _source Turn frequently used runtime fields into indexed fields Benefits: ● Save time and effort ● Add fields if and when required, without knowing everything in advance ● Only index what you need - save index size - performance and hardware cost
  • 15. Fix mapping errors Benefits: • Fix immediately, without reindexing • Queries and schema don’t change (performance impacted) Index Index data for optimal performance Retrospective Fix Identify an error in the ingest instructions and override the indexed field with runtime field for indexed documents Index Index new documents with the revised mapping
  • 16. Field per context Query, visualization, or completely ad-hoc "runtime_mappings": { "ip": { "type": "runtime", "runtime_type": "ip", "script": "String m = doc["message"].value; emit(m.substring(0, m.indexOf(" ")));" } Benefits: • Avoid polluting everyone’s schema with fields that answer a need only for a subset of the users • Analyze more efficiently with fields designed to answer a specific need What’s the average size of an article in my index? I need to know for relevance ranking tuning. Please don’t add it to everyone’s articles index… You’re the only one interested in it, and even you just look at it once a month.
  • 17. Autonomy Anyone is free to create new fields No collateral impact Adding a Runtime field (not indexed) Low permission barrier Benefits: ● Administrators avoid spending time on creating schema for specific needs ● Employees that are permitted to define their own data structure can achieve more with fewer resources
  • 18. Agenda Slide Use color to highlight What are runtime fields?1 How will runtime fields be implemented?3 Why are runtime fields useful?2
  • 19. The complex parts are things we already have Putting pre-existing mechanisms together • Calculate a field value per document and do that quickly – Prefered Painless script over ingest processor adaptation • Index to rely on for the heavy lifting • Logic to minimize the cases in which the calculation is performed • Async search to deal with slow queries
  • 21. Sync search Query Results or or Query Partial Results & ID Call w. ID Complete Result set Timeout Query Query Results Async search
  • 22. Efficient calculation at query time • Calculate only upon need – Aggregations – Filter only after filtering by indexed fields – Display fields for top documents per query • Initial performance tests prove the important of indexed timestamp
  • 23. 23 Matching is done by the query Only extract and transform are made with a script
  • 24. Define a field with the script PUT /test { "mappings": { "properties": { "@timestamp": { "type": "date", "format": "strict_date_optional_time||epoch_second" }, "message": { "type": "wildcard" }, "status": { "type": "runtime", "runtime_type": "long", "script": "String m = doc["message"].value; int end = m.lastIndexOf(" "); int start = m.lastIndexOf(" ", end - 1) + 1; emit(Long.parseLong(m.substring(start, end)));" } } } POST /test/_doc?refresh { "timestamp" : "1998-04-30T14:30:17-05:00" , "message" : "40.135.0.0 - - [1998-04-30T14:30:17-05:00] "GET /images/hm_bg.jpg HTTP/1.0 " 200 24736" }
  • 25. Matching logic is in the query POST /_async_search { "query": { "bool": { "must" : [ { "match": { "status": "200" } }, { "range" : { "@timestamp" : { "gte": "1998-05-01T00:00:00Z" , "lt": "1998-05-02T00:00:00Z" } } } ] } } }
  • 26. Summary • Runtime fields - schema on read in Elasticsearch • Gaining in flexibility, index size and ingest pace, at a cost to performance • Leveraging existing mechanisms, e.g. index, async search, painless, query optimization • Facilitating new workflows: – Field per context (query, visualization, schema, etc.) – Fixing ingest errors in retrospect – New field creation and ingest workflow: start working and gradually create the schema Runtime fields Coming soon to an elasticsearch cluster near you