Integrating advanced analytics with Elasticsearch

Integrating Advanced Analytics with Elasticsearch
Emily Bicks & Alan Gersch
Elasticon Public Sector, April 13, 2021

Copyright © 2021 Deloitte Development LLC. All rights reserved. 2
Agenda
3. Implementation Details
1. The Challenge
Our Solution
2.
4. Lessons Learned
Alan Gersch
Specialist Master
Alan has many years of experience in
Computer Science and software
engineering (and Astronomy). At Deloitte,
he has worked on graph analytics, ML/NLP
and currently is on an analytics team
working on FOIA modernization.
Emily Bicks
Senior Consultant
Emily is a data scientist and focuses on
building Natural Language Processing
(NLP) solutions for Federal Government
clients.

Copyright © 2019 Deloitte Development LLC. All rights reserved. | 3
3
The Challenge
We sought to improve
the efficiency and
effectiveness of
document review for
eDiscovery

Use Cases
Elastic has many advanced tools to help with eDiscovery of unstructured text
Responding to FOIA Requests
Investigations and Security
Declassification Reviews
Historical Research
Litigation
Human Resources
Etc.

Government records management policy requires archiving billions of documents which
often must be searched
The Problem
I save everything !
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ?
? ?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ?
? ?
?
?
? ?
? ?
?
?
? ?
? ?
?
?
? ?
? ?
?
?
Keyword Search
? ?
? ?
?
?
We can use advanced search functionality within Elasticsearch to help
searchers find what they are looking for amongst the chaff
I search everything !

It takes a team with a variety of skillsets to implement this program
Roles in Search System Development Program
“As data scientists we are
responsible for selecting,
training, and tuning the
algorithms”
“As business process
consultants, we are responsible
for ensuring tools will enhance
existing processes”
Data Scientists Business Process
Consultants
Developers Testers
Business
Analysts
DevOps
Engineers
Deloitte plays the role of Data Scientists and
Business Process Consultants for this program

AI Tolerance
Different stakeholder have different appetites for advanced features
Certain stakeholders will be
hesitant to change existing
business processes
Certain stakeholders are
excited about leveraging
cutting-edge technologies
I am hesitant to use relevancy
ranking because I don’t know
why it ranks certain documents
over others.
I want an algorithm to find all
of the relevant documents for
me so that I don’t waste hours
reading.
I want to use AI/ML to enable
and expedite my decisions, but
not make them for me.

Our
Solution
We took an agile
approach to building and
integrating features with
Elasticsearch

[TAR is the] process for prioritizing or coding a collection of Electronically Stored Information using a
computerized system that harnesses human judgments of subject matter expert(s) on a smaller set of
documents and then extrapolates those judgments to the remaining documents in the collection.
- The Sedona Conference1
“ ”
1. The Sedona Conference Glossary: E-Discovery and Digital Information Management, Fourth Edition, 15 SEDONA CONF. J. 305 (2014) (definition adopted from Maura R. Grossman & Gordon V. Cormack, The Grossman-Cormack Glossary of Technology-Assisted
Review with Foreword by John M. Facciola, U.S. Magistrate Judge, 7 FED. CTS. L. REV. 1, 32 (2013))
VS.
Agile TAR features
based on Elasticsearch
1.0
2.0
Traditional
Monolithic
TAR System
TAR leverages analytics to improve the efficiency and effectiveness of document review
What is Technology Assisted Review (TAR)?

We have targeted our development of features to built tools with the highest utility for
searchers while minimizing difficulty of implementation
An Agile Approach to “Agile TAR” Feature Identification
Feature
Identification
Identify potential search
enhancements to facilitate document
review
Identified More Like This and
Relevancy Ranking as a high value
and technically feasible solution
Productionize
Simulate search with the addition of
potential features
Rapid Prototyping
Evaluated utility of features for search on
real data and designed system with
specific parameters that will provide the
highest impact
Action
Outcome
Design and develop features in search
system
Allowed users access to features to
facilitate more efficient and effective
search

We identified these features to improve efficiency and effectiveness of the search and
document review process
A Collection of Potential Features
Relevancy Ranking
Rank search results based on tf-idf to
raise more relevant content to top of
results page
Learn relevant categories based on user
behavior during review (i.e. responsive vs.
not-responsive, eligible for
declassification vs. not eligible)
Dynamic Text Classification
Use machine learning and natural language
processing to add additional metadata based on
content in documents and filter on the metadata
ML/NLP Enabled Filtering
Extract people, places and things from
text to help user understand what is
contained in documents
Entity Extraction
Suggest additional facets to users during
review to help include or exclude large
swaths quickly based on available
metadata
Suggested Facets
Find distinctive topics amongst a set of
documents to help searchers explore
results
Significant Terms Aggregation
Specify documents which match your
search need and let the system find
more like them to improve relevancy
More Like This
Explore linkages between documents to
identify those with similar content and
find patterns
Graph Analytics
Advanced
Elasticsearch
Functionality
Custom
Functionality
Integrated
with
Elasticsearch
1.0
Solution

Implementation
Details
12
We simulated reviews to
prove ROI and then
integrated features with the
production system

We leveraged simulations to demonstrate to our users the ROI of integrating the
suggested features
Simulation Results
Reviews to
Find All
Documents
Reviews to
Get to 75%
Recall
Reviews to
Get to 75%
Recall
11,650 reviews 6,800 reviews
Sorting by date
Using the built-in relevance scoring
in Elasticsearch to sort results
Using the built-in relevance scoring and
customized Elasticsearch MLT capability
to sort results
12,750 reviews
D A T E O R D E R E D R E L E V A N C E O R D E R E D
R E L E V A N C E O R D E R E D W I T H
M L T
1 2 3
17,000 reviews 12,350 reviews
11,500 reviews
Reviews to
Find All
Documents
18,700 reviews
___CASE
#1____
___CASE
#2____
9% savings 53% savings
13,450 reviews 21% savings 27% savings
5,600 reviews 51% savings 2,650 reviews 77% savings
18,550 reviews 1% savings 14,800 reviews 21% savings

We leveraged Kibana to visualize the ROI and show how these features would help users
find relevant documents earlier in their review
Speed of Review Completion
KEY More Like This Relevance Date Ordered
Percent
of
(known)
responsive
documents
found
Number of Documents Reviewed
Case 1
Case 2

The 1.0 Solution Was Designed to Present Users With the Most Likely Relevant Content
First
Business Process Redesigned
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ?
? ?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ? ?
? ? ?
?
?
?
?
? ?
? ?
?
?
? ?
? ?
?
?
? ?
? ?
?
?
? ?
? ?
?
?
Keyword Search
? ?
? ?
?
?
ML/NLP Enabled Filtering
? ?
? ?
?
?
? ?
? ?
?
?
? ?
? ?
?
?
Relevancy Sorting
More Like This
ML/NLP is used to identify documents
that may be considered “junk”, for
example spam emails, media clippings,
etc. that users can filter out with one
click.
Elastic’s tf-idf based relevancy sorting
is used to present documents in order of
importance of search terms to the
meaning of the document
Elastic’s “More Like This” queries are
used to resort the original order based on
similarity to documents identified to be
relevant.

Sample Data 1
The following workflow shows how we have integrated machine learning into our ingest
pipelines
What We Built: ML/NLP Enabled Filtering
Custom Ingest Pipeline
Custom
Text
Vectorizer
File Storage
Prep Data
2
3 Train Model
Deploy Model
4
Elastic
Index
Elastic
Aggregations
Filters on
Interface
Production Archive System
Analytics Sandbox Env
Steps

What We Built: Sorting/Resorting With Relevancy and MLT
frontend
backend
Reviewer & UI
Search Terms
“Elasticsearch”,
“Query”
TF-IDF Scoring
Search Results
Sorted by
Relevance
1. Document A
2. Document C
3. Document B
4. Document F
5. Document …
reviews
Non-relevant
Documents
1. Document T
2. Document Y
3. Document … “More Like This”
Query
Search Results
Re-sorted by
New Score
1. Document T
2. Document D
3. Document N
4. Document F
5. Document …
continues
review
Relevant
Documents
1. Document D
2. Document N
3. Document …
TF-IDF scores are calculated
for each term in the “similar”
and “not similar” documents to
select additional query terms
to add to the original query and
rescore documents
New Search
Query
Terms
“Elasticsearch”
, “Query”,
“Search”,
“Documents”
Search Query
Terms
“Elasticsearch”,
“Query”
Initial Query New Query
Similar
Add doc ids to
query of docs
similar to what
we’re looking
for
Not Similar
Add doc ids to
query of docs
that aren’t like
what you’re
looking for
Each document is
scored based on the tf-
idf of each query term
in that document
term frequency = out of all terms in a
document, how many are the search term?
document frequency = out of all documents,
how many contain the search term?

Lessons
Learned
Focusing on enabling
human review as
opposed to replacing it
was the key to our
success

Lessons Learned
Based on these observations, we rely on the following guiding principles
Incremental Releases Customer Resistance
Model Performance vs System
Performance
Availability and Reliability of
Training Data
Building search functionality in
‘bite-size’ pieces and not rushing
to the finish line is beneficial
both technically and practically
Depending on the use-case,
many users may be hesitant to
adopt features that they don’t
understand
Finding features that work
requires optimizing both model
performance statistics such as
Precision and Recall, as well as
considering speed and impacts
on performance of the system as
a whole
Machine learning approaches
may require a lot of training data
to build durable, reliable models
Finding the Right Toolset
Exploratory vs Decision Focused
Features
Explainable AI
Ensuring we are using the tools we have to their
potential and then integrating external tools to
compensate will ensure we are optimizing both
performance of the models and system
Focusing on tools that help users explore the dataset
they are presented with rather than making decisions
for them will help with adoption, likely provide more
utility, and make us less reliant on historical training
data
Focusing on explaining model outcomes and avoiding
black box approaches helps customers have
confidence in search features

Leveraging Elasticsearch features to build Technology Assisted Review has accelerated
multiple business processes for our clients
Technology Assisted Review with Elasticsearch
FASTER
Helps reviewers find
documents more
efficiently
SMARTER
Learns patterns from user
decisions
TRANSPARENT
Avoids black boxes and
relies on understandable
algorithms

21
This publication contains general information only and Deloitte is not, by means of this publication, rendering accounting, business,
financial, investment, legal, tax, or other professional advice or services. This publication is not a substitute for such professional
advice or services, nor should it be used as a basis for any decision or action that may affect your business. Before making any
decision or taking any action that may affect your business, you should consult a qualified professional advisor. Deloitte shall not
be responsible for any loss sustained by any person who relies on this publication.
About Deloitte
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee (“DTTL”), its
network of member firms, and their related entities. DTTL and each of its member firms are legally separate and independent
entities. DTTL (also referred to as “Deloitte Global”) does not provide services to clients. In the United States, Deloitte refers to one
or more of the US member firms of DTTL, their related entities that operate using the “Deloitte” name in the United States and
their respective affiliates. Certain services may not be available to attest clients under the rules and regulations of public
accounting. Please see www.deloitte.com/about to learn more about our global network of member firms.
Copyright © 2021 Deloitte Development LLC. All rights reserved.

Integrating advanced analytics with Elasticsearch

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Integrating advanced analytics with Elasticsearch

Similar to Integrating advanced analytics with Elasticsearch (20)

More from Elasticsearch

More from Elasticsearch (20)

Recently uploaded

Recently uploaded (20)

Integrating advanced analytics with Elasticsearch