The Elastic Stack has been integral to the growth of E*Trade’s real-time operational intelligence pipeline. Hear their journey to adopting Elastic machine learning features and see firsthand how they’re using them to identify anomalies using full-text aggregations and performance data.

1. Jeff Vestal
September 25, 2018
Machine Learning at
1

2. PUBLIC
First Portal Solution for
Mobile Devices
2000
3.9M
1983
Brokerage Accounts
Industry First Online
Trade by an Individual
~3700
Employees
16.1B
Market Cap
2
E*TRADE TECHNOLOGICALLY FOCUSED

3. PUBLIC
Indexed / Sec
Raw documents
indexed
Indexed / Sec
Curated events
indexed
Historical Events
Available on
demand real-time
1M+ 200K 50B
ELASTIC CLUSTERS AT E*TRADE
3

4. PUBLIC
Indexed Data On-Line
115TB
38
140B
Master Nodes
On-Line Documents
99Data Nodes
34Coordinator Nodes
4
ELASTIC CLUSTERS AT E*TRADE

5. PUBLIC
Business Portals
25
145
450
Curated Dashboards
User Logins / Day
150+Base Indices
4B+Micro Service Calls / Day
5
ELASTIC CLUSTERS AT E*TRADE

6. PUBLIC
NEW PIPELINE OVERVIEW
6

7. 7
Portal Access

8. Individual Business Portal
8

9. PUBLIC
Slackbot Integration
with elastic
9

10. PUBLIC
ML Jobs On Multiple Clusters
10

11. PUBLIC
Anomalies and Watches
Across Multiple Clusters
11

12. PUBLIC
REAL-WORLD
EXAMPLES 12

13. PUBLIC
UNSTRUCTURED
TEXT 13

14. PUBLIC 14
Customer Service Case Index

15. PUBLIC
Significant Text Aggregation
An aggregation that returns interesting or unusual occurrences of free-text
terms in a set. It is like the significant terms aggregation but differs in that:
• It is specifically designed for use on type text fields
• It does not require field data or doc-values
• It re-analyzes text content on-the-fly meaning it can also filter duplicate
sections of noisy text that otherwise tend to skew statistics.
Elastic Documentation
15

16. PUBLIC
Datafeed
"aggregations": {
"buckets": {
"date_histogram": {
"field": "LogTime",
"time_zone": "UTC",
"interval": "300s"
},
"aggregations": {
"LogTime": {
"max": {
"field": "LogTime"
}
},
"Notes": {
"significant_text": {
"size": 10,
"min_doc_count": 3,
"shard_min_doc_count": 0,
"field": "Notes",
"filter_duplicate_text": true,
"exclude": [...],
"jlh": {}
Significant Text Aggregation
16

17. PUBLIC
Customer Service Notes
Significant Text Anomaly Detection
17

18. PUBLIC 18

19. PUBLIC 19
Slack Anomaly Alerts

20. PUBLIC 20
Significant Text Anomaly Explorer

21. PUBLIC 21

22. PUBLIC 22

23. PUBLIC
SERVICE CALL
METRICS 23

24. PUBLIC
Page Finder ML Job Split by Service Name
Looking for Slow Services
24

25. PUBLIC
Page Finder ML Slack Alert

26. PUBLIC
Page Finder Anomaly Explorer
26

27. PUBLIC
Page Finder Single Metric
27

28. PUBLIC
Page Finder ML Job Split by Service Name
28

29. PUBLIC 29
Jeff Vestal
September 25, 2018
Machine Learning at