Michael will present an overview of Elastic's machine learning capabilities. As we know, data science work can be messy, fractured, and challenging as data volumes increase. This session will explore how the Elastic stack can offer a single destination for data ingestion and exploration, time series modeling, and communication of results through data visualizations by focusing on a few sample data sources. We will also explore new functionality offered by Elastic machine learning, in particular an integration with our APM solution. Trained as a mathematician, Michael Hirsch started his career with no development experience. His first task - "model the world in a relational database." Over the last 7 years Michael has established himself a data scientist, with a focus on building end-to-end systems. In his career, he has built machine learning powered platforms for clients including Nike, Samsung, and Marvel, and approaches his work with the idea that machine learning is only as useful as the interfaces that users interact with. Currently, Michael is a Product Engineer for Machine Learning at Elastic. He focuses on tailoring Elastic's ML offering to customer use cases, as well as integrating machine learning capabilities across the entire Elastic Stack.

1. Data Science
in the Elastic Stack

2. A Data Science Process?

3. [1]: df = pd.read_csv(“data.csv”)
[2]: train, test = preprocess(df)
[3]: pipe = Pipeline([('transform', ct), ('lr', LR())])
[4]: pipe.fit(train, y)
[5]: plt.plot(results)
[6]: plt.save_fig(‘results.png’)
> mkdir project
> cd project
> mkdir data
> mv ~/Downloads/data/* data
> virtualenv venv
> source venv/bin/activate
> pip install pandas numpy
matplotlib sklearn nltk requests
bs4 boto3 jupyter
> jupyter notebook

4. {api}
/f(x)
f(x)

5. f(x)

6. du -sch data/*
du -sch data/*
98.4G total
919.8M total

7. data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

8. google: data too big for
pandas
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

9. jupyter notebook
google: data too big for
pandas
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

10. jupyter notebook
json.dump(“export_FINAL.json”)
with open(“export_FINAL_2.csv”)
google: data too big for
pandas
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

11. jupyter notebook
json.dump(“export_FINAL.json”)
with open(“export_FINAL_2.csv”)
google: data too big for
pandas
google: deploy model to production
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

12. jupyter notebook
import dask
json.dump(“export_FINAL.json”)
with open(“export_FINAL_2.csv”)
google: data too big for
pandas
google: deploy model to production
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

13. data results

14. A Data Science Process

15. ModelDiscover Ingest Operationalize

16. Methods OutcomesGoals
● work with customers and
stakeholders to understand and
identify business problems
● data audit
● project charter (README.md)
● data source definitions
● data dictionaries
● define objectives - specify key
variables and related metrics
● identify relevant data sources
ModelDiscover Ingest Operationalize

17. Methods ArtifactsGoals
● ad-hoc exploratory data analysis
of raw data.
● development of data pipeline that
transforms raw data.
● charter updates (README.md)
● architecture
● produce high-quality datasets with
a clear relationship to the target
objectives
● provide data to an analytical
environment
● develop architecture to keep data
fresh and up to date
ModelDiscover Ingest Operationalize

18. ModelDiscover Ingest Operationalize

19. ModelDiscover Ingest Operationalize
POST _bulkhelpers.bulk(es, data)

20. Methods ArtifactsGoals
● feature engineering
● modeling training
● feature sets
● A standardized way of
benchmarking model results.
● in our case, job configs :D
● identification of optimal features
for machine learning modeling
● creation of a model that best fits
the business objectives and
modelling task
● production ready model!
ModelDiscover Ingest Operationalize

21. ModelDiscover Ingest Operationalize
Logging and Metrics: Spot an unusual drop in application
requests and drill in on the troublesome server contributing to
the problem.
Security Analytics: Identify unusual network activity or user
behavior to pinpoint attackers before they do damage.
Business Analytics: Get notified if there is an unusual
increase in abandoned shopping carts in your ecommerce
site.
Application Performance Monitoring: Catch bottlenecks
and slow response times so your apps can keep running
smoothly.

22. Methods OutcomesGoals
● validation
● project hand-off
● operationally useful KPIs powered
by ML
● user acceptance
ModelDiscover Ingest Operationalize

24. - Identify demand in real time
Data
Metrics
Goals & Objectives
- raw taxi trip information
ModelDiscover Ingest Operationalize

25. ?
Data
Metrics
Goals & Objectives
Data
Features
Success Factors
Goals & Objectives
- decrease waiting times by 50%
- aggregate rides to and from specific locations
ModelDiscover Ingest Operationalize

26. KPIs
Data
Features
Success Factors
Goals & Objectives
average waiting time
raw taxi trip information
aggregate rides to and from specific locations
decrease waiting times by 50%
Identify demand in real time
ModelDiscover Ingest Operationalize

27. ModelDiscover Ingest Operationalize
http://www.nyc.gov/html/tlc/downloads/pdf/data_dictionary_trip_r
ecords_yellow.pdf

28. ModelDiscover Ingest Operationalize
TLC Objectives
● identify pockets of demand as
they materialize
● identify potential failures of
payment systems
● get a sense of volume in the
coming weeks
helpers.bulk(es, data)

29. ModelDiscover Ingest Operationalize

30. Thanks