Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016

•

2 likes•2,041 views

"Elasticsearch has come a long way: Started as a distributed search engine in 2009, it's now the tool of choice for even the largest websites (e.g. Facebook, Github, Ebay). Half-way to 2016 the ELK stack helped it become firmly embedded in many centralised log management systems (e.g. Netflix, Uber). We're now midair in the next step, with the first folks using it for metrics. NASA is using it to monitor the Curiosity rover, Blizzard and Riot to monitor vast online gaming worlds. This talk will focus on what makes this transition from more unstructured to structured data possible. "

Technology

USING
ELASTIC
TO
MONITOR
EVERYTHING
Christoph Wurm
Solutions Architect
wurm@elastic.co
@ChristophWurm

BATTLE PLAN
15:05 – 15:40 SLIDES
15:40 – 15:55 COFFEE
15:55 – 16:55 HANDS ON

Unstructured Data Structured Data
Search Logging Security MetricsAnalytics
Social SensorLogsDocuments User Activity Location

Elastic Cloud
Security
X-Pack
Kibana
User Interface
ElasticsearchStore, Index,
& Analyze
Ingest
Logstash Beats
+
Elastic Stack
Alerting
Monitoring
Reporting
Graph

DATA STORE
FAST
SCALEABLE
FULL TEXT
Numbers need 60% less
memory in 5.0

USER INTERFACE
VISUALIZATIONS
DASHBOARDS
REAL TIME
CHARTING LANGUAGE (5.0)

DATA PROCESSING
REAL TIME PIPELINE
200+ PLUGINS
SYSLOG
REGEX
MULTILINE
LOOKUPS
GEOIP

DATA COLLECTION
AT THE SOURCE
LIGHTWEIGHT
REAL TIME
Preconfigured Dashboards

Filebeat
Tail files
Packetbeat
Capture network packets
Metricbeat
System (Topbeat), Apache,
Docker, HAProxy, MongoDB,
MySQL, Nginx, PostgreSQL,
Redis, Zookeeper
Winlogbeat
Windows event logs

apachebeat
httpd server-status
burrowbeat
kafka consumer lag
cassandrabeat
nodetool cfstats
cloudtrailbeat
aws events
dockerbeat
container stats
elasticbeat
elasticsearch status
execbeat
shell cmd output
flowbeat
sflow
factbeat
puppet facter
httpbeat
poll http(s) endpoints
jmxproxybeat
tomcat jmx metrics
lmsensorsbeat
temp, voltage, fans
mysqlbeat
any query -> elasticsearch
packagebeat
dpkg, rpm
pingbeat
icmp ping w/ rtt
redisbeat
redis info
twitterbeat
tweets by screen name
wmibeat
windows monitoring
https://www.elastic.co/guide/en/beats/libbeat/master/community-beats.html

IN PRODUCTION
SECURITY
MONITORING
REPORTING
ALERTING
MACHINE LEARNING
GRAPH

Beats
Log Files Metrics
Wire Data
Elasticsearch Kibana
Master / Data Nodes (3)
Ingest Nodes (X)
your{beat}
X-Pack X-Pack
Simple
Beats to Ingest Node
Instances (X)

Beats Logstash Elasticsearch Kibana
X-Pack X-Pack
Additional Processing
Nodes (X)
Instances (X)
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
Log Files Metrics
Wire Data your{beat}
LDAP
Authentication
AD
Notification
SSO

Deployment at Scale
Kibana
X-Pack
Instances (X)
Beats
Log Files Metrics
Wire Data
Datastore Web APIs
Social Sensors
Kafka
Redis
Messaging
Queue
Logstash
Elasticsearch
Nodes (X)
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
your{beat}
X-Pack
LDAP
Authentication
AD
Notification
SSO

Beats
Log Files Metrics
Wire Data
Datastore Web APIs
Social Sensors
Kafka
Redis
Messaging
Queue
Logstash
ES-Hadoop
Elasticsearch
Kibana
Nodes (X)
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
Instances (X)
your{beat}
X-Pack X-Pack
Custom UI
LDAP
Authentication
AD
Notification
SSO
Deployment in the Enterprise
Hadoop Ecosystem

Metricbeat
Packetbeat
Metricbeat
Packetbeat
Metricbeat
Packetbeat
Metricbeat
Packetbeat
TEL AVIV
DUBLIN +

github.com/cwurm/devopsdays-telaviv
Elasticsearch http://tinyurl.com/devopsdays-elasticsearch:9243
Kibana http://tinyurl.com/devopsdays-kibana
Credentials devops / telaviv2016

YouTube: https://www.youtube.com/watch?v=1cCD5axQf9U&list=PLnKL6-WWWE_VtIMfNLW3N3RGuCUcQkDMl&index=7 Time-based data, especially logs are all around us. Every application, system or hardware piece logs something - from simple messages, to large stack traces. In this talk we will learn how to build and tune resilient log aggregation pipeline using Elasticsearch and Kafka as its heart. We will start by looking at the overall architecture and how we can connect Elasticsearch and Kafka together. We will look at how to scale our system through a hybrid approach using a combination of time- and size-based indices, and also how to divide the cluster in tiers in order to handle the potentially spiky load in real-time. Then, we'll look at tuning individual nodes. We'll cover everything from commits, buffers, merge policies and doc values to OS settings like disk scheduler, SSD caching, and huge pages. Finally, we'll take a look at the pipeline of getting the logs to Elasticsearch and how to make it fast and reliable: where should buffers live, which protocols to use, where should the heavy processing be done (like parsing unstructured data), and which tools from the ecosystem can help.

This talk was given during DevOps Con 2017. Have you ever spent time digging through various terminals, greping, lessing, awking and trying to find that few log lines that may be important? Have you every done that under time pressure, because mission critical services were not working? Have you every heard from your developers that they can’t tell you anything, because they don’t have access to application logs? Have you ever considered a centralized storage for logs, but time and resources are not on your side? If you said yes, to any of the above questions, than this talk is for you. During the talk we’ll introduce you to the world of log centralization and analysis, both when it comes to open source, but also commercial tools. We will go from top to bottom and learn how to setup log centralization and analysis for servers, virtualized environments and containers. We will get from log shipping, through centralized buffering to storage and analysis to show you, that having a centralized log analysis tool is not a rocket science. Finally, you will see how useful is to combine the logs from all your servers in a single place for blazingly fast correlation.

Elasticsearch in Netflix

Danny Yuan

ELK Stack

Phuc Nguyen

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...

ForgeRock

ELK introduction

Waldemar Neto

Elastic stack Presentation

Amr Alaa Yassen

Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)

Cohesive Networks

Slides from the Chicago AWS user group on May 5th, 2016. Asaf Yigal, Co-Founder and VP Product at Logz.io, presented on using Elasticsearch, Logstash, and Kibana in Amazon Web Services. "Setting up the increasingly-popular open-source ELK Stack (Elasticsearch, Logstash, and Kibana) on AWS might seem like an easy task, but we have gone through several iterations in our architecture and have made some mistakes in our deployments that have turned out to be common in the industry. In this talk, we will go through what we did and explain what worked and what failed -- and why. We will also provide a complete blueprint of how to set up ELK for production on AWS." ~ @asafyigal

Monitoring with Graylog - a modern approach to monitoring?

inovex GmbH

Optimizing Elastic for Search at McQueen Solutions

Elasticsearch

Scaling an ELK stack at bol.com

Renzo Tomà

A presentation about the deployment of an ELK stack at bol.com At bol.com we use Elasticsearch, Logstash and Kibana in a logsearch system that allows our developers and operations people to easilly access and search thru logevents coming from all layers of its infrastructure. The presentations explains the initial design and its failures. It continues with explaining the latest design (mid 2014). Its improvements. And finally a set of tips are giving regarding Logstash and Elasticsearch scaling. These slides were first presented at the Elasticsearch NL meetup on September 22nd 2014 at the Utrecht bol.com HQ.

'Scalable Logging and Analytics with LogStash'

Cloud Elements

Rich Viet, Principal Engineer at Cloud Elements presents 'Scalable Logging and Analytics with LogStash' at All Things API meetup in Denver, CO. Learn more about scalable logging and analytics using LogStash. This will be an overview of logstash components, including getting started, indexing, storing and getting information from logs. Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching).

ELK: a log management framework

Giovanni Bechis

Elk devops

Ideato

La gestione dei log è da sempre un argomento complesso e nel tempo si sono cercate varie soluzioni più o meno complesse, spesso difficili da integrare nel proprio stack applicativo. Daremo un’ overview generale dei principali sistemi di aggregazione evoluta dei log in realtime (Fluentd, Greylog, eccetera) e illustreremo del motivo ci ha spinto a scegliere ELK per risolvere un’esigenza del nostro cliente; ovvero di consultare i log in modo piu comprensibile da persone non tecniche. Lo stack ELK (Elasticsearch Logstash Kibana) permette agli sviluppatori di consultare i log in fase di debug / produzione senza avvalersi dello staff sistemistico. Dimostreremo come abbiamo eseguito il deployment dello stack ELK e lo abbiamo implementato per interpretare e strutturare i log applicativi di Magento.

Elastic - ELK, Logstash & Kibana

SpringPeople

Provisioning Datadog with Terraform

Matt Spurlin

elk_stack_alexander_szalonnasAlexander Szalonnas

6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan...

Jürgen Ambrosi

In questa sessione vedremo, con il solito approccio pratico di demo hands on, come utilizzare il linguaggio R per effettuare analisi a valore aggiunto, Toccheremo con mano le performance di parallelizzazione degli algoritmi, aspetto fondamentale per aiutare il ricercatore nel raggiungimento dei suoi obbiettivi. In questa sessione avremo la partecipazione di Lorenzo Casucci, Data Platform Solution Architect di Microsoft.

Minio Red Herring

Minio

What's hot

Elk

Caleb Wang

Log analysis using Logstash,ElasticSearch and Kibana

Avinash Ramineni

More kibana

琛琳饶

Logging with Elasticsearch, Logstash & KibanaAmazee Labs

Introducing log analysis to your organization

Sematext Group, Inc.

Elasticsearch in Netflix

Danny Yuan

ELK Stack

Phuc Nguyen

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...

ForgeRock

ELK introduction

Waldemar Neto

Elastic stack Presentation

Amr Alaa Yassen

Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)

Cohesive Networks

Monitoring with Graylog - a modern approach to monitoring?

inovex GmbH

Optimizing Elastic for Search at McQueen Solutions

Elasticsearch

Scaling an ELK stack at bol.com

Renzo Tomà

'Scalable Logging and Analytics with LogStash'

Cloud Elements

ELK: a log management framework

Giovanni Bechis

Elk devops

Ideato

Elastic - ELK, Logstash & Kibana

SpringPeople

Provisioning Datadog with Terraform

Matt Spurlin

elk_stack_alexander_szalonnasAlexander Szalonnas

What's hot (20)

Elk

Log analysis using Logstash,ElasticSearch and Kibana

More kibana

Logging with Elasticsearch, Logstash & Kibana

Introducing log analysis to your organization

Elasticsearch in Netflix

ELK Stack

Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...

ELK introduction

Elastic stack Presentation

Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)

Monitoring with Graylog - a modern approach to monitoring?

Optimizing Elastic for Search at McQueen Solutions

Scaling an ELK stack at bol.com

'Scalable Logging and Analytics with LogStash'

ELK: a log management framework

Elk devops

Elastic - ELK, Logstash & Kibana

Provisioning Datadog with Terraform

elk_stack_alexander_szalonnas

Similar to Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016

6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan...

Jürgen Ambrosi

Minio Red Herring

Minio

Designing big data analytics solutions on azure

Mohamed Tawfik

Big Data Modeling Challenges and Machine Learning with No Code

Liana Ye

Presented at SF BAY ACM_202001015_by_Karthik Chinnusamy What are the Big Data model challenges in today's field? With a few best practice recommendations and Machine Learning approaches, I will use Knime to show the modeling advantages for Big Data with the following themes: .Performance: Good data models can help us quickly query the required data and reduce I/O throughput. .Cost: Good data models can significantly reduce unnecessary data redundancy, reuse computing results, and reduce the storage and computing costs for the big data system. .Efficiency: Good data models can greatly improve user experience and increase the efficiency of data utilization. .Quality: Good data models make data statistics more consistent and reduce the possibility of computing errors. I will also describe tools for Sources, Ingestion, Exploration, Modeling and Machine Learning.

Binary Studio Academy 2016. MS Azure. Cloud hosting.

Binary Studio

Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto

idsecconf

Data Streaming: What? Why? How?

Ofir Manor

IOT, Streaming Analytics and Machine Learning

DataWorks Summit/Hadoop Summit

AWS Webcast - Amazon Kinesis and Apache Storm

Amazon Web Services

Join us for a for a Amazon Kinesis tutorial webinar. In this session we will provide a reference architecture and instructions for building a system that performs real-time sliding-windows analysis over streaming clickstream data. We will use Amazon Kinesis for managed ingestion of streaming data at scale with the ability to replay past data, and run sliding-window computation using Apache Storm. We’ll demonstrate in the webinar on how to build the system and deploy on AWS and walkthrough all the steps from ingestion, processing, and storing to visualizing of the data in real-time.

ELK Solutions Enablement Session - 17th March'2020

Ashnikbiz

Opening Keynote - AWS Summit SG 2017

Amazon Web Services

Opening Keynote - AWS Summit SG 2017

Amazon Web Services

Track 6 Session 2_ 搭建現代化的資料數據湖.pptxAmazon Web Services

Workshop: Big Data Visualization for Security

Raffael Marty

Big Data is the latest hype in the security industry. We will have a closer look at what big data is comprised of: Hadoop, Spark, ElasticSearch, Hive, MongoDB, etc. We will learn how to best manage security data in a small Hadoop cluster for different types of use-cases. Doing so, we will encounter a number of big-data open source tools, such as LogStash and Moloch that help with managing log files and packet captures. As a second topic we will look at visualization and how we can leverage visualization to learn more about our data. In the hands-on part, we will use some of the big data tools, as well as a number of visualization tools to actively investigate a sample data set.

The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...

The Hive

Until recently, data was gathered for well-defined objectives such as auditing, forensics, reporting and line-of-business operations; now, exploratory and predictive analysis is becoming ubiquitous, and the default increasingly is to capture and store any and all data, in anticipation of potential future strategic value. These differences in data heterogeneity, scale and usage are leading to a new generation of data management and analytic systems, where the emphasis is on supporting a wide range of very large datasets that are stored uniformly and analyzed seamlessly using whatever techniques are most appropriate, including traditional tools like SQL and BI and newer tools, e.g., for machine learning and stream analytics. These new systems are necessarily based on scale-out architectures for both storage and computation. Hadoop has become a key building block in the new generation of scale-out systems. On the storage side, HDFS has provided a cost-effective and scalable substrate for storing large heterogeneous datasets. However, as key customer and systems touch points are instrumented to log data, and Internet of Things applications become common, data in the enterprise is growing at a staggering pace, and the need to leverage different storage tiers (ranging from tape to main memory) is posing new challenges, leading to caching technologies, such as Spark. On the analytics side, the emergence of resource managers such as YARN has opened the door for analytics tools to bypass the Map-Reduce layer and directly exploit shared system resources while computing close to data copies. This trend is especially significant for iterative computations such as graph analytics and machine learning, for which Map-Reduce is widely recognized to be a poor fit. While Hadoop is widely recognized and used externally, Microsoft has long been at the forefront of Big Data analytics, with Cosmos and Scope supporting all internal customers. These internal services are a key part of our strategy going forward, and are enabling new state of the art external-facing services such as Azure Data Lake and more. I will examine these trends, and ground the talk by discussing the Microsoft Big Data stack.

Elasticsearch features and ecosystem

Pavel Alexeev

Scalable POSIX File Systems in the Cloud

Red_Hat_Storage

Fom io t_to_bigdata_step_by_step-final

Luis Filipe Silva

Why apache Flink is the 4G of Big Data Analytics Frameworks

Slim Baltagi

Apache Flink is a community-driven open source and memory-centric Big Data analytics framework. It provides the only hybrid (Real-Time Streaming + Batch) open source distributed data processing engine supporting many use cases. Flink uses a mixture of Scala and Java internally, has very good Scala APIs and some of its libraries are basically pure Scala (FlinkML and Table). At its core, it is a streaming dataflow execution engine and it also provides several APIs for batch processing (DataSet API), real-time streaming (DataStream API) and relational queries (Table API) and also domain-specific libraries for machine learning (FlinkML) and graph processing (Gelly). In this talk, you will learn in more details about: What is Apache Flink, how it fits into the Big Data ecosystem and why it is the 4G (4th Generation) of Big Data Analytics frameworks? How Apache Flink integrates with Apache Hadoop and other open source tools for data input and output as well as deployment? Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark? What are the benchmarking results between Apache Flink and those other Big Data analytics frameworks?

Mai-Lan Tomsen Bukovec- Keynote-AWS Summit Manila

Amazon Web Services

Similar to Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016 (20)

6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan...

Minio Red Herring

Designing big data analytics solutions on azure

Big Data Modeling Challenges and Machine Learning with No Code

Binary Studio Academy 2016. MS Azure. Cloud hosting.

Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto

Data Streaming: What? Why? How?

IOT, Streaming Analytics and Machine Learning

AWS Webcast - Amazon Kinesis and Apache Storm

ELK Solutions Enablement Session - 17th March'2020

Opening Keynote - AWS Summit SG 2017

Track 6 Session 2_ 搭建現代化的資料數據湖.pptx

Workshop: Big Data Visualization for Security

The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...

Elasticsearch features and ecosystem

Scalable POSIX File Systems in the Cloud

Fom io t_to_bigdata_step_by_step-final

Why apache Flink is the 4G of Big Data Analytics Frameworks

Mai-Lan Tomsen Bukovec- Keynote-AWS Summit Manila

More from DevOpsDays Tel Aviv

YOUR OPEN SOURCE PROJECT IS LIKE A STARTUP, TREAT IT LIKE ONE, EYAR ZILBERMAN...

Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016

Similar to Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016 (20)

More from DevOpsDays Tel Aviv

More from DevOpsDays Tel Aviv (20)

Recently uploaded

Recently uploaded (20)

Using Elastic to Monitor Everything - Christoph Wurm, Elastic - DevOpsDays Tel Aviv 2016