Stefan Thies, profile picture
Uploaded byStefan Thies
3,600 views

Metrics & more

The document discusses the importance of monitoring big data systems at scale, highlighting the need to tune performance, detect bugs, and ensure stability while managing vast amounts of data points from various applications. It outlines key components of a monitoring infrastructure, including metric sources, alerting workflows, and the integration of machine learning for event analysis. Effective monitoring solutions require defining criteria, mapping application landscapes, and custom metrics management, along with tools like ELK for log centralization and visualization.

Embed presentation

Downloaded 15 times
Metrics & more how to monitor big data systems @scale!
About me
 Stefan Thies
 @seti321
 ! DevOps Evangelist @sematext!
Why monitoring is important •  Tuning ! •  Detecting Bugs! •  Stability! •  Benchmarks! •  Capacity planning!
Monitoring tools must endure the load
Would you start building own scales, when you would operate a real zoo? - What’s your mechanical engineering expertise? - How long does it take to get tools and raw material? - Who feeds the animals while being in the workshop? - When do we need it and could it be ‚in time‘?
Let’s take something from the shelf and build a custom interface ‚load balancers‘!‚Custom Interface‘!
What happens @scale?
•  Many VM’s & Apps - each one generates ~ 5-130 metrics in short intervals! •  Aggregation, Compromises on resolutions etc.! •  Transactions - each creates N log entries ! •  limit recording, time based indices + aliases! •  High throughput - high rate of logs & metrics! •  build a monitoring infrastructure (remember this)! !
METRIC SOURCE! NUMBER OF METRICS TO COLLECT! OS (CPU. Mem, Disk) 21 Hadoop 133 Hbase 68 Elasticsearch 62 Apache Storm 25 Total 309 ~ 3,1 Mio. data points per week x N machines ! ! Example - No. of metrics per application!
25 Metric Categories ! Metrics – Apache Kafka!
•  Find out and define metrics to collect ! •  Install, configure collectd, statsd, graphite, …! •  Build, install / configure available agents! •  Define reports or arrange all collected metrics to dashboards e.g. grafana, …! •  This are the basics!! •  automate deployment for agents! #monitoringsucks
#monitoringlove •  Integrate with the organization ! •  alerting workflows + multi-user + security! •  Scale out: ! •  Distributed event processing (e.g. Kafka)! •  Scalable data stores (e.g. Elasticsearch, HBase)! •  Add intelligence: ! •  Machine learning for metrics & events! •  Alerting & Reporting based on it!
Monitoring Architecture Receiver! Aggregator! Scalable! Storage! Reporting! Machine Learning! Alerting! Forwarding! User Management! Agents for all monitored applications! Visualisation! Admin!
What can we find
 in the wild?
Network Level •  Packets: loss, size, counts! •  Latency, jitters, delays! •  Bandwidth – total, per link, per service, ! •  Firewalls / security breaches! •  IDS, IPS – yet another malware detected ! •  On physical, transport, application layer, ...!
Server Level •  Disk I/O! •  CPU load! •  Disk Space ! •  Memory! •  Logs / security / events / syslog!
Standard Applications •  Webservers, Databases, Search Engines, MQ‘s! •  Request rates, disk space, partitions, locks, connections, queue sizes, cache sizes! •  Logfiles!
Hadoop, Elasticsearch, Cassandra, Kafka, Storm Spark, ...!
Example: Elasticsearch Link: Top Metrics !
Own Application 
 Custom Metrics & Logs •  Logs & API for measurement! •  Time measurements, KPI‘s, Usage tracking, Object counters, Click Streams!
Application Traces •  Post mortem analysis! process.on (‚exit‘, heapdumpAndDie) •  Dtrace ! •  Call Traces, Error stacks! •  Heapdumps & Flamegraphs!
Log files as source of metrics •  Simplest: log rate of an application! •  Generate Count for operations! •  Apply search and count related events! •  E.g. count slow operations! •  Extract values from logs ! •  Apply regex or field search to extract numbers !
Logs2Metrics Logs! Index! Scheduled Queries! aggregate all messages matching e.g. „session opened“ every Minute e.g. on auth.log Custom ! Metric! Monitoring ! System!
A Checklist for the introduction of monitoring solutions
Define your criterias •  Coverage of monitors/agents! •  Quality of agents & setup! •  Multi-User Support! •  Reporting Capability & Secure Sharing! •  Alerting capabilities! •  Integrations / Notifications / API‘s! •  Estimate required resources !
Map your landscape •  Quantity of servers & applications to monitor! •  What are the components of your App-Stack?! •  Linux on AWS, NGINX, Node.js, REDIS, Elasticsearch! •  Which programming languages are used?! •  Can you find agents/monitors for all your ‚Apps‘?! •  List missing parts -> find other or build a monitor!
Customizing – custom metrics/plugins •  What metrics are relevant for each ‚App‘?! •  What is covered by existing agents?! •  How to aggregate each of this metrics? ! •  min, max, sum, avg! •  Pre-Aggregation vs. Query Time Aggregation!
Dashboards •  Graphs! •  Which metrics belong together?! •  Display options ….! •  Query language ! •  Dashboards! •  What combination of graphs provides best insight?! •  Can you share and re-use arranged dashboards for similar setups or situations? ! •  Or do you need to configure it again for other servers?! •  Is sharing secured? Or just a link to your UI?!
Alerts •  Threshold based alerts! •  Status changes ! •  Heartbeat alerts! •  Anomaly detection! •  Challenges: Number of alert rules and queries ! & tuning ‘noise level’!
Alert notifications anomaly detection and alerting!
•  Metrics show „something happens“! •  Logs provide evidence „what happened“! •  Faster insights by reporting them together! •  Correlate logs and metrics! •  Metrics could be created from logs! Integrate metrics & logs
Correlate Logs & Metrics
A brief overview of 
 Centralizing Logs

raw logs! parser! Log shipper! storage! Visualization! Kibana!Elasticsearch!Logstash! Where is the work?! Centralizing Logs with ELK ! files, syslog! Format adaption,! & transport! Tuning ! Maintenance! Queries! Security !
•  Input: Unstructured log lines! •  Filter & Parser: Grok / RegEx! •  Output: Structured JSON! •  Forwarder: ! •  Elasticsearch, …!
•  Schema: Define the right Mapping •  Insert rate:! •  Use bulk indexing! •  Increase refresh time for higher insert rate! •  Volume: ! •  Aliases and time based indices! •  Memory usage: configure caching limits! Setup Elasticsearch
•  How to secure it? ! •  Proxies, Security plugins, Hosted Solutions! •  Queries and dashboard creation! •  generators/templates for specific setups! •  Learn Lucene query language!
Kibana
Thank you for ! your attention! http://blog.sematext.com!

More Related Content

PDF
node-crate: node.js and big data
byStefan Thies
 
PDF
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
byHakka Labs
 
PDF
Scalable and Reliable Logging at Pinterest
byKrishna Gade
 
PDF
Análisis de las novedades del Elastic Stack
byElasticsearch
 
PDF
Elastic Stack roadmap deep dive
byElasticsearch
 
PPTX
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
byData Con LA
 
PDF
Análisis del roadmap del Elastic Stack
byElasticsearch
 
PPTX
Azure Stream Analytics - Webinar
byHARIHARAN R
 
node-crate: node.js and big data
byStefan Thies
 
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
byHakka Labs
 
Scalable and Reliable Logging at Pinterest
byKrishna Gade
 
Análisis de las novedades del Elastic Stack
byElasticsearch
 
Elastic Stack roadmap deep dive
byElasticsearch
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
byData Con LA
 
Análisis del roadmap del Elastic Stack
byElasticsearch
 
Azure Stream Analytics - Webinar
byHARIHARAN R
 

What's hot

PPTX
Server Log Files & Technical SEO Audits: What You Need to Know
bySamuel Scott
 
PDF
Multi-Tenant Log Analytics SaaS Service using Solr: Presented by Chirag Gupta...
byLucidworks
 
PDF
Searching for Better Code: Presented by Grant Ingersoll, Lucidworks
byLucidworks
 
PDF
Big data at AWS Chicago User Group - 2014
byAWS Chicago
 
PPTX
xPatterns - Spark Summit 2014
byClaudiu Barbura
 
PDF
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
byHakka Labs
 
PDF
Logging, Metrics, and APM: The Operations Trifecta
byElasticsearch
 
PDF
Redash: Open Source SQL Analytics on Data Lakes
byDatabricks
 
PDF
#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)
byPivotalOpenSourceHub
 
PPTX
Sarine's Big Data Journey by Rostislav Aaronov
byIdan Tohami
 
PDF
Fighting Cybercrime: A Joint Task Force of Real-Time Data and Human Analytics...
bySpark Summit
 
PDF
Accelerate Data Science Initiatives: Databricks & Privacera
byDatabricks
 
PPTX
Solr + Hadoop: Interactive Search for Hadoop
bygregchanan
 
PPTX
Time Series Anomaly Detection with Azure and .NETT
byMarco Parenzan
 
PDF
Presto: Fast SQL on Everything
byDavid Phillips
 
PDF
Semantic Image Logging Using Approximate Statistics & MLflow
byDatabricks
 
PDF
Plazma - Treasure Data’s distributed analytical database -
byTreasure Data, Inc.
 
PDF
Elasticsearch in Netflix
byDanny Yuan
 
PDF
DataEngConf SF16 - Data Asserts: Defensive Data Science
byHakka Labs
 
PDF
Cloud Connect 2012, Big Data @ Netflix
byJerome Boulon
 
Server Log Files & Technical SEO Audits: What You Need to Know
bySamuel Scott
 
Multi-Tenant Log Analytics SaaS Service using Solr: Presented by Chirag Gupta...
byLucidworks
 
Searching for Better Code: Presented by Grant Ingersoll, Lucidworks
byLucidworks
 
Big data at AWS Chicago User Group - 2014
byAWS Chicago
 
xPatterns - Spark Summit 2014
byClaudiu Barbura
 
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
byHakka Labs
 
Logging, Metrics, and APM: The Operations Trifecta
byElasticsearch
 
Redash: Open Source SQL Analytics on Data Lakes
byDatabricks
 
#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)
byPivotalOpenSourceHub
 
Sarine's Big Data Journey by Rostislav Aaronov
byIdan Tohami
 
Fighting Cybercrime: A Joint Task Force of Real-Time Data and Human Analytics...
bySpark Summit
 
Accelerate Data Science Initiatives: Databricks & Privacera
byDatabricks
 
Solr + Hadoop: Interactive Search for Hadoop
bygregchanan
 
Time Series Anomaly Detection with Azure and .NETT
byMarco Parenzan
 
Presto: Fast SQL on Everything
byDavid Phillips
 
Semantic Image Logging Using Approximate Statistics & MLflow
byDatabricks
 
Plazma - Treasure Data’s distributed analytical database -
byTreasure Data, Inc.
 
Elasticsearch in Netflix
byDanny Yuan
 
DataEngConf SF16 - Data Asserts: Defensive Data Science
byHakka Labs
 
Cloud Connect 2012, Big Data @ Netflix
byJerome Boulon
 

Viewers also liked

PPTX
Metrics Monitoring Is So Critical - What's Your Best Approach?
byWavefront
 
PPT
What Every Organization Should Log And Monitor
byAnton Chuvakin
 
PDF
Crate Shared Nothing Web Backends - Web Backend Meetup May 2014
byMatthias Wahl
 
PDF
Node.js and couchbase Full Stack JSON - Munich NoSQL
byPhilipp Fehre
 
PDF
Quality Assurance and Testing of Automated Business Processes
byTammo van Lessen
 
PDF
Logging & Metrics
byTammo van Lessen
 
PPTX
Logging & Metrics with Docker
byStefan Zier
 
PPTX
Volta: Logging, Metrics, and Monitoring as a Service
byLN Renganarayana
 
PDF
Metrics Driven Design by Joshua Porter
byAndrew Chen
 
PDF
Pengembangan Aplikasi Cloud Computing Menggunakan Node.js
byBambang Purnomosidi D. P.
 
Metrics Monitoring Is So Critical - What's Your Best Approach?
byWavefront
 
What Every Organization Should Log And Monitor
byAnton Chuvakin
 
Crate Shared Nothing Web Backends - Web Backend Meetup May 2014
byMatthias Wahl
 
Node.js and couchbase Full Stack JSON - Munich NoSQL
byPhilipp Fehre
 
Quality Assurance and Testing of Automated Business Processes
byTammo van Lessen
 
Logging & Metrics
byTammo van Lessen
 
Logging & Metrics with Docker
byStefan Zier
 
Volta: Logging, Metrics, and Monitoring as a Service
byLN Renganarayana
 
Metrics Driven Design by Joshua Porter
byAndrew Chen
 
Pengembangan Aplikasi Cloud Computing Menggunakan Node.js
byBambang Purnomosidi D. P.
 

Similar to Metrics & more

PDF
Monitoring Big Data Systems - "The Simple Way"
byDemi Ben-Ari
 
PDF
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
byCodemotion
 
PDF
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
byDemi Ben-Ari
 
PDF
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
byCodemotion
 
PDF
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
byDemi Ben-Ari
 
PDF
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
byDemi Ben-Ari
 
PPTX
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
byDevOpsDays Tel Aviv
 
PPTX
Time to say goodbye to your Nagios based setup
byCheck my Website
 
PDF
OSMC 2014: Time to say goodbye to your Nagios setup | Oliver Jan
byNETWAYS
 
PPTX
Java one2013 monitoringatscaleincloud
byRaju Kolluru
 
ODP
Devconf 17 metrics collection using open-source tools is easy
byYaniv Bronhaim
 
PDF
Monitoring and Scaling Redis at DataDog - Ilan Rabinovitch, DataDog
byRedis Labs
 
PDF
Combining Logs, Metrics, and Traces for Unified Observability
byElasticsearch
 
PDF
Monitoring Drupal In an Infrastructure as Code Age
byKris Buytaert
 
PDF
Thinking DevOps in the era of the Cloud - Demi Ben-Ari
byDemi Ben-Ari
 
PDF
Server Monitoring (Scaling while bootstrapped)
byAjibola Aiyedogbon
 
PPTX
Evolution of Monitoring and Prometheus (Dublin 2018)
byBrian Brazil
 
PDF
Observability cookbook JUGtoberfest Poznan 2024-10-16
bymichniczscribd
 
PDF
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
byRuby Meditation
 
PPTX
The Art of Container Monitoring
byDerek Chen
 
Monitoring Big Data Systems - "The Simple Way"
byDemi Ben-Ari
 
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
byCodemotion
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
byDemi Ben-Ari
 
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
byCodemotion
 
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
byDemi Ben-Ari
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
byDemi Ben-Ari
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
byDevOpsDays Tel Aviv
 
Time to say goodbye to your Nagios based setup
byCheck my Website
 
OSMC 2014: Time to say goodbye to your Nagios setup | Oliver Jan
byNETWAYS
 
Java one2013 monitoringatscaleincloud
byRaju Kolluru
 
Devconf 17 metrics collection using open-source tools is easy
byYaniv Bronhaim
 
Monitoring and Scaling Redis at DataDog - Ilan Rabinovitch, DataDog
byRedis Labs
 
Combining Logs, Metrics, and Traces for Unified Observability
byElasticsearch
 
Monitoring Drupal In an Infrastructure as Code Age
byKris Buytaert
 
Thinking DevOps in the era of the Cloud - Demi Ben-Ari
byDemi Ben-Ari
 
Server Monitoring (Scaling while bootstrapped)
byAjibola Aiyedogbon
 
Evolution of Monitoring and Prometheus (Dublin 2018)
byBrian Brazil
 
Observability cookbook JUGtoberfest Poznan 2024-10-16
bymichniczscribd
 
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
byRuby Meditation
 
The Art of Container Monitoring
byDerek Chen
 

Recently uploaded

PPTX
Internship ppt about multiple disease prediction
byGOUTHAMTr2
 
PDF
Macro Energy Trust Shock - current developments in the Venezuelan energy sector
byAurélie Dupassieux
 
PPTX
Quartiles_Ungrouped_Data_Grade_10-1.pptx
byxiandexter21
 
PDF
"Master Excel with CA Suvidha Chaplot: 40 Must-Know Tips for Efficiency"
byCA Suvidha Chaplot
 
PDF
Content Strategy Presentation.pdfjjjjhhhhh
byCerineAmmarkhodja1
 
PDF
"Top Excel Tips and Tricks: Conditional Formatting, Sorting, and More - Learn...
byCA Suvidha Chaplot
 
PDF
"Essential Excel Tips for Beginners & Professionals – 50+ Techniques to Maste...
byCA Suvidha Chaplot
 
PPTX
Merge Sort Algorithm: Analysis of Time and Space Complexity.pptx
byrigurahu
 
PDF
What is Context Engineering for Agentforce Developer and Architect
byParis Salesforce Developer Group
 
PDF
Whitepaper / No code end-user app development (2026)
byOptymyze
 
PDF
SF User Group - Account-Contact Relationship Feb 2026
byLukas Lunow
 
PPTX
Smart Crop prediction presentation for final year
bySurya386647
 
PPTX
Query Parsing: The SEO Gatekeeper That Decides Rankings
bySemantic SEO BD
 
PPT
Data Warehousing and Data Mining – Concepts, Architecture and Applications
bystark317700
 
PPTX
7.__Developing_a_Research_Proposal[1].pptx
bynahomeshetu340
 
PDF
Whitepaper Optymyze / Expand further (2026)
byOptymyze
 
PDF
제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계
byBOAZ Bigdata
 
PDF
Maricar Payaoan Belarma Portfolio :)))))
bymaricarbelarmava
 
PDF
TOS in History.pdf ENGLISH macro.pdf mnbv
bykristinegorre7
 
PDF
Machine_Psychology_2050_Strategic_Scenarios.pdf
byKambs Consulting
 
Internship ppt about multiple disease prediction
byGOUTHAMTr2
 
Macro Energy Trust Shock - current developments in the Venezuelan energy sector
byAurélie Dupassieux
 
Quartiles_Ungrouped_Data_Grade_10-1.pptx
byxiandexter21
 
"Master Excel with CA Suvidha Chaplot: 40 Must-Know Tips for Efficiency"
byCA Suvidha Chaplot
 
Content Strategy Presentation.pdfjjjjhhhhh
byCerineAmmarkhodja1
 
"Top Excel Tips and Tricks: Conditional Formatting, Sorting, and More - Learn...
byCA Suvidha Chaplot
 
"Essential Excel Tips for Beginners & Professionals – 50+ Techniques to Maste...
byCA Suvidha Chaplot
 
Merge Sort Algorithm: Analysis of Time and Space Complexity.pptx
byrigurahu
 
What is Context Engineering for Agentforce Developer and Architect
byParis Salesforce Developer Group
 
Whitepaper / No code end-user app development (2026)
byOptymyze
 
SF User Group - Account-Contact Relationship Feb 2026
byLukas Lunow
 
Smart Crop prediction presentation for final year
bySurya386647
 
Query Parsing: The SEO Gatekeeper That Decides Rankings
bySemantic SEO BD
 
Data Warehousing and Data Mining – Concepts, Architecture and Applications
bystark317700
 
7.__Developing_a_Research_Proposal[1].pptx
bynahomeshetu340
 
Whitepaper Optymyze / Expand further (2026)
byOptymyze
 
제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계
byBOAZ Bigdata
 
Maricar Payaoan Belarma Portfolio :)))))
bymaricarbelarmava
 
TOS in History.pdf ENGLISH macro.pdf mnbv
bykristinegorre7
 
Machine_Psychology_2050_Strategic_Scenarios.pdf
byKambs Consulting
 

Metrics & more

  • 1.
    Metrics & more howto monitor big data systems @scale!
  • 2.
  • 3.
    Why monitoring isimportant •  Tuning ! •  Detecting Bugs! •  Stability! •  Benchmarks! •  Capacity planning!
  • 4.
  • 5.
    Would you startbuilding own scales, when you would operate a real zoo? - What’s your mechanical engineering expertise? - How long does it take to get tools and raw material? - Who feeds the animals while being in the workshop? - When do we need it and could it be ‚in time‘?
  • 6.
    Let’s take something from theshelf and build a custom interface ‚load balancers‘!‚Custom Interface‘!
  • 7.
  • 8.
    •  Many VM’s& Apps - each one generates ~ 5-130 metrics in short intervals! •  Aggregation, Compromises on resolutions etc.! •  Transactions - each creates N log entries ! •  limit recording, time based indices + aliases! •  High throughput - high rate of logs & metrics! •  build a monitoring infrastructure (remember this)! !
  • 9.
    METRIC SOURCE! NUMBER OFMETRICS TO COLLECT! OS (CPU. Mem, Disk) 21 Hadoop 133 Hbase 68 Elasticsearch 62 Apache Storm 25 Total 309 ~ 3,1 Mio. data points per week x N machines ! ! Example - No. of metrics per application!
  • 10.
  • 11.
    •  Find outand define metrics to collect ! •  Install, configure collectd, statsd, graphite, …! •  Build, install / configure available agents! •  Define reports or arrange all collected metrics to dashboards e.g. grafana, …! •  This are the basics!! •  automate deployment for agents! #monitoringsucks
  • 12.
    #monitoringlove •  Integrate withthe organization ! •  alerting workflows + multi-user + security! •  Scale out: ! •  Distributed event processing (e.g. Kafka)! •  Scalable data stores (e.g. Elasticsearch, HBase)! •  Add intelligence: ! •  Machine learning for metrics & events! •  Alerting & Reporting based on it!
  • 13.
  • 15.
    What can wefind
 in the wild?
  • 16.
    Network Level •  Packets:loss, size, counts! •  Latency, jitters, delays! •  Bandwidth – total, per link, per service, ! •  Firewalls / security breaches! •  IDS, IPS – yet another malware detected ! •  On physical, transport, application layer, ...!
  • 17.
    Server Level •  DiskI/O! •  CPU load! •  Disk Space ! •  Memory! •  Logs / security / events / syslog!
  • 18.
    Standard Applications •  Webservers,Databases, Search Engines, MQ‘s! •  Request rates, disk space, partitions, locks, connections, queue sizes, cache sizes! •  Logfiles!
  • 19.
  • 20.
  • 21.
    Own Application 
 CustomMetrics & Logs •  Logs & API for measurement! •  Time measurements, KPI‘s, Usage tracking, Object counters, Click Streams!
  • 22.
    Application Traces •  Postmortem analysis! process.on (‚exit‘, heapdumpAndDie) •  Dtrace ! •  Call Traces, Error stacks! •  Heapdumps & Flamegraphs!
  • 23.
    Log files assource of metrics •  Simplest: log rate of an application! •  Generate Count for operations! •  Apply search and count related events! •  E.g. count slow operations! •  Extract values from logs ! •  Apply regex or field search to extract numbers !
  • 24.
    Logs2Metrics Logs! Index! Scheduled Queries! aggregateall messages matching e.g. „session opened“ every Minute e.g. on auth.log Custom ! Metric! Monitoring ! System!
  • 25.
    A Checklist forthe introduction of monitoring solutions
  • 26.
    Define your criterias • Coverage of monitors/agents! •  Quality of agents & setup! •  Multi-User Support! •  Reporting Capability & Secure Sharing! •  Alerting capabilities! •  Integrations / Notifications / API‘s! •  Estimate required resources !
  • 27.
    Map your landscape • Quantity of servers & applications to monitor! •  What are the components of your App-Stack?! •  Linux on AWS, NGINX, Node.js, REDIS, Elasticsearch! •  Which programming languages are used?! •  Can you find agents/monitors for all your ‚Apps‘?! •  List missing parts -> find other or build a monitor!
  • 28.
    Customizing – custom metrics/plugins • What metrics are relevant for each ‚App‘?! •  What is covered by existing agents?! •  How to aggregate each of this metrics? ! •  min, max, sum, avg! •  Pre-Aggregation vs. Query Time Aggregation!
  • 29.
    Dashboards •  Graphs! •  Whichmetrics belong together?! •  Display options ….! •  Query language ! •  Dashboards! •  What combination of graphs provides best insight?! •  Can you share and re-use arranged dashboards for similar setups or situations? ! •  Or do you need to configure it again for other servers?! •  Is sharing secured? Or just a link to your UI?!
  • 30.
    Alerts •  Threshold basedalerts! •  Status changes ! •  Heartbeat alerts! •  Anomaly detection! •  Challenges: Number of alert rules and queries ! & tuning ‘noise level’!
  • 31.
  • 32.
    •  Metrics show„something happens“! •  Logs provide evidence „what happened“! •  Faster insights by reporting them together! •  Correlate logs and metrics! •  Metrics could be created from logs! Integrate metrics & logs
  • 33.
  • 34.
    A brief overviewof 
 Centralizing Logs

  • 36.
    raw logs! parser! Log shipper!storage! Visualization! Kibana!Elasticsearch!Logstash! Where is the work?! Centralizing Logs with ELK ! files, syslog! Format adaption,! & transport! Tuning ! Maintenance! Queries! Security !
  • 37.
    •  Input: Unstructuredlog lines! •  Filter & Parser: Grok / RegEx! •  Output: Structured JSON! •  Forwarder: ! •  Elasticsearch, …!
  • 38.
    •  Schema: Definethe right Mapping •  Insert rate:! •  Use bulk indexing! •  Increase refresh time for higher insert rate! •  Volume: ! •  Aliases and time based indices! •  Memory usage: configure caching limits! Setup Elasticsearch
  • 39.
    •  How tosecure it? ! •  Proxies, Security plugins, Hosted Solutions! •  Queries and dashboard creation! •  generators/templates for specific setups! •  Learn Lucene query language!
  • 40.
  • 41.
    Thank you for! your attention! http://blog.sematext.com!