SlideShare a Scribd company logo

ELK Stack

•
•
Eberhard Wolff
Eberhard Wolff

Short presentation about the ELK stack (Elasticsearch, Logstash, Kibana) running on top of Docker / Vagrant.

Technology
1 of 19
Download to read offline
ELK
Eberhard Wolff - @ewolff Docker •  No true virtualization •  Linux Containers (lxc) •  i.e. shared kernel •  i.e. separate file systems
Eberhard Wolff - @ewolff Docker File Systems •  Read only base images •  +read/write image •  Can be stacked
Eberhard Wolff - @ewolff Docker •  Linux only •  So: Run it in VM •  Controlled by Vagrant
Docker Demo
Eberhard Wolff - @ewolff Communication Between Docker Container •  Via ports •  Via data volumes
Docker is a Component System
Eberhard Wolff - @ewolff Example •  Log file analysis •  ELK Stack •  Elasticsearch: Store •  Logstash: Parse •  Kibana: Web based GUI •  Install three servers •  Automate installation ... L
Eberhard Wolff - @ewolff Dockerized ELK Stack •  1 Vagrantfile (25 lines) •  6 Dockerfiles (total 28 lines) •  1 HTML file (11 lines) •  1 Config (Logstash) (29 lines) •  Total: 93 lines, 9 files •  Including one RegExp L
Eberhard Wolff - @ewolff Vagrant VM Container user-registration 8080 8081 Port 8081 Container elasticsearch Container logstash 9200 9200 Port 9200 9200 link Container kibana Port 8080 8080 8080 Browser Volume /log
Demo ELK Stack
Eberhard Wolff - @ewolff date time level 1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : email=eberhard.wolff@gmail.com geloesc pid thread class logmessage
Eberhard Wolff - @ewolff Scaling Elasticsearch •  More servers •  Shards •  Easy to set up •  Index per day or week or month •  Smaller dataset
Eberhard Wolff - @ewolff Ideas about Logstash •  No parsing •  Send JSON
Eberhard Wolff - @ewolff Break Logstash apart •  Logstash does parsing and communication •  Not too sensible in a distributed sytem •  Separate process per responsibility
Eberhard Wolff - @ewolff Shipper •  Ship data from logging system •  Logstash •  Beaver https://github.com/josegonzalez/ beaver •  Woodchuck https://github.com/danryan/ woodchuck
Eberhard Wolff - @ewolff Broker •  Buffer data for later processing •  Redis •  In-Memory NoSQL database •  Small •  Fast •  ...and then Parse (Logstash)
Eberhard Wolff - @ewolff Links •  https://github.com/ewolff/user- registration
Eberhard Wolff - @ewolff Thank You!! eberhard.wolff@gmail.com @ewolff

Recommended

Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack PresentationAmr Alaa Yassen
 
Elk - An introduction
Elk - An introductionElk - An introduction
Elk - An introductionHossein Shemshadi
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack IntroductionVikram Shinde
 
ELK Stack
ELK StackELK Stack
ELK StackPhuc Nguyen
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackRohit Sharma
 
Log analysis using elk
Log analysis using elkLog analysis using elk
Log analysis using elkRushika Shah
 
Elk
Elk Elk
Elk Caleb Wang
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELKGeert Pante
 

Recommended

Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack PresentationAmr Alaa Yassen
 
Elk - An introduction
Elk - An introductionElk - An introduction
Elk - An introductionHossein Shemshadi
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack IntroductionVikram Shinde
 
ELK Stack
ELK StackELK Stack
ELK StackPhuc Nguyen
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackRohit Sharma
 
Log analysis using elk
Log analysis using elkLog analysis using elk
Log analysis using elkRushika Shah
 
Elk
Elk Elk
Elk Caleb Wang
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELKGeert Pante
 
ELK introduction
ELK introductionELK introduction
ELK introductionWaldemar Neto
 
Elastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaElastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaSpringPeople
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELKYuHsuan Chen
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackRich Lee
 
Introducing ELK
Introducing ELKIntroducing ELK
Introducing ELKAllBits BVBA (freelancer)
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
 
The Elastic ELK Stack
The Elastic ELK StackThe Elastic ELK Stack
The Elastic ELK Stackenterprisesearchmeetup
 
Elk stack
Elk stackElk stack
Elk stackJilles van Gurp
 
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...Edureka!
 
Kibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stackKibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stackSylvain Wallez
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stackVikrant Chauhan
 
Elk devops
Elk devopsElk devops
Elk devopsIdeato
 
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - PreviewVictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics
 
Log analytics with ELK stack
Log analytics with ELK stackLog analytics with ELK stack
Log analytics with ELK stackAWS User Group Bengaluru
 
Prometheus - basics
Prometheus - basicsPrometheus - basics
Prometheus - basicsJuraj Hantak
 
Elastic Observability
Elastic Observability Elastic Observability
Elastic Observability FaithWestdorp
 
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashAmazon Web Services
 
Elastic 101 - Get started
Elastic 101 - Get startedElastic 101 - Get started
Elastic 101 - Get startedIsmaeel Enjreny
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For OperatorsKevin Brockhoff
 
Introducing log analysis to your organization
Introducing log analysis to your organization Introducing log analysis to your organization
Introducing log analysis to your organization Sematext Group, Inc.
 
Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is BeautifulEberhard Wolff
 
Java Architectures - a New Hope
Java Architectures - a New HopeJava Architectures - a New Hope
Java Architectures - a New HopeEberhard Wolff
 

More Related Content

What's hot

Elastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaElastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaSpringPeople
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELKYuHsuan Chen
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackRich Lee
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
 
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...Edureka!
 
Kibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stackKibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stackSylvain Wallez
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stackVikrant Chauhan
 
Elk devops
Elk devopsElk devops
Elk devopsIdeato
 
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - PreviewVictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics
 
Prometheus - basics
Prometheus - basicsPrometheus - basics
Prometheus - basicsJuraj Hantak
 
Elastic Observability
Elastic Observability Elastic Observability
Elastic Observability FaithWestdorp
 
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashAmazon Web Services
 
Elastic 101 - Get started
Elastic 101 - Get startedElastic 101 - Get started
Elastic 101 - Get startedIsmaeel Enjreny
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For OperatorsKevin Brockhoff
 
Introducing log analysis to your organization
Introducing log analysis to your organization Introducing log analysis to your organization
Introducing log analysis to your organization Sematext Group, Inc.
 

What's hot (20)

ELK introduction
ELK introductionELK introduction
ELK introduction
 
Elastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaElastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & Kibana
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELK
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stack
 
Introducing ELK
Introducing ELKIntroducing ELK
Introducing ELK
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log Management
 
The Elastic ELK Stack
The Elastic ELK StackThe Elastic ELK Stack
The Elastic ELK Stack
 
Elk stack
Elk stackElk stack
Elk stack
 
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
 
Kibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stackKibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stack
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stack
 
Elk devops
Elk devopsElk devops
Elk devops
 
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - PreviewVictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - Preview
 
Log analytics with ELK stack
Log analytics with ELK stackLog analytics with ELK stack
Log analytics with ELK stack
 
Prometheus - basics
Prometheus - basicsPrometheus - basics
Prometheus - basics
 
Elastic Observability
Elastic Observability Elastic Observability
Elastic Observability
 
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
 
Elastic 101 - Get started
Elastic 101 - Get startedElastic 101 - Get started
Elastic 101 - Get started
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For Operators
 
Introducing log analysis to your organization
Introducing log analysis to your organization Introducing log analysis to your organization
Introducing log analysis to your organization
 

Similar to ELK Stack

Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is BeautifulEberhard Wolff
 
Java Architectures - a New Hope
Java Architectures - a New HopeJava Architectures - a New Hope
Java Architectures - a New HopeEberhard Wolff
 
NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?Eberhard Wolff
 
Micro Service – The New Architecture Paradigm
Micro Service – The New Architecture ParadigmMicro Service – The New Architecture Paradigm
Micro Service – The New Architecture ParadigmEberhard Wolff
 
High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...Eberhard Wolff
 
Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?Eberhard Wolff
 
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Docker, Inc.
 
DockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and DockerDockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and DockerTaylor Brown
 
Java Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short VersionJava Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short VersionEberhard Wolff
 
Java Application Servers Are Dead!
Java Application Servers Are Dead!Java Application Servers Are Dead!
Java Application Servers Are Dead!Eberhard Wolff
 
Microservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring CloudMicroservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring CloudEberhard Wolff
 
Openstack Cactus Survey
Openstack Cactus SurveyOpenstack Cactus Survey
Openstack Cactus SurveyPjack Chen
 
Introduction to Docker
Introduction to DockerIntroduction to Docker
Introduction to DockerAlan Forbes
 
Spring Boot
Spring BootSpring Boot
Spring Bootgedoplan
 
Docker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker NetworkingDocker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker NetworkingDocker, Inc.
 
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anyninesCloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anyninesanynines GmbH
 
Magnum Networking Update
Magnum Networking UpdateMagnum Networking Update
Magnum Networking UpdateDaneyon Hansen
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with DockerRavindu Fernando
 
Amazon Elastic Beanstalk
Amazon Elastic BeanstalkAmazon Elastic Beanstalk
Amazon Elastic BeanstalkEberhard Wolff
 

Similar to ELK Stack (20)

Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is Beautiful
 
Java Architectures - a New Hope
Java Architectures - a New HopeJava Architectures - a New Hope
Java Architectures - a New Hope
 
NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?
 
Micro Service – The New Architecture Paradigm
Micro Service – The New Architecture ParadigmMicro Service – The New Architecture Paradigm
Micro Service – The New Architecture Paradigm
 
High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...
 
Docker Dojo
Docker DojoDocker Dojo
Docker Dojo
 
Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?
 
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
 
DockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and DockerDockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and Docker
 
Java Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short VersionJava Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short Version
 
Java Application Servers Are Dead!
Java Application Servers Are Dead!Java Application Servers Are Dead!
Java Application Servers Are Dead!
 
Microservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring CloudMicroservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring Cloud
 
Openstack Cactus Survey
Openstack Cactus SurveyOpenstack Cactus Survey
Openstack Cactus Survey
 
Introduction to Docker
Introduction to DockerIntroduction to Docker
Introduction to Docker
 
Spring Boot
Spring BootSpring Boot
Spring Boot
 
Docker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker NetworkingDocker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker Networking
 
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anyninesCloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
 
Magnum Networking Update
Magnum Networking UpdateMagnum Networking Update
Magnum Networking Update
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with Docker
 
Amazon Elastic Beanstalk
Amazon Elastic BeanstalkAmazon Elastic Beanstalk
Amazon Elastic Beanstalk
 

More from Eberhard Wolff

Architectures and Alternatives
Architectures and AlternativesArchitectures and Alternatives
Architectures and AlternativesEberhard Wolff
 
The Frontiers of Continuous Delivery
The Frontiers of Continuous DeliveryThe Frontiers of Continuous Delivery
The Frontiers of Continuous DeliveryEberhard Wolff
 
Four Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, AsyncFour Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, AsyncEberhard Wolff
 
Microservices - not just with Java
Microservices - not just with JavaMicroservices - not just with Java
Microservices - not just with JavaEberhard Wolff
 
Deployment - Done Right!
Deployment - Done Right!Deployment - Done Right!
Deployment - Done Right!Eberhard Wolff
 
Data Architecture not Just for Microservices
Data Architecture not Just for MicroservicesData Architecture not Just for Microservices
Data Architecture not Just for MicroservicesEberhard Wolff
 
How to Split Your System into Microservices
How to Split Your System into MicroservicesHow to Split Your System into Microservices
How to Split Your System into MicroservicesEberhard Wolff
 
Microservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale AgileMicroservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale AgileEberhard Wolff
 
How Small Can Java Microservices Be?
How Small Can Java Microservices Be?How Small Can Java Microservices Be?
How Small Can Java Microservices Be?Eberhard Wolff
 
Data Architecturen Not Just for Microservices
Data Architecturen Not Just for MicroservicesData Architecturen Not Just for Microservices
Data Architecturen Not Just for MicroservicesEberhard Wolff
 
Microservices: Redundancy=Maintainability
Microservices: Redundancy=MaintainabilityMicroservices: Redundancy=Maintainability
Microservices: Redundancy=MaintainabilityEberhard Wolff
 
Self-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to MicroservicesSelf-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to MicroservicesEberhard Wolff
 
Microservices Technology Stack
Microservices Technology StackMicroservices Technology Stack
Microservices Technology StackEberhard Wolff
 
Software Architecture for Innovation
Software Architecture for InnovationSoftware Architecture for Innovation
Software Architecture for InnovationEberhard Wolff
 
Five (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous DeliveryFive (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous DeliveryEberhard Wolff
 
Nanoservices and Microservices with Java
Nanoservices and Microservices with JavaNanoservices and Microservices with Java
Nanoservices and Microservices with JavaEberhard Wolff
 
Microservices: Architecture to Support Agile
Microservices: Architecture to Support AgileMicroservices: Architecture to Support Agile
Microservices: Architecture to Support AgileEberhard Wolff
 
Microservices: Architecture to scale Agile
Microservices: Architecture to scale AgileMicroservices: Architecture to scale Agile
Microservices: Architecture to scale AgileEberhard Wolff
 
Microservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three BuzzwordsMicroservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three BuzzwordsEberhard Wolff
 

More from Eberhard Wolff (20)

Architectures and Alternatives
Architectures and AlternativesArchitectures and Alternatives
Architectures and Alternatives
 
Beyond Microservices
Beyond MicroservicesBeyond Microservices
Beyond Microservices
 
The Frontiers of Continuous Delivery
The Frontiers of Continuous DeliveryThe Frontiers of Continuous Delivery
The Frontiers of Continuous Delivery
 
Four Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, AsyncFour Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, Async
 
Microservices - not just with Java
Microservices - not just with JavaMicroservices - not just with Java
Microservices - not just with Java
 
Deployment - Done Right!
Deployment - Done Right!Deployment - Done Right!
Deployment - Done Right!
 
Data Architecture not Just for Microservices
Data Architecture not Just for MicroservicesData Architecture not Just for Microservices
Data Architecture not Just for Microservices
 
How to Split Your System into Microservices
How to Split Your System into MicroservicesHow to Split Your System into Microservices
How to Split Your System into Microservices
 
Microservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale AgileMicroservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale Agile
 
How Small Can Java Microservices Be?
How Small Can Java Microservices Be?How Small Can Java Microservices Be?
How Small Can Java Microservices Be?
 
Data Architecturen Not Just for Microservices
Data Architecturen Not Just for MicroservicesData Architecturen Not Just for Microservices
Data Architecturen Not Just for Microservices
 
Microservices: Redundancy=Maintainability
Microservices: Redundancy=MaintainabilityMicroservices: Redundancy=Maintainability
Microservices: Redundancy=Maintainability
 
Self-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to MicroservicesSelf-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to Microservices
 
Microservices Technology Stack
Microservices Technology StackMicroservices Technology Stack
Microservices Technology Stack
 
Software Architecture for Innovation
Software Architecture for InnovationSoftware Architecture for Innovation
Software Architecture for Innovation
 
Five (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous DeliveryFive (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous Delivery
 
Nanoservices and Microservices with Java
Nanoservices and Microservices with JavaNanoservices and Microservices with Java
Nanoservices and Microservices with Java
 
Microservices: Architecture to Support Agile
Microservices: Architecture to Support AgileMicroservices: Architecture to Support Agile
Microservices: Architecture to Support Agile
 
Microservices: Architecture to scale Agile
Microservices: Architecture to scale AgileMicroservices: Architecture to scale Agile
Microservices: Architecture to scale Agile
 
Microservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three BuzzwordsMicroservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three Buzzwords
 

Recently uploaded

Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 

Recently uploaded (20)

Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 

ELK Stack

  • 1. ELK
  • 2. Eberhard Wolff - @ewolff Docker •  No true virtualization •  Linux Containers (lxc) •  i.e. shared kernel •  i.e. separate file systems
  • 3. Eberhard Wolff - @ewolff Docker File Systems •  Read only base images •  +read/write image •  Can be stacked
  • 4. Eberhard Wolff - @ewolff Docker •  Linux only •  So: Run it in VM •  Controlled by Vagrant
  • 6. Eberhard Wolff - @ewolff Communication Between Docker Container •  Via ports •  Via data volumes
  • 8. Eberhard Wolff - @ewolff Example •  Log file analysis •  ELK Stack •  Elasticsearch: Store •  Logstash: Parse •  Kibana: Web based GUI •  Install three servers •  Automate installation ... L
  • 9. Eberhard Wolff - @ewolff Dockerized ELK Stack •  1 Vagrantfile (25 lines) •  6 Dockerfiles (total 28 lines) •  1 HTML file (11 lines) •  1 Config (Logstash) (29 lines) •  Total: 93 lines, 9 files •  Including one RegExp L
  • 10. Eberhard Wolff - @ewolff Vagrant VM Container user-registration 8080 8081 Port 8081 Container elasticsearch Container logstash 9200 9200 Port 9200 9200 link Container kibana Port 8080 8080 8080 Browser Volume /log
  • 12. Eberhard Wolff - @ewolff date time level 1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : email=eberhard.wolff@gmail.com geloesc pid thread class logmessage
  • 13. Eberhard Wolff - @ewolff Scaling Elasticsearch •  More servers •  Shards •  Easy to set up •  Index per day or week or month •  Smaller dataset
  • 14. Eberhard Wolff - @ewolff Ideas about Logstash •  No parsing •  Send JSON
  • 15. Eberhard Wolff - @ewolff Break Logstash apart •  Logstash does parsing and communication •  Not too sensible in a distributed sytem •  Separate process per responsibility
  • 16. Eberhard Wolff - @ewolff Shipper •  Ship data from logging system •  Logstash •  Beaver https://github.com/josegonzalez/ beaver •  Woodchuck https://github.com/danryan/ woodchuck
  • 17. Eberhard Wolff - @ewolff Broker •  Buffer data for later processing •  Redis •  In-Memory NoSQL database •  Small •  Fast •  ...and then Parse (Logstash)
  • 18. Eberhard Wolff - @ewolff Links •  https://github.com/ewolff/user- registration
  • 19. Eberhard Wolff - @ewolff Thank You!! eberhard.wolff@gmail.com @ewolff