Downloaded 34 times
Uploaded byOutlyer
DOXLON November 2016 - ELK Stack and Beats
The document discusses the ELK stack and Beats, emphasizing their role in addressing issues of visibility across distributed microservices within organizations. It highlights various Beats components—Filebeat, Metricbeat, Packetbeat, and Winlogbeat—designed for lightweight data shipping and monitoring. Additionally, it outlines EPAM Systems' growth and industry focus, detailing the need for efficient logging and metrics in modern cloud-native systems.
More Related Content
Similar to DOXLON November 2016 - ELK Stack and Beats
DOXLON November 2016 - ELK Stack and Beats
- 1. ELK STACK WITHBEATS November, 2016 Jon Hammant – Head of DevOps & Cloud UK/EU EPAM Systems
- 2.
- 3. ABOUT EPAM Q1 2016 Revenue $264.5M CONSTANT GROWTH 4 Continents 25 Countries REVENUE BY GEOGRAPHY North America Europe APAC CIS 58% 36% 2% 4% 20,000+ Engineers, designers and consultants FOUNDED IN 1993 US HEADQUARTERED PUBLIC COMPANY (NYSE:EPAM) SERVICE MIX Software Engineering & Product/Platform Development QA and Test Automation Managed Services Infrastructure & Licensing 20+% YOY organic growth 21 Reported Consecutive Quarters 2016 Revenue Guidance $1.15B FinancialServices Travel & Consumer Software & Hi-tech Media & Entertainment Life sciences & Healthcare INDUSTRY FOCUS 27% 24% 14% 21% 8% Emerging 6%
- 4. PROBLEM Too many systems an d n ot e n ou gh visib ility Massive ly d istrib u te d In cre asin g n u mb e r of microser vices Fu ll d e -centralization Painfu l p roce ss We need log ging & metrics
- 5. WHY DO WENEED METRICS? Bloodletting Starte d arou n d 100BCE C ontin u e d u ntil 19 th C e ntu r y H u n d re d s of Th ou san d s h ave d ie d It was d on e b e cau se p e op le cared Th ey ju st d id n ’t h ave right th e information
- 6. WHY ELK? Easy to setu p Massive ly Powe rfu l Scale s ve r y we ll Op e n sou rce Availab le as a se r vice 10 min u te setu p
- 7.
- 8. • WHO WE SERVEWE NEEDA WAY OF GETTING LOGS IN We don’t want to run Syslog everywhere Increasingly the applications are running on cloud native systems For a lightweight process we can’t add heavyweight logging No point writing loads of logging code
- 9. • WHO WE SERVEWHAT AREBEATS? Beats are the Elasticsearch platform for single purpose, lightweight data shippers. Designed to be small & portable Logstash is still important for data enrichment, reformatting Replaces Logstash Forwarder & more
- 10. • WHO WE SERVECORE BEATS Fi l e b e a t Pa c ke t b e a t M e t r i c b e a t W i n l o g b e a t
- 11.
- 12. • WHO WE SERVEMETRICBEAT System level monitoring – CPU, Memory, filesystem, IO statistics Includes modules for common services – Apache, Nginx, MongoDB, MySQL, Postgres & more Container ready –deploy one copy to monitor all other Docker containers
- 13.
- 14.
- 15. WHEN LOGGING &METRICS WORK “Every th in g we kn ow in aviation , eve r y ru le in th e ru le b ook, eve r y p roce d u re we h ave , we kn ow b e cau se some on e somewh e re d ie d … We h ave p u rch ase d at gre at cost, lesson s literally b rou ght with b lood ” - " Su lly" Su lle n b e rge r
- 16.
- 17. • WHO WE SERVEOPENSOURCE HIGHLIGHTS ht t p b e a t Po l l a h t t p e n d p o i n t my s q l b e a t R u n a s c h e d u l e d q u e r y o n a my S q l s e r v e r M a n y m o r e u s e f u l B e a t s a v a i l a b l e o r w r i t e y o u r o w n C l o u d t ra i l b e a t , P i n g b e a t , C o n s u l b e a t e t c . . exe c b e a t Pe r i o d i c a l l y r u n c o m m a n d s a n d s e n d o u t p u t a n d e r r o r
- 18. • WHO WE SERVEdockbeat git cloneclone https://github.com/Ingensi/dockbeat.git wget https://github.com/Ingensi/dockbeat/releases/download/v1.0.0/dockbeat-v1.0.0-x86_64 chmod +x dockbeat-v1.0.0-x86_64 vi dockbeat/dockbeat.yml Replace Docker_Socket & Elasticsearch or Logstash host ./dockbeat-v1.0.0-x86_64 -c dockbeat/dockbeat.yml -v –e (can also be started in a container or swarm and permissioned)
- 19. • WHO WE SERVEEXAMPLE DASHBOARD- Metricbeat
- 20. • WHO WE SERVEUSE! D is c o v e r L i s t h i s t o r i c C P U u s a g e F i n d o u t w h i c h c o n t a i n e r s w e r e r u n A n a l y ze fo r i n s e c u r e c o n t a i n e r s M e t r i c s S h o w r e a l t i m e m e t r i c s o f sy s t e m u s e D i s p l ay b u s i n e s s v a l u e V i e w t h e w h o l e sy s t e m a t o n e V i s u a l i z e L o o k b a c k a t p e r fo r m a n c e s t a t s C o r r e l a t e c o s t / p e r fo r m a n c e a n d r e v e n u e S h o w l o n g t e r m t r e n d s A l e r t U s e E l a s t A l e r t o n c o n t a i n e r s B e i n fo r m e d w h e n t h i n g s s t o p K n o w w h e n c a p a c i t y i s a n i s s u e
- 21.