Jon Hammant, Head of Cloud & DevOps for UK & EU for Epam Systems, presented an overview of using the ELK stack together with the Beats Plugin data shippers to provide detailed system metrics, network traffic, file analysis, and more. In addition, he provided an overview of how to monitor multiple Docker containers in a cloud native environment, with logs sent back to a central host.

1. ELK STACK WITH BEATS
November, 2016
Jon Hammant – Head of DevOps & Cloud UK/EU
EPAM Systems

2. INTRO
jonathan_hammant@epam.com
Head of Cloud & DevOps UK & EU
email me.
www.epam.com/careers
our careers portal.

3. ABOUT EPAM
Q1 2016
Revenue
$264.5M
CONSTANT
GROWTH
4
Continents
25
Countries
REVENUE BY
GEOGRAPHY
North America
Europe
APAC
CIS
58%
36%
2%
4%
20,000+
Engineers, designers and
consultants
FOUNDED IN
1993
US HEADQUARTERED
PUBLIC COMPANY
(NYSE:EPAM)
SERVICE
MIX
Software Engineering &
Product/Platform Development
QA and Test Automation
Managed Services
Infrastructure & Licensing
20+%
YOY organic growth
21
Reported Consecutive
Quarters
2016 Revenue
Guidance
$1.15B
Financial Services Travel & Consumer Software & Hi-tech
Media &
Entertainment
Life sciences &
Healthcare
INDUSTRY FOCUS
27% 24%
14%
21%
8%
Emerging
6%

4. PROBLEM
Too many syste ms an d n ot
e n ou gh visib ility
Massive ly d istrib u te d
In cre asin g n u mb e r of
microser vices
Fu ll d e -centralization
Painfu l p roce ss
We need log ging & metrics

5. WHY DO WE NEED METRICS?
Bloodletting
Starte d arou n d 100BCE
C ontin u e d u ntil 19 th C e ntu r y
H u n d re d s of Th ou san d s h ave
d ie d
It was d on e b e cau se p e op le
cared
Th ey ju st d id n ’t h ave right
th e information

6. WHY ELK?
Easy to setu p
Massive ly Powe rfu l
Scale s ve r y we ll
Op e n sou rce
Availab le as a se r vice
10 min u te setu p

7. • WHO WE SERVESO THAT’S IT?

8. • WHO WE SERVEWE NEED A WAY OF GETTING LOGS IN
We don’t want to run Syslog everywhere
Increasingly the applications are running on
cloud native systems
For a lightweight process we can’t add
heavyweight logging
No point writing loads of logging code

9. • WHO WE SERVEWHAT ARE BEATS?
Beats are the Elasticsearch platform for
single purpose, lightweight data shippers.
Designed to be small & portable
Logstash is still important for data
enrichment, reformatting
Replaces Logstash Forwarder & more

10. • WHO WE SERVECORE BEATS
F i l e b e a t Pa c ke t b e a t
M e t r i c b e a t W i n l o g b e a t

11. • WHO WE SERVEFILEBEAT
Simplest of the Beat plugins
Think of it as cat on steroids
Can send a text file to central host
Replaces Logstash Forwarder
Has concept of backpressure to stop
remote host being overloaded

12. • WHO WE SERVEMETRICBEAT
System level monitoring – CPU, Memory,
filesystem, IO statistics
Includes modules for common services –
Apache, Nginx, MongoDB, MySQL,
Postgres & more
Container ready – deploy one copy to
monitor all other Docker containers

13. • WHO WE SERVEPACKETBEAT
Network Packet Capture
Understands application layer protocols –
HTTP, DNS, ICMP, AMQP
Great for security and latency analysis
Can offer ”what went wrong” packet flow
analysis

14. • WHO WE SERVEWINLOGBEAT
Monitoring of Windows Log channels
Pull Windows logs along with Linux Logs

15. WHEN LOGGING & METRICS WORK
“Every th in g we kn ow in
aviation , eve r y ru le in th e
ru le b ook, eve r y p roce d u re
we h ave , we kn ow b e cau se
some on e somewh e re d ie d …
We h ave p u rch ase d at gre at
cost, lesson s literally
b rou ght with b lood ”
-
" Su lly" Su lle n b e rge r

16. COMMUNITY BEATS
Everything based on Go - libbeat
Over 34 different community created Beats now available
https://github.com/elastic/beats/blob/master/libbeat/
docs/communitybeats.asciidoc

17. • WHO WE SERVEOPENSOURCE HIGHLIGHTS
h t t p b e a t
Po l l a h t t p e n d p o i n t
my s q l b e a t
R u n a s c h e d u l e d q u e r y o n a my S q l
s e r v e r
M a n y m o r e u s e f u l B e a t s a v a i l a b l e
o r w r i t e y o u r o w n
C l o u d t ra i l b e a t , P i n g b e a t ,
C o n s u l b e a t e t c . .
exe c b e a t
Pe r i o d i c a l l y r u n c o m m a n d s a n d
s e n d o u t p u t a n d e r r o r

18. • WHO WE SERVEdockbeat
git clone clone https://github.com/Ingensi/dockbeat.git
wget https://github.com/Ingensi/dockbeat/releases/download/v1.0.0/dockbeat-v1.0.0-x86_64
chmod +x dockbeat-v1.0.0-x86_64
vi dockbeat/dockbeat.yml
Replace Docker_Socket & Elasticsearch or Logstash host
./dockbeat-v1.0.0-x86_64 -c dockbeat/dockbeat.yml -v –e
(can also be started in a container or swarm and permissioned)

19. • WHO WE SERVEEXAMPLE DASHBOARD - Metricbeat

20. • WHO WE SERVEUSE!
D i s c o v e r
L i s t h i s t o r i c C P U u s a g e
F i n d o u t w h i c h c o n t a i n e r s w e r e r u n
A n a l y ze fo r i n s e c u r e c o n t a i n e r s
M e t r i c s
S h o w r e a l t i m e m e t r i c s o f sy s t e m u s e
D i s p l ay b u s i n e s s v a l u e
V i e w t h e w h o l e sy s t e m a t o n e
V i s u a l i z e
L o o k b a c k a t p e r fo r m a n c e s t a t s
C o r r e l a t e c o s t / p e r fo r m a n c e a n d r e v e n u e
S h o w l o n g t e r m t r e n d s
A l e r t
U s e E l a s t A l e r t o n c o n t a i n e r s
B e i n fo r m e d w h e n t h i n g s s t o p
K n o w w h e n c a p a c i t y i s a n i s s u e

21. • WHO WE SERVESUCCESS!
Storage is cheap
Log everything and remove later
Packetbeat is extremely useful
go-audit (auditctl) and syslog are
fantastic
Black-box thinking, learn from mistakes