In this presentation, you’ll learn how use Metricbeat to monitor bare Docker containers and shipping container data to Elasticsearch and Kibana.

1. Monitoring Docker
Containers With Metricbeat,
Elasticsearch, and Kibana

2. In this presentation, you’ll learn how use Metricbeat
to monitor bare Docker containers and shipping
container data to Elasticsearch and Kibana.

3. Prerequisites
Examples in this tutorial were tested in the
following environment:
• Ubuntu 16.04 (Xenial Xerus)
• Metricbeat 6.3.2 download from the
apt repository
• Docker 18.03.1-ce

4. Prerequisites
NOTE: We assume that you already have a
working Docker environment on your system and
a few containers running.
If not, see the official Docker installation
guide and learn how to run Docker containers as
daemons.
You’ll need to have at least one container running
in Docker to ship some useful data to
Elasticsearch and Kibana.

5. Install Metricbeat
wget -qO - https://artifacts.elastic.co/GPG-KEY-
elasticsearch | sudo apt-key add -
To install Metricbeat, you first need to add
Elastic’s signing key used to verify the
downloaded package:
Next, add the Elastic repository to your repository
source list:
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable
main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

6. Install Metricbeat
sudo apt-get update && sudo apt-get install metricbeat
Finally, update the repos on your system and
install Metricbeat using apt-get:

7. Metricbeat General
Configuration
To run Metricbeat, you should configure input
(metrics sources like Docker), output (a remote
service or database to send these metrics to), and
various modules if needed. This configuration is
located in the inside the Metricbeat
folder.
Take a look at the edits we've made:
metricbeat.yml

8. #================ Modules configuration ============================
metricbeat.config.modules:
# Glob pattern for configuration loading path:
${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
reload.period: 10s
#==========Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
#_source.enabled: false
#===================Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana #index. Loading the dashboards is disabled by default
and can be
# enabled either by setting the options here, or by using the `-#setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true
#=======================Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via #the Kibana API. This requires a Kibana endpoint
configuration.
setup.kibana.host: "YOUR_KIBANA_HOST"
setup.kibana.protocol: "https"
setup.kibana.username: "YOUR_KIBANA_USERNAME"
setup.kibana.password: "YOUR_KIBANA_PASSWORD"
#====================Outputs =====================================
# Configure what output to use when sending the data collected by
# the beat.
#-------------------Elasticsearch output ---------------------------
output.elasticsearch:
hosts: ["YOUR_ELASTICSEARCH_HOST"]
username: "YOUR_ELASTICSEARCH_USERNAME"
password: "YOUR_ELASTICSEARCH_PASSWORD"

9. — we load module
configurations from external files to keep things
isolated. All module configuration files are located under
the folder so to target them we used glob
pattern. We have also enabled config reloading.
Metricbeat will periodically monitor our configuration
files, and, if any changes are detected, it will reload the
entire configuration.
Metricbeat General
Configuration
metricbeat.config.modules.path
/modules.d/ *.yml
There are several configuration options to pay attention to:

10. specifies the index template for
Metricbeat. Our Metricbeat index will have 1 shard and
will be compressed using type based on
high compression ratio.
Metricbeat General
Configuration
setup.template.setting
best_compression
setup.dashboards.enabled — we will be loading Kibana example
dashboards for Metricbeat. These dashboards include
visualization and searches examples for our metrics.

11. — for the dashboards to work, we need to
specify the Kibana endpoint. You'll need to enter the URL
of your Kibana host and any credentials (username/
password) if needed.
Metricbeat General
Configuration
setup.kibana
output.elasticsearch — specifies the output to which we
send Metricbeat metrics. We are using Elasticsearch, so
you'll need to provide Elasticsearch host, protocol, and
credentials if needed.

12. Metricbeat Docker Module
To fetch metrics from Docker containers, we are
going to use Metricbeat Docker module. It comes
with a number of default metricsets we, such as
container cpu diskio healthcheck info memory network&, , , , ,

13. Metricbeat Docker Module
First, we need to manually enable the Docker module
because we load external configuration files into our general
configuration. The default Docker module configuration sits
in the directory. We can enable or disable any
module configuration under by running
commands. For example, to enable the config in the
directory, you can run:
modules.d modules.d
modules enable modules disable
docker modules.d
/metricbeat modules enable docker

14. Metricbeat Docker Module
Now, as the module is enabled, let’s tweak its configuration.
This is how it looks like in the file:docker.yml
metricsets:
- "container"
- "cpu"
- "diskio"
- "healthcheck"
- "info"
- "image"
- "memory"
- "network"
hosts: ["unix:///var/run/docker.sock"]
period: 10s
enabled: true

15. Metricbeat Docker Module
This is a minimal configuration suffice to get Metricbeat
going. We have specified 8 metricsets including “image”
metricset not included by default. Also, your Docker module
needs access to the Docker daemon.
By default, Docker listens on the Unix socket
We can use this socket to communicate with the daemon
from within a container. Through this endpoint, Docker
exposes the Docker API which can be used to get a stream
of all events and statistics generated by Docker.
"unix:///var/run/docker.sock"

16. Metricbeat Docker Module
The next important configuration field we need to mention is
This field defines how often Metricbeat accesses the Docker API.
According to the official Metricbeat documentation, it is strongly
recommended to run a Docker module with a period of at least 3
seconds or longer.
That is because the request to Docker API takes up to 2 seconds
itself, so specifying less than 3 seconds can cause request
timeouts and no data returned to those requests.
period

17. Metricbeat Docker Module
Great! Now, everything is ready to run Metricbeat. One last advice
is to have your Metricbeat instance as close to Docker as
possible (preferably on the same host) to minimize network
latency.
On Linux, you can run Metricbeat in the shell specifying the
config file as a parameter.
sudo ./metricbeat -e -c metricbeat.yml

18. Metricbeat Docker Module
Alternatively, you can start Metricbeat as a service to run at
startup:
sudo service metricbeat start
Almost immediately after being started, Metricbeat will begin
sending Docker metrics to Elasticsearch index. You can verify
this by curling your Elasticsearch host:
curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size
pri.store.size
yellow open metricbeat-6.3.2-2018.08.09 BikpOgqQR-pU_SuKrm5vw 1 1
2374 0 1.4mb 1.3mb

19. Metricbeat Docker Module
As you see, our index was created! Now, you can create an index
pattern for this index in Kibana and access metrics directly from
the Kibana dashboard.
Log in to your Kibana dashboard and follow the simple steps for
creating a new Index Pattern under Management -> Index Patterns
-> Create Index Pattern.
If everything is ok, you’ll see the index mapping similar to the
screenshot in the next slide.

21. Metricbeat Docker Module
You can see that Metricbeat created fields for various Docker
module metricsets such as container, CPU, etc. You can also find
batches of metrics shipped to Elasticsearch by clicking on
Discovery tab in Kibana dashboard. You should see something
like this:

23. Metricbeat Docker Module
Awesome! As you remember, we loaded Kibana dashboards, so
you can access example visualizations of the Metricbeat data in
Kibana.

25. Metricbeat Docker Module
Pretty cool, isn’t it? In the dashboard, we have the statistics of
running containers as well some data about their memory and
CPU usage. You can experiment with Docker data by creating
your custom visualizations. Check out our tutorial about building
visualizations for Metricbeat data in Kibana.

26. Conclusion
That’s it! You have learned how to monitor Docker containers
using Metricbeat, Elasticsearch, and Kibana.
Metricbeat Docker module exposes key metrics and events
provided by the Docker API. Making them accessible for
subsequent processing and analysis in Kibana is really a great
monitoring solution that requires minimal configuration.

27. Conclusion
Installing Metricbeat and configuring Docker module should not
take more than half an hour.
Once the Docker container data is available in Elasticsearch, you
can use powerful Kibana visualization tools like Visual Builder
and Timelion to analyze your Docker containers.
This presentation originally appeared as an article on Qbox blog:
Monitoring Docker Containers With Metricbeat, Elasticsearch, and Kibana

28. Resources
Read the Qbox blog
Watch our YouTube channel
Read our Medium tutorials
We have a wealth of resources to help you with your
Elasticsearch journey. Here are just a few sites for you check out:

29. Visit the Qbox website to
learn about our hosted
Elasticsearch service
qbox.io