Docker メトリクスの可視化 in Japanese

1. VISUALIZATION OF DOCKER
METRICS
2015/12/09 by janus_wel

2. WHO?
高木健介janus_wel
archtect/programmer for full-stack

3. PURPOSE OF
VISUALIZATION ?

4. TO CHECK LIFE OR DEATH ?
should be covered by orchestration
ex) consul

5. TO USE RESOURCES EFFICIENTLY !!
figure out bottlenecks
decision a number of containers on a machine

6. TO MAKE DOCUMENTS
for customers
for executives

7. TO FOLLOW THE PDCA CYCLE
figure out influences of changes

8. COLLECTING METRICS USUALLY
is important to trace back

9. TOOLS
Docker remote API
InfluxDB

10. DOCKER REMOTE API
https://docs.docker.com/engine/reference/api/docker_remote_

11. REST LIKE API
The API tends to be REST, but for some
complex commands, like attach or pull, the
HTTP connection is hijacked to transport
STDOUT, STDIN, and STDERR.

12. AVAILABILITY
maybe, all operations in commands are available.
https://docs.docker.com/engine/reference/api/docker_remote_

13. FOR EXAMPLE
List containers
Create a container
Start a container
Stop a container
Remove a container
List Images
Build image from a Dockerfile
Create an image
Display system-wide information
Show the docker version information
...

14. REQUIREMENTS
needs some settings for access control
$DOCKER_OPTS in /etc/init/docker.conf on Ubuntu
$OPTIONS in /etc/sysconfig/docker on other Linux
if 192.0.2.1 is a client,
DOCKER_OPTS="-H 192.0.2.1:2376 -H unix:///var/run/docker.sock"
OPTIONS="-H 192.0.2.1:2376 -H unix:///var/run/docker.sock"
https://docs.docker.com/engine/userguide/basics/#bind-
docker-to-another-host-port-or-a-unix-socket

15. NOTE
make sure to hear from unix domain socket.
OPTIONS="-H 192.0.2.1:2376 -H unix:///var/run/docker.sock"
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
or, docker commands are not available.

16. FOR DOCKER MACHINE
machines created by docker-machine needs https
by default, in your ~/.docker/machine/machines/(machine
name)/
cert.pem
key.pem
this path is set by "$(docker-machine env (machine name))"
as $DOCKER_CERT_PATH

17. INFO
wget http://(machine ip):2376/info -O - -q | jq .
for docker-machine
wget
--no-check-certificate
--certificate=${DOCKER_CERT_PATH}/cert.pem
--private-key=${DOCKER_CERT_PATH}/key.pem
https://$(docker-machine ip (machine name)):2376/info
-O - -q | jq .
"curl" can't handle non-checking local cert, so use "wget"

19. STATS API
to get container's metrics, use stats API
GET /containers/(id)/stats
https://docs.docker.com/engine/reference/api/docker_remote_
container-stats-based-on-resource-usage

21. INFLUXDB
https://influxdb.com/

23. POINTS
schemaless
SQL like query
REST like interface

24. SCHEMALESS
there is no need to define somethings
when writing data, InfluxDB needs only
TABLE NAME and VALUE
these are optional
timestamp
tags
https://influxdb.com/docs/v0.9/concepts/schema_and_data_la

25. SQL LIKE QUERY
to read
SELECT * FROM cpu_load_short WHERE host=server01 AND region=us-west
-- ^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- table name tags
https://influxdb.com/docs/v0.9/query_language/data_explor

26. REST LIKE INTERFACE
# CREATE DATABASE
curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"
# INSERT
curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'cpu_load_
# SELECT
curl -G 'http://localhost:8086/query?db=mydb' --data-urlencode "q=SELECT valu
https://influxdb.com/docs/v0.9/guides/writing_data.html
https://influxdb.com/docs/v0.9/guides/querying_data.html

27. SYSTEM
pull archtecture
collect metrics minutely
run with various OSs
Linux
Windows
others can run Docker
for now, test only in Linux...

29. METRICS COLLECTOR
get metrics from Docker remote API and insert them
InfluxDB
in Ruby
requires gem "clockwork"
runs a collection minutely

30. VISUALIZE
in HTML and JavaScript
use chart.js
use some EcmaScript 2015 features
"let" statement
Promise
Object.assign
http://www.chartjs.org/
now available with only Google Chrome

31. DEMO
http://52.193.71.92/

32. METRICS
legend description type
rx_bytes received bytes counter
tx_bytes transferred bytes counter
total_cpu_usage cpu usage time % counter

33. SOURCES
https://github.com/januswel/docker-metrics

34. PROBLEMS
load
stats API is heavy
pull architecture
calling API is proportional to number of containers
handling mass containers
id is not human-friendly

35. PUSH ARCHITECTURE AND AGENT
to collect with low load, we need inspect /sys
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.stat
cat /sys/fs/cgroup/cpuacct/docker/<container-id>/cpuacct.stat
pull architecture is unfit to cat local files
agent is needed
https://docs.docker.com/engine/articles/runmetrics/

36. AFTER
increase collectable metrices
I don't know Docker metrics means what is...
agent-ize collector