This document discusses how to build custom data shippers or "Beats" using the Elastic Beats framework. It recommends a modular design, automating setup tasks, including documentation and tests, and leveraging existing Beats infrastructure to make the custom Beat easy to contribute to and share. The goal is to help new Beat creators get started by providing common building blocks and tools out of the box.

1. Batteries
Included1 — Aaron Aldrich - @CrayZeigh

2. Aaron Aldrich
Community Engineer
, , DevOpsCT, The Elasticast
@CrayZeigh
aaron.aldrich@elastic.co
2 — Aaron Aldrich - @CrayZeigh

3. COMMUNITY
3 — Aaron Aldrich - @CrayZeigh

4. Pull requests welcome IS OPEN SOURCE SPEAK FOR
Go away, kid, you're bothering me.
4 — Aaron Aldrich - @CrayZeigh

5. Welcome
BY
ENABLING SUCCESS
5 — Aaron Aldrich - @CrayZeigh

6. Where are we?
6 — Aaron Aldrich - @CrayZeigh

7. THINGS WE'RE NOT TALKING ABOUT
7 — Aaron Aldrich - @CrayZeigh

8. THINGS WE'RE NOT TALKING ABOUT
▸ root vegetables
7 — Aaron Aldrich - @CrayZeigh

9. THINGS WE'RE NOT TALKING ABOUT
▸ root vegetables
▸ things by Dre
7 — Aaron Aldrich - @CrayZeigh

10. Killer Tofu
8 — Aaron Aldrich - @CrayZeigh

11. 9 — Aaron Aldrich - @CrayZeigh

12. 10 — Aaron Aldrich - @CrayZeigh

13. EXISTING BEATS INTENTED USE
filebeat logs, csv, text
metricbeat metrics
packetbeat transaction logs & flow metrics
winlogbeat Windows events
auditbeat system activity, changes
heartbeat network stats
11 — Aaron Aldrich - @CrayZeigh

14. 12 — Aaron Aldrich - @CrayZeigh

15. Automate
13 — Aaron Aldrich - @CrayZeigh

16. BY DESIGN
14 — Aaron Aldrich - @CrayZeigh

17. FILEBEAT MODULES
.../filebeat/modules.d/
apache2.yml.disabled iis.yml.disabled mongodb.yml.disabled postgresql.yml.disabled
auditd.yml.disabled kafka.yml.disabled mysql.yml redis.yml.disabled
elasticsearch.yml.disabled kibana.yml.disabled nginx.yml system.yml
icinga.yml.disabled logstash.yml.disabled osquery.yml.disabled traefik.yml.disabled
15 — Aaron Aldrich - @CrayZeigh

18. METRICBEAT MODULES
.../metricbeat/modules.d/
aerospike.yml.disabled etcd.yml.disabled kibana.yml.disabled mysql.yml system.yml
apache.yml.disabled golang.yml.disabled kubernetes.yml.disabled nginx.yml uwsgi.yml.disabled
ceph.yml.disabled graphite.yml.disabled kvm.yml.disabled php_fpm.yml.disabled vsphere.yml.disabled
couchbase.yml.disabled haproxy.yml.disabled logstash.yml.disabled postgresql.yml.disabled windows.yml.disabled
docker.yml.disabled http.yml.disabled memcached.yml.disabled prometheus.yml.disabled zookeeper.yml.disabled
dropwizard.yml.disabled jolokia.yml.disabled mongodb.yml.disabled rabbitmq.yml.disabled
elasticsearch.yml.disabled kafka.yml.disabled munin.yml.disabled redis.yml.disabled
16 — Aaron Aldrich - @CrayZeigh

19. > filebeat setup
Flags:
--dashboards Setup dashboards only
--machine-learning Setup machine learning job configurations only
--template Setup index template only
--modules string List of enabled modules (comma separated)
17 — Aaron Aldrich - @CrayZeigh

20. FOR CONTRIBUTORS
python $GOPATH/src/github.com/elastic/beats/script/generate.py
18 — Aaron Aldrich - @CrayZeigh

21. > make setup
mkdir -p vendor/github.com/elastic/
cp -R /Users/aaronaldrich/go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git
/Library/Developer/CommandLineTools/usr/bin/make update
New python executable in /.../github.com/crayzeigh/examplebeat/build/python-env/bin/python
Installing setuptools, pip, wheel...done.
Updating generated files for examplebeat
-- The index pattern was created under /.../examplebeat/_meta/kibana/5/index-pattern/examplebeat.json
-- The index pattern was created under /.../examplebeat/_meta/kibana/6/index-pattern/examplebeat.json
Setup, Binary builds and index-pattern updates handled through
Makefiles
19 — Aaron Aldrich - @CrayZeigh

22. MAIN-IMPORT TRICK
package main
import (
"os"
"github.com/elastic/beats/libbeat/beat"
"github.com/elastic/beats/metricbeat/beater"
// load my plugins
_ "my/module/package/name"
)
func main() {
if err := beat.Run("mymetricbeat", "", beater.New); err != nil {
os.Exit(1)
}
}
20 — Aaron Aldrich - @CrayZeigh

23. DOCUMENTATION INCLUDED
fields.yml
21 — Aaron Aldrich - @CrayZeigh

24. WHAT IS IT?
For all events generated by beats we provide:
▸ Elasticsearch index template
▸ Kibana index pattern
▸ Documentation!
▸ Common definition and documentation in fields.yml
22 — Aaron Aldrich - @CrayZeigh

25. SYNTAX
field ::= #'type' must be group
name: <field name> fields list ::=
type: <type> - <field>
[format: <format>] - <field>
description: <docstring> ...
[fields: <fields list>]
type ::= group, long, keyword
format ::= bytes, percent, ...
23 — Aaron Aldrich - @CrayZeigh

26. EXAMPLE
- name: memory
type: group
description: >
'memory' contains local memory stats.
fields:
- name: total
type: long
format: bytes
description: >
Total memory
24 — Aaron Aldrich - @CrayZeigh

27. SHIP THE TEST ENVIRONMENT
-OR-
DOCKER ALL THE THINGS
25 — Aaron Aldrich - @CrayZeigh

28. Environment
> ls vendor/github.com/elastic/beats/testing/environments/
2x.yml Dockerfile README.md
docker local.yml 5.0.0-cgroups.yml
Makefile args.yml latest.yml
snapshot.yml
26 — Aaron Aldrich - @CrayZeigh

29. Environment
/demobeat/vendor/github.com/elastic/beats/testing/environments> make start
# This is run every time to make sure the environment is up-to-date
docker-compose -f snapshot.yml -f local.yml build
...
Building kibana...
Building elasticsearch...
Building logstash...
docker-compose -f snapshot.yml -f local.yml run beat bash
Creating environments_elasticsearch_1 ... done
Creating environments_kibana_1 ... done
Creating environments_logstash_1 ... done
root@3a834aedbb86:/#
27 — Aaron Aldrich - @CrayZeigh

30. <DEMO SLIDE>
28 — Aaron Aldrich - @CrayZeigh

31. Further test shipping
29 — Aaron Aldrich - @CrayZeigh

32. MAKE SHARING
EASY
30 — Aaron Aldrich - @CrayZeigh

33. amazonbeat, apachebeat, apexbeat, burrowbeat, hsnburrowbeat, cassandrabeat, cloudflarebeat,
cloudfrontbeat, cloudtrailbeat, cloudwatchmetricbeat, cloudwatchlogsbeat, collectbeat,
connbeat, consulbeat, dockbeat, elasticbeat, etcbeat, execbeat, factbeat, fastcombeat, flowbeat,
gabeat, gcsbeat, githubbeat, gpfsbeat, hackerbeat, hsbeat, httpbeat, hwsensorsbeat, icingabeat,
iobeat, jmxproxybeat, journalbeat, kafkabeat, kafabeat2, krakenbeat, lmsensorbeat,
logstashbeat, mcqbeat, mongobeat, mqttbeat, mysqlbeat, nagioscheckbeat, nginxbeat,
nginxuipstreambeat, nsqbeat, nvidiagpubeat, openconfigbeat, packagebeat, phpfpmbeat,
pingbeat, prombeat, prometheusbeat, protologbeat, pubsubbeat, redditbeat, redisbeat, retsbeat,
rsbeat, saltbeat, serialbeat, springbeat, tracebeat, twitterbeat, udpbeat, udplogbeat,
unifiedbeat, uwsgibeat, varnishlogbeat, varnishstatbeat, vaultbeat, wmibeat
HTTPS://WWW.ELASTIC.CO/GUIDE/EN/BEATS/LIBBEAT/CURRENT/COMMUNITY-BEATS.HTML
31 — Aaron Aldrich - @CrayZeigh

34. Help YOUR Contributors CONTRIBUTE
1. Modular design
2. Automate tedium
3. Include documentation
4. Include testing
5. Make sharing easy
32 — Aaron Aldrich - @CrayZeigh

35. BATTERIES
INCLUDED
33 — Aaron Aldrich - @CrayZeigh