Presented at DevOpsDays Austin. How does Elastic enable customization and contribution from their Beats project which inherently is deployed in unique and varied environments. Provide Documentation, automate tedious tasks, ship easy to use testing environments, and make a clear path for sharing

1. Batteries
Included1 — Aaron Aldrich - @CrayZeigh

2. Aaron Aldrich
Developer Advocate
DevOpsCT
@CrayZeigh
aaron.aldrich@elastic.co
2 — Aaron Aldrich - @CrayZeigh

3. COMMUNITY
3 — Aaron Aldrich - @CrayZeigh

4. "Pull requests welcome"is Open Source
speak for"Go away, kid, you're
bothering me."
4 — Aaron Aldrich - @CrayZeigh

5. Welcome
BY
ENABLING SUCCESS
5 — Aaron Aldrich - @CrayZeigh

6. Context6 — Aaron Aldrich - @CrayZeigh

7. THINGS WE'RE NOT TALKING ABOUT
▸ root vegetables
▸ things by Dre
7 — Aaron Aldrich - @CrayZeigh

8. Killer Tofu
8 — Aaron Aldrich - @CrayZeigh

9. 9 — Aaron Aldrich - @CrayZeigh

10. 10 — Aaron Aldrich - @CrayZeigh

11. EXISTING BEATS INTENTED USE
filebeat logs, csv, text
metricbeat metrics
packetbeat transaction logs & flow metrics
winlogbeat Windows events
auditbeat system activity, changes
heartbeat network stats
11 — Aaron Aldrich - @CrayZeigh

12. 12 — Aaron Aldrich - @CrayZeigh

13. DOCUMENTATION INCLUDED
fields.yml
13 — Aaron Aldrich - @CrayZeigh

14. WHAT IS IT?
For all events generated by beats we provide:
▸ Elasticsearch index template
▸ Kibana index pattern
▸ Documentation!
▸ Common definition and documentation in fields.yml
14 — Aaron Aldrich - @CrayZeigh

15. SYNTAX
field ::= #'type' must be group
name: <field name> fields list ::=
type: <type> - <field>
[format: <format>] - <field>
description: <docstring> ...
[fields: <fields list>]
type ::= group, long, keyword
format ::= bytes, percent, ...
15 — Aaron Aldrich - @CrayZeigh

16. EXAMPLE
- name: memory
type: group
description: >
'memory' contains local memory stats.
fields:
- name: total
type: long
format: bytes
description: >
Total memory
16 — Aaron Aldrich - @CrayZeigh

17. Automate!
17 — Aaron Aldrich - @CrayZeigh

18. BY DESIGN
18 — Aaron Aldrich - @CrayZeigh

19. FILEBEAT MODULES
filebeat/modules.d/
apache2.yml kafka.yml nginx.yml redis.yml
auditd.yml logstash.yml osquery.yml system.yml
icinga.yml mysql.yml postgresql.yml traefik.yml
19 — Aaron Aldrich - @CrayZeigh

20. METRICBEAT MODULES
metricbeat/modules.d/
aerospike.yml golang.yml logstash.yml rabbitmq.yml
apache.yml graphite.yml memcached.yml redis.yml
ceph.yml haproxy.yml mongodb.yml system.yml
couchbase.yml http.yml mysql.yml uwsgi.yml
docker.yml jolokia.yml nginx.yml vsphere.yml
dropwizard.yml kafka.yml php_fpm.yml windows.yml
elasticsearch.yml kibana.yml postgresql.yml zookeeper.yml
etcd.yml kubernetes.yml prometheus.yml
20 — Aaron Aldrich - @CrayZeigh

21. > filebeat setup
Flags:
--dashboards Setup dashboards only
--machine-learning Setup machine learning job configurations only
--template Setup index template only
--modules string List of enabled modules (comma separated)
21 — Aaron Aldrich - @CrayZeigh

22. FOR CONTRIBUTORS
python $GOPATH/src/github.com/elastic/beats/script/generate.py
22 — Aaron Aldrich - @CrayZeigh

23. > make setup
mkdir -p vendor/github.com/elastic/
cp -R /Users/aaronaldrich/go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git
/Library/Developer/CommandLineTools/usr/bin/make update
New python executable in /.../github.com/crayzeigh/examplebeat/build/python-env/bin/python
Installing setuptools, pip, wheel...done.
Updating generated files for examplebeat
-- The index pattern was created under /.../examplebeat/_meta/kibana/5/index-pattern/examplebeat.json
-- The index pattern was created under /.../examplebeat/_meta/kibana/6/index-pattern/examplebeat.json
Setup, Binary builds and index-pattern updates handled through
Makefiles
23 — Aaron Aldrich - @CrayZeigh

24. MAIN-IMPORT TRICK
package main
import (
"os"
"github.com/elastic/beats/libbeat/beat"
"github.com/elastic/beats/metricbeat/beater"
// load my plugins
_ "my/module/package/name"
)
func main() {
if err := beat.Run("mymetricbeat", "", beater.New); err != nil {
os.Exit(1)
}
}
24 — Aaron Aldrich - @CrayZeigh

25. SHIP THE TEST ENVIRONMENT
-OR-
DOCKER ALL THE THINGS
25 — Aaron Aldrich - @CrayZeigh

26. Environment
> ls vendor/github.com/elastic/beats/testing/environments/
2x.yml Dockerfile README.md
docker local.yml 5.0.0-cgroups.yml
Makefile args.yml latest.yml
snapshot.yml
26 — Aaron Aldrich - @CrayZeigh

27. Environment
/demobeat/vendor/github.com/elastic/beats/testing/environments> make start
# This is run every time to make sure the environment is up-to-date
docker-compose -f snapshot.yml -f local.yml build
...
Building kibana...
Building elasticsearch...
Building logstash...
docker-compose -f snapshot.yml -f local.yml run beat bash
Creating environments_elasticsearch_1 ... done
Creating environments_kibana_1 ... done
Creating environments_logstash_1 ... done
root@3a834aedbb86:/#
27 — Aaron Aldrich - @CrayZeigh

28. Data
28 — Aaron Aldrich - @CrayZeigh

29. ENABLE SHARING
29 — Aaron Aldrich - @CrayZeigh

30. amazonbeatbeat, apachebeatbeat, apexbeatbeat, burrowbeat, cassandrabeat, cloudflarebeat,
cloudfrontbeat, cloudtrailbeat, cloudwatchmetricbeat, cloudwatchlogsbeat, collectbeat,
connbeat, consulbeat, dockbeat, elasticbeat, stcdbeat, execbeat, factbeat, flowbeat, gabeat,
githubbeat, gpfsbeat, hsbeat, httpbeat, hwsensorsbeat, icingabeat, iobeat, jmxproxybeat,
journalbeat, kafkabeat, krakenbeat, lmsensorsbeat, logstashbeat, mcqbeat, mongobeat,
mqttbeat, mysqlbeat, nagioscheckbeat, nginxbeat, nginxupstreambeat, nvidagpubeat,
openconfigbeat, packagebeat, phpfpmbeat, pingbeat, prombeat, prometheusbeat, protologbeat,
redditbeat, redisbeat, retsbeat, rsbeat, saltbeat, springbeat, tracebeat, twitterbeat, udpbeat,
udplogbeat, unifiedbeat, uwsgibeat, varnishlogbeat, varnishstatbeat, wmibeat
HTTPS://WWW.ELASTIC.CO/GUIDE/EN/BEATS/LIBBEAT/CURRENT/COMMUNITY-BEATS.HTML
30 — Aaron Aldrich - @CrayZeigh

31. BATTERIES
INCLUDED
31 — Aaron Aldrich - @CrayZeigh