Bio: Jeff Moore is a Systems Engineer at Bandwidth with a focus on administering and extending Kubernetes, AWS, and the Elastic Stack to support internal customer use-cases. Originally a die-hard Kubernetes fan, his passions have now also extended to the Elastic Stack due to insights of the amazing things that can be done with well-architected data. He has lived in Raleigh for most of his life and currently has a love/hate relationship with the factory automation game Factorio. Jeff is a Certified Kubernetes Administrator with plans to complete the Elastic Certified Engineer exam. Abstract: In this talk, Jeff will talk about how he designed a command-line tool built on the go-elasticsearch project - making interactions with the elasticsearch APIs much easier. He will also go into the inspirations of the project, lessons learned, and future work.

1. 1
Jeff Moore
08/25/20, Systems Engineer, Bandwidth
esctl in Action
or How I Built a CLI tool
to Interact with
Elasticsearch

2. 2
About this presentation
It was fairly difficult deciding between catering towards Devs or Ops. In the
end, I decided to side with the Ops side of the house. Based on audience
request, I will go into the code in depth - or that can be the topic of a later talk.
Feel free to stop me at any point* so we can discuss this utility - however,
suggestions are encouraged to be held until the end of the slide deck

3. 3
Contributors Welcome!
https://github.com/geoffmore/esctl

4. 4
Hopeful Takeaways
- An extensible CLI tool that may impact both
yourself and your Org
- Lessons learned from over a year of
becoming a stronger Golang developer

5. 5
Caveats
- This is not complete.
- Work has been focused on getting good boilerplate
- Not all endpoints have been implemented (templates, indices, etc)
- This is not intended to replace the SDKs

6. 6
Why esctl?
Devs have a lot of SDKs, but Ops doesn’t
appear to have anything

7. 77
We needed an easier way to
interact with Elasticsearch as
operators of the cluster

8. 8
Intent of esctl
1. Standard endpoints are mocked, with basic config options exposed as
flags
2. Extra commands are added based on functions that could be necessary
(i.e admin, util)

9. 99
Inspiration

10. 10

11. 11

12. 12
escli and esctl
A brief history of two cli tools

13. 13
escli vs esctl
Comparison of metrics
Metric escli esctl
Language Bash Golang
Function implementation Bash functions Subcommands
# of commands (Aug
2020)
117 44
New functions Easy to add Tedious
Extensibility Difficult Easy
Refactorability Difficult Not as difficult
Barrier to entry Easy Difficult

14. 14
Getting Started Demo
Overview of Demo
Environment
1 2 3 4
Overview of local
cluster init script
Overview of
Kubernetes
manifests
Demo of
commands and
features

15. 15
Homelab Architecture

16. 16

17. 17
On Command Structure
- Elasticsearch api structure is very different to that of Kubernetes
- Format in v0.1.4 is esctl <endpoint/resource> <function>
- Format in Kubectl is kubectl <function> <endpoint/resource>

18. 18
Command Structure Comparison pt. 1
kubectl vs. esctl - Similarities
CLI Tool Command Function Resource
kubectl kubectl get nodes Get Nodes
esctl esctl cat nodes Get Nodes

19. 19
Command Structure Comparison pt. 2
kubectl vs. esctl - Differences
CLI Tool Command Function Resource
kubectl kubectl get ns Get Namespaces
esctl esctl index get Get Indices
(Yes, I am comparing namespaces to
shards. Feel free to discuss offline)

20. 20
Command Structure Comparison pt. 3
kubectl vs. esctl - the Justification
The endpoint behind commands like
esctl cat nodes is /_cat/nodes. By
mocking functions after their API
syntax, there is no mental translation
tax needed to move to/from esctl and
the REST API

21. 21
Proudly Built on go-elasticsearch SDK
Elasticsearch SDK, mostly written by Karel Minarik from Elastic
Defines api endpoint options as struct fields
Allows client initialization
Located at https://github.com/elastic/go-elasticsearch

22. 22

23. 23
esctl Extensibility Examples - Showing Inactive Watchers
Problem: Watchers not currently active (read: disabled) need to be listed
Solution:
- Get all watchers (SearchRequest on .watches index)
- Filter to desired activity state
- Return output (esutil helper function)

24. 24
esctl Extensibility Examples - Getting Search Metrics
Problem: User queries need to be validated before a DeleteByQuery is run
Solution:
• Take a query
• Merge a known aggregation definition to that query
• Execute that query (SearchRequest)
• Return output (esutil helper function)

25. 25
esctl Extensibility Examples - Removing Empty Indices
Problem: Empty indices created by ILM policies cause disk utilization
inbalances
Solution:
- Get a list of all indices (IndicesGetRequest)
- Filter to indices containing 0 documents
- Delete those indices (IndicesDeleteRequest)

26. 2626
Still Not Sold?
esctl api <HTTP verb> endpoint

27. 27
Lessons Learned
- Writing a CLI tool isn’t easy
- Design should come before writing code to avoid a large number of
refactors
- Reading documentation thoroughly before using imported code is
extremely valuable
- This CLI tool continues to be a great learning opportunity for me

28. 28
Overall, building this CLI tool was
a painful experience; but, if it was
painless, I would be a Java dev
instead of a Golang dev
Me, just now

29. 29
Future Improvements
- Add documentation
- Discuss command structure
- Need a logo
- Codegen?

30. 30
What I Need from YOU!
- Use it
- File bugs and RFEs

31. 3131
Questions?

32. 3232
https://www.linkedin.com/in/jeff-moore-k8s
https://github.com/geoffmore
Reach Out!

33. 3333
Thanks!