Elasticsearch's aggregations & esctl in action or how i built a cli tool to interact with elasticsearch

1. Aggregations
How do they work?

2. 2
Sum
POST /test/_doc {"foo": 1}
POST /test/_doc {"foo": 2}
POST /test/_doc {"foo": 3}
POST /test/_doc {"foo": 4}
POST /test/_doc {"foo": 5}
POST /test/_refresh
POST /test/_search {
"size": 0,
"aggs": {
"sum_foo": { "sum": { "field": "foo" } }
}
}
==== 15
As simple as possible
class SumAggregator implements Aggregator {
final DocValues vals;
double sum;
SumAggregator(Context ctx) {
vals = ctx.docValues("foo");
}
@Override
void collect(int docId) {
if (false == vals.advanceExact(docId)) continue;
sum += vals.nextValue();
}
@Override
SumResult result() {
return new SumResult(sum);
}
}

3. 3
Sum
POST /test/_doc {"foo": 1}
POST /test/_doc {"foo": 2}
POST /test/_doc {"foo": 3}
POST /test/_doc {"foo": 4}
POST /test/_doc {"foo": 5}
POST /test/_refresh
POST /test/_search {
"size": 0,
"aggs": {
"sum_foo": { "sum": { "field": "foo" } }
}
}
==== 15
As simple as possible
class SumAggregator implements Aggregator {
final DocValues vals;
double sum;
SumAggregator(Context ctx) {
vals = = ctx.docValues("foo");
}
@Override
void collect(int docId) {
if (false == vals.advanceExact(docId) continue;
sum += vals.nextValue();
}
@Override
SumResult result() {
return new SumResult(sum);
}
}

4. 4
Sum
POST /test/_doc {"foo": 1}
POST /test/_refresh
POST /test/_doc {"foo": 2}
POST /test/_doc {"foo": 3}
POST /test/_refresh
POST /test/_doc {"foo": 4}
POST /test/_doc {"foo": 5}
POST /test/_refresh
POST /test/_search {
"size": 0,
"aggs": {
"sum_foo": { "sum": { "field": "foo" } }
}
}
==== 15
Less lying
class SumAggregator implements Aggregator {
double sum;
@Override
void leafCollector(LeafReaderContext ctx) {
DocValues vals = ctx.docValues("foo");
return docId -> {
if (false == vals.advanceExact(docId) continue;
sum += vals.nextValue();
};
}
@Override
SumResult result() {
return new SumResult(sum);
}
}

5. 5
Date Histogram
POST /test/_doc {"t":"2020-08-20","foo": 1}
POST /test/_doc {"t":"2020-08-20","foo": 2}
POST /test/_doc {"t":"2020-08-21","foo": 3}
POST /test/_doc {"t":"2020-08-21","foo": 4}
POST /test/_doc {"t":"2020-08-21","foo": 5}
POST /test/_search {
"size": 0, "aggs": {
"days": {
"date_histogram": {
"field": "t",
"calendar_interval": "days"
},
"aggs": {
"sum_foo": {
"sum": { "field": "foo" }
}
}
}
}
}
==== { "2020-08-20": 3, "2020-08-21": 12 }
Buckets!
class DateHistoAggregator implements Aggregator {
private final Aggregator child;
private final LongKeyedBucketOrds ords;
@Override
LeafCollector leafCollector(LeafReaderContext ctx) {
DocValues vals = ctx.docValues("t");
LeafCollector childLeaf = child.leafCollector(ctx);
return (ord, docId) -> {
if (false == vals.advanceExact(docId) continue;
long rounded = round(vals.nextValue());
long childOrd = ords.find(ord, rounded);
childLeaf.collect(childOrd, docId);
};
}
}

6. 6
Sum
POST /test/_doc {"t":"2020-08-20","foo": 1}
POST /test/_doc {"t":"2020-08-20","foo": 2}
POST /test/_doc {"t":"2020-08-21","foo": 3}
POST /test/_doc {"t":"2020-08-21","foo": 4}
POST /test/_doc {"t":"2020-08-21","foo": 5}
POST /test/_search {
"size": 0, "aggs": {
"days": {
"date_histogram": {
"field": "t",
"calendar_interval": "days"
},
"aggs": {
"sum_foo": {
"sum": { "field": "foo" }
}
}
}
}
}
==== { "2020-08-20": 3, "2020-08-21": 12 }
Minimal lying
class SumAggregator implements Aggregator {
double[] sum;
@Override
void leafCollector(LeafReaderContext ctx) {
DocValues vals = ctx.docValues(ctx);
return (ord, docId) -> {
if (false == vals.advanceExact(docId) continue;
if (sum.length >= ord) {
sum = grow(sum, ord + 1);
}
sum[ord] += vals.nextValue();
};
}
@Override
SumResult result(long ord) {
return new SumResult(sum[ord]);
}
}

7. 7
But

8. 8
Grow
double[] grow(double[] old, int target) {
int extra = min(3, target >> 3);
int newSize = target + extra;
return Arrays.copyOf(old, newSize);
}
Oversizing to asymptotically remove allocations
class SumAggregator implements Aggregator {
double[] sum;
@Override
void leafCollector(LeafReaderContext ctx) {
DocValues vals = ctx.docValues(ctx);
return (ord, docId) -> {
if (false == vals.advanceExact(docId) continue;
if (sum.length >= ord) {
sum = grow(sum, ord + 1);
}
sum[ord] += vals.nextValue();
};
}
@Override
SumResult result(long ord) {
return new SumResult(sum[ord]);
}
}

9. 9
DocValues
POST /test/_doc {"t":"2020-08-20","foo": 1}
POST /test/_doc {"t":"2020-08-20","foo": 2}
POST /test/_doc {"t":"2020-08-21","foo": 3}
POST /test/_doc {"t":"2020-08-21","foo": 4}
POST /test/_doc {"t":"2020-08-21","foo": 5}
POST /test/_search {
"size": 0, "aggs": {
"days": {
"date_histogram": {
"field": "t",
"calendar_interval": "days"
},
"aggs": {
"sum_foo": {
"sum": { "field": "foo" }
}
}
}
}
}
==== { "2020-08-20": 3, "2020-08-21": 12 }
Just one please
class SumAggregator implements Aggregator {
double[] sum;
@Override
void leafCollector(LeafReaderContext ctx) {
DocValues vals = ctx.docValues(ctx);
return (ord, docId) -> {
if (false == vals.advanceExact(docId) continue;
if (sum.length >= ord) {
sum = grow(sum, ord + 1);
}
sum[ord] += vals.nextValue();
};
}
@Override
SumResult result(long ord) {
return new SumResult(sum[ord]);
}
}

10. 10
Check out this hashtable
POST /test/_doc {"t":"2020-08-20","foo": 1}
POST /test/_doc {"t":"2020-08-20","foo": 2}
POST /test/_doc {"t":"2020-08-21","foo": 3}
POST /test/_doc {"t":"2020-08-21","foo": 4}
POST /test/_doc {"t":"2020-08-21","foo": 5}
POST /test/_search {
"size": 0, "aggs": {
"days": {
"date_histogram": {
"field": "t",
"calendar_interval": "days"
},
"aggs": {
"sum_foo": {
"sum": { "field": "foo" }
}
}
}
}
}
==== { "2020-08-20": 3, "2020-08-21": 12 }
Brains?!
class DateHistoAggregator implements Aggregator {
private final Aggregator child;
private final LongKeyedBucketOrds ords;
@Override
LeafCollector leafCollector(LeafReaderContext ctx) {
DocValues vals = ctx.docValues("t");
LeafCollector childLeaf = child.leafCollector(ctx);
return (ord, docId) -> {
if (false == vals.advanceExact(docId) continue;
long rounded = round(vals.nextValue());
long childOrd = ords.find(ord, rounded);
childLeaf.collect(childOrd, docId);
};
}
}

11. Thanks!

12. Thanks!

13. 13
Jeff Moore
08/25/20, Systems Engineer, Bandwidth
esctl in Action
or How I Built a CLI tool
to Interact with
Elasticsearch

14. 14
About this presentation
It was fairly difficult deciding between catering towards Devs or Ops. In the
end, I decided to side with the Ops side of the house. Based on audience
request, I will go into the code in depth - or that can be the topic of a later talk.
Feel free to stop me at any point* so we can discuss this utility - however,
suggestions are encouraged to be held until the end of the slide deck

15. 15
Contributors Welcome!
https://github.com/geoffmore/esctl

16. 16
Hopeful Takeaways
- An extensible CLI tool that may impact both
yourself and your Org
- Lessons learned from over a year of
becoming a stronger Golang developer

17. 17
Caveats
- This is not complete.
- Work has been focused on getting good boilerplate
- Not all endpoints have been implemented (templates, indices, etc)
- This is not intended to replace the SDKs

18. 18
Why esctl?
Devs have a lot of SDKs, but Ops doesn’t
appear to have anything

19. 1919
We needed an easier way to
interact with Elasticsearch as
operators of the cluster

20. 20
Intent of esctl
1. Standard endpoints are mocked, with basic config options exposed as
flags
2. Extra commands are added based on functions that could be necessary
(i.e admin, util)

21. 2121
Inspiration

22. 22

23. 23

24. 24
escli and esctl
A brief history of two cli tools

25. 25
escli vs esctl
Comparison of metrics
Metric escli esctl
Language Bash Golang
Function implementation Bash functions Subcommands
# of commands (Aug
2020)
117 44
New functions Easy to add Tedious
Extensibility Difficult Easy
Refactorability Difficult Not as difficult
Barrier to entry Easy Difficult

26. 26
Getting Started Demo
Overview of Demo
Environment
1 2 3 4
Overview of local
cluster init script
Overview of
Kubernetes
manifests
Demo of
commands and
features

27. 27
Homelab Architecture

28. 28

29. 29
On Command Structure
- Elasticsearch api structure is very different to that of Kubernetes
- Format in v0.1.4 is esctl <endpoint/resource> <function>
- Format in Kubectl is kubectl <function> <endpoint/resource>

30. 30
Command Structure Comparison pt. 1
kubectl vs. esctl - Similarities
CLI Tool Command Function Resource
kubectl kubectl get nodes Get Nodes
esctl esctl cat nodes Get Nodes

31. 31
Command Structure Comparison pt. 2
kubectl vs. esctl - Differences
CLI Tool Command Function Resource
kubectl kubectl get ns Get Namespaces
esctl esctl index get Get Indices
(Yes, I am comparing namespaces to
shards. Feel free to discuss offline)

32. 32
Command Structure Comparison pt. 3
kubectl vs. esctl - the Justification
The endpoint behind commands like
esctl cat nodes is /_cat/nodes. By
mocking functions after their API
syntax, there is no mental translation
tax needed to move to/from esctl and
the REST API

33. 33
Proudly Built on go-elasticsearch SDK
Elasticsearch SDK, mostly written by Karel Minarik from Elastic
Defines api endpoint options as struct fields
Allows client initialization
Located at https://github.com/elastic/go-elasticsearch

34. 34

35. 35
esctl Extensibility Examples - Showing Inactive Watchers
Problem: Watchers not currently active (read: disabled) need to be listed
Solution:
- Get all watchers (SearchRequest on .watches index)
- Filter to desired activity state
- Return output (esutil helper function)

36. 36
esctl Extensibility Examples - Getting Search Metrics
Problem: User queries need to be validated before a DeleteByQuery is run
Solution:
• Take a query
• Merge a known aggregation definition to that query
• Execute that query (SearchRequest)
• Return output (esutil helper function)

37. 37
esctl Extensibility Examples - Removing Empty Indices
Problem: Empty indices created by ILM policies cause disk utilization
inbalances
Solution:
- Get a list of all indices (IndicesGetRequest)
- Filter to indices containing 0 documents
- Delete those indices (IndicesDeleteRequest)

38. 3838
Still Not Sold?
esctl api <HTTP verb> endpoint

39. 39
Lessons Learned
- Writing a CLI tool isn’t easy
- Design should come before writing code to avoid a large number of
refactors
- Reading documentation thoroughly before using imported code is
extremely valuable
- This CLI tool continues to be a great learning opportunity for me

40. 40
Overall, building this CLI tool was
a painful experience; but, if it was
painless, I would be a Java dev
instead of a Golang dev
Me, just now

41. 41
Future Improvements
- Add documentation
- Discuss command structure
- Need a logo
- Codegen?

42. 42
What I Need from YOU!
- Use it
- File bugs and RFEs

43. 4343
Questions?

44. 4444
https://www.linkedin.com/in/jeff-moore-k8s
https://github.com/geoffmore
Reach Out!

45. 4545
Thanks!