Philipp lives to demo interesting technology. Having worked as a web, infrastructure, and database engineer for over ten years, Philipp is now working as a developer advocate at Elastic — the company behind the open-source Elastic Stack consisting of Elasticsearch, Kibana, Beats, and Logstash. Based in Vienna, Austria, he is constantly traveling Europe and beyond to speak and discuss open-source software, search, databases, infrastructure, and security. Speech overview: “Containers are the new ZIP format to distribute software” is a fitting description of today’s development world. However, it is not always that easy and this talk highlights the development of Elastic’s container strategy over time: Docker images: A new distribution model. Docker Compose: Local demos and a little more. Helm Chart: Going from demo to production. Kubernetes Operator: Full control with upgrades, scaling,… Besides the strategy, we are also discussing specific technical details and hurdles that appeared during the development. Or why the future will be a combination of Helm Chart and Operator (for now).

1. From Containers to
Kubernetes Operators
Philipp Krenn @xeraa
@xeraa

2. Developer
@xeraa

3. @xeraa

4. Who uses
containers?
@xeraa

5. Who uses
containers
in production?
@xeraa

6. Who uses
stateful
containers?
@xeraa

7. Who uses
containers
from Elastic?
@xeraa

8. Agenda
Docker images
Docker Compose
Helm Chart
Kubernetes Operator
@xeraa

9. @xeraa

11. Docker: the world's
most heavily funded
college project
— Internal quote from Slack
@xeraa

12. !
Containers are the new ZIP
format to distribute software
@xeraa

13. One of many…
RPM, DEB, TAR.GZ, MSI
Ansible, Chef, Puppet
@xeraa

14. …but not without
issues
@xeraa

15. Fallacy
root and chmod 777
@xeraa

16. @xeraa

17. The container runs
Elasticsearch as user
elasticsearch using uid:gid
1000:0.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
@xeraa

21. Those who do not
understand Unix are
condemned to
reinvent it,
poorly.
— Henry Spencer
@xeraa

22. Fallacy
:latest
@xeraa

24. :7 and :7.7?
@xeraa

25. Fallacy
Runtime mutation
@xeraa

26. Feature Request
if [ -f /custom/user_init.sh ]
then
. /custom/user_init.sh
fi
Or customization through environment variables
@xeraa

27. Dockerfile
ARG ELASTIC_VERSION
FROM docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
RUN bin/elasticsearch-plugin install analysis-phonetic --batch
ARG ELASTIC_VERSION
RUN bin/elasticsearch-plugin install
https://github.com/spinscale/elasticsearch-ingest-langdetect/releases/download/
${ELASTIC_VERSION}.1/ingest-langdetect-${ELASTIC_VERSION}.1.zip --batch
@xeraa

28. Generate Keystore
$ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node"
-v /Users/philipp/Desktop/demo/config/:/usr/share/elasticsearch/config/
-it docker.elastic.co/elasticsearch/elasticsearch:7.6.1 /bin/bash
[root@1006ed50b646 elasticsearch]# ./bin/elasticsearch-keystore create
Created elasticsearch keystore in /usr/share/elasticsearch/config
[root@1006ed50b646 elasticsearch]# ./bin/elasticsearch-keystore add test
Enter value for test:
[root@1006ed50b646 elasticsearch]# exit
exit
$ cat config/elasticsearch.keystore
??lelasticsearch.keystore?@g?o!?$?K?Lf?w?VAEŠԨm?[?a6?B??? y?,!В}??Ħ?ǣ?AU=?C?:?o?
?W?O8?}U?;p?ӷ???cQ????7?JY? 2A?:???ZUY??2V?9?ϧ??(??0?q
@xeraa

29. Mount Keystore
Docker Compose
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
secrets:
- source: elasticsearch.keystore
target: /usr/share/elasticsearch/config/elasticsearch.keystore
@xeraa

30. Fallacy
Base image
@xeraa

31. Common base image since 5.4+
CentOS 7
@xeraa

32. Upside
Similar setup
Shared layers
@xeraa

33. Upside
JVM large anyway
glibc tested
@xeraa

34. Downside
Size
@xeraa

35. Does it matter?
Stateful vs
Stateless
@xeraa

36. Windows?!
@xeraa

37. @xeraa

38. Who uses
Docker Compose?
@xeraa

39. More than
development /
demo?
@xeraa

40. ---
version: '2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:$ELASTIC_VERSION
environment:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.type=single-node
mem_limit: 1g
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- 9200:9200
kibana:
image: docker.elastic.co/kibana/kibana:$ELASTIC_VERSION
links:
- elasticsearch
ports:
- 5601:5601
volumes:
esdata:
driver: local

41. Deprecated
https://github.com/elastic/stack-docker
@xeraa

42. @xeraa

43. Who uses
Kubernetes?
@xeraa

44. Kubernetes is the
answer. What was
the question?
— https://twitter.com/charlesfitz/status/
1068203930683752448
@xeraa

45. Who uses
Swarm?
@xeraa

46. Who uses
Nomad?
@xeraa

47. Who uses
Mesos?
@xeraa

48. Static configuration to
deploy resources to K8s
@xeraa

50. @xeraa

51. …lots of it
@xeraa

52. Fun with YAML
http://www.yamllint.com
ports:
- 80:80
- 20:20
@xeraa

53. Fun with YAML
https://docs.docker.com/compose/compose-file/#short-syntax-1
ports:
- "80:80"
- 73200
@xeraa

54. Kubernetes 1.8 allows dots in
env vars
https://github.com/kubernetes/kubernetes/issues/2707
@xeraa

55. @xeraa

56. Advanced package management
with support for templating
and more complex resources
@xeraa

57. Building on existing
Kubernetes primitives like
StatefulSet, Service,
Deployment,…
@xeraa

58. Elastic Helm
Charts
Elasticsearch, Kibana,
Filebeat, Metricbeat,
APM Server, Logstash
https://github.com/elastic/helm-charts
@xeraa

59. StatefulSet (STS)
By default rolling upgrade
Waiting for cluster health being green after
each instance is updated
@xeraa

60. Tested on GKE
Default storage pd-ssd (network attached)
Kubernetes >=1.10 supports Local
PersistentVolumes for increased performance
@xeraa

61. Un-Opinionated
Expose environment variables & mount secrets
Multiple upgrade strategies
@xeraa

62. Minikube Example
https://github.com/elastic/helm-charts/tree/master/elasticsearch/examples/
minikube
helm repo add elastic https://helm.elastic.co
helm install --name elasticsearch elastic/elasticsearch [--set imageTag=7.7.1]
minikube addons enable default-storageclass
minikube addons enable storage-provisioner
cd examples/minikube
make
@xeraa

63. ---
# Permit co-located instances for solitary minikube virtual machines
antiAffinity: "soft"
# Shrink default JVM heap
esJavaOpts: "-Xmx128m -Xms128m"
# Allocate smaller chunks of memory per pod
resources:
requests:
cpu: "100m"
memory: "512M"
limits:
cpu: "1000m"
memory: "512M"
# Request smaller persistent volumes
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "standard"
resources:
requests:
storage: 100M

64. @xeraa

65. Expands K8s interfaces and
allows to customize
management of applications
@xeraa

66. Custom Resource Definition
(CRD)
Think: Elasticsearch, Kibana, APM
Contrast: Built-in resources like Pods,
Services, Secrets, StatefulSets,…
@xeraa

67. Custom Resource (CR)
CRD == type definition (class)
CR == instance (object)
@xeraa

68. Custom Controller
Brings CRDs to "life" (reconciliation loop)
Upgrades, secrets, certificate management,…
@xeraa

69. @xeraa

70. @xeraa

71. Elastic Operator
Elasticsearch, Kibana,
APM Server
https://github.com/elastic/cloud-on-k8s
@xeraa

72. Golang 1.13
Kubebuilder 2
SDK for building Kubernetes APIs using CRDs
Kustomize
Generate patched CRDs for specific flavors
@xeraa

73. @xeraa

74. Opinionated
Encode best practices & operational knowledge
Built-in certificate management, security,…
@xeraa

75. Example Opinions
Scale down: Drain nodes first
Upgrade: Disable shard allocation
@xeraa

76. You Can Still Shoot Yourself
in the Foot
Configure 0 replicas and do an upgrade for
example
@xeraa

77. Running on Minikube
minikube config set memory 16384
minikube config set cpus 4
minikube start
@xeraa

78. Running on Minikube
# Get the current version
kubectl apply -f https://download.elastic.co/downloads/eck/1.1.2/all-in-one.yaml
# Monitor logs
kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
# Deploy
kubectl apply -f apm_es_kibana.yaml
@xeraa

79. ---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch-sample
spec:
version: 7.7.1
nodes:
- nodeCount: 1
podTemplate:
spec:
containers:
- name: elasticsearch
resources:
limits:
memory: 2Gi
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi

80. ---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-server-sample
spec:
version: 7.7.1
nodeCount: 1
elasticsearchRef:
name: "elasticsearch-sample"
@xeraa

81. ---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-sample
spec:
version: 7.7.1
nodeCount: 1
elasticsearchRef:
name: "elasticsearch-sample"
@xeraa

82. Running on Minikube
# Check status
kubectl get elasticsearch,kibana,apmserver
# Expose Kibana
kubectl port-forward service/kibana-sample-kb-http 5601
# Get the credentials
echo `kubectl get secret elasticsearch-sample-es-elastic-user
-o=jsonpath='{.data.elastic}' | base64 --decode`
@xeraa

83. Changes
Instance size / number, version,…
kubectl apply -f apm_es_kibana.yaml
@xeraa

84. Support
GKE (Google Cloud)
EKS (AWS)
AKS (Azure)
OpenShift (Redhat)
@xeraa

85. StatefulSets
Rolling Upgrades with Volume reuse
"Standard" way to run stateful workloads —
stable network ID, stable data volume that is
re-attachable during rolling upgrades
@xeraa

86. Storage (1)
Persistent Volumes (default)
Local PV, Cloud vendor specific network
attached
@xeraa

87. Storage (2)
EmptyDir / HostPath
Not durable, no enforceable quota, may lead to
eviction
@xeraa

88. Storage (Future)
Container Storage Interface
(CSI)
@xeraa

89. Deployment
CRDs require cluster admin level permissions to
install
Privileged Containers — Elasticsearch host
kernel settings like vm.max_map_count
@xeraa

90. Global Namespace
@xeraa

91. Single Namespace
@xeraa

92. Other Operators
MongoDB, Kafka, Redis, CockroachDB,…
Operator "Marketplaces": https://operatorhub.io
@xeraa

94. Conclusion
@xeraa

95. "Containers are disrupting
the industry!"
@xeraa

96. "Can I run Elasticsearch on
Docker or Kubernetes?"
@xeraa

97. "Should I run Elasticsearch
on Docker or Kubernetes?"
@xeraa

98. Kubernetes Paradox
"You don't have an Operator? That's a blocker!"
@xeraa

99. Kubernetes Paradox
"So you run the majority of your apps on
Kubernetes?"
"Actually, we are just starting with Kubernetes
and have 2% of our workloads on it."
@xeraa

100. Helm Charts vs Operator
@xeraa

101. Questions &
Discussion
Philipp Krenn @xeraa
@xeraa