Message Intelligence Service collects connection and message metadata from thousands of production email clusters for near real-time monitoring, reporting, and searching. The service processes billions of messages a day and manages hundreds of terabytes of indices with a standard 30 day retention. MIS consists of conventional data pipeline components from the Kafka and Elastic ecosystems. What might be unusual is the choice of using Kubernetes to deploy these components. Kubernetes is the current choice for deploying stateless services in the cloud. Unfortunately, many of the simplifying assumptions that makes Kubernetes outstanding for deploying stateless services cause complications for stateful services such as Kafka and Elasticsearch. This presentation discusses some of these challenges and how we dealt with them.

1. Some Challenges Deploying a
Kafka / Elasticsearch Pipeline
with Kubernetes
08/28/2019
© 2019 Proofpoint. All rights reserved

2. Very Attacked People
2© 2019 Proofpoint. All rights reserved

3. Message Intelligence Service
Near real time connection and metadata omnibus
3© 2019 Proofpoint. All rights reserved
2B2TB
5000
60 r/s

4. Prior and Current Generation
4© 2019 Proofpoint. All rights reserved
© 2019 Proofpoint. All rights reserved

5. Why Kubernetes?
•Available for all public clouds and bare metal
•Widely known
•Highly Scalable
•Likely “winner” for long time
5© 2019 Proofpoint. All rights reserved

6. Kubernetes Advantages
•System resources are shared across services
•Automatic service scaling as needed
– Nodes and pods are created and destroyed ad-hoc
•Pods scheduled on any node with capacity
6© 2019 Proofpoint. All rights reserved

7. Kubernetes Stateful Service Challenges
•Services are scaled as needed
– Nodes and pods are created and destroyed ad-hoc
•Pods scheduled on any node with capacity
•Service pods do not receive any disk resources.
7© 2019 Proofpoint. All rights reserved

8. Kubernetes Challenge:
Upgrades are upsetting
•KOPS prevents using local storage
Solutions:
– EBS volumes (which have own problems)
– Upgrade using in-place container OS
•Coreos/Tectonic
•Flatcar/Lokomotiv
•Juju
8© 2019 Proofpoint. All rights reserved

9. Kubernetes Challenge:
Disk Resources
•Stateful Set (STS)
– Guaranteed ordering and uniqueness of Pods
– Persistent pod identity
– Storage available
•PVC / SC / PV
– Pod specification requests a Persistent Volume Claim
– PVC requests a Storage Class
– SC is producer of Persistent Volume
– MIS uses Elastic Block Storage PVs
•EBS Volumes
– Access Limited to Availability Zone
9© 2019 Proofpoint. All rights reserved

10. Kubernetes Challenge:
Unpredictable Nodes
•Node taints prevent stateless pods being scheduled on
stateful nodes
•Each node class is part of an Instance Group
•Pod tolerations allow stateful sets to be scheduled on
stateful nodes
•Pod anti-affinities spread the STS across zones and nodes
•Single zone cluster would probably still need anti-affinities
•Non-shared cluster still requires taints/tolerations
10© 2019 Proofpoint. All rights reserved

11. Instance Groups
Instance
Group
# EC2 Type Node Taint Node Label EBS
(GB)
kafka 10 m5.xlarge dedicated=mis service: kafka 500
es-data 51 m5.4xlarge dedicated=mis service: es-data 4,000
es-master 3 m5.large dedicated=mis service: es-
master
10
es-client 3 m5.2xlarge dedicated=mis service: es-client
apps 10 m5.2xlarge dedicated=mis service: apps
pod-
logging
3 m5.2xlarge dedicated=mis service: pod-
logging
11© 2019 Proofpoint. All rights reserved

12. Applications
12© 2019 Proofpoint. All rights reserved
Application Instance Group
Kafka kafka-nodes
Zookeeper kafka-nodes
Secor apps-nodes
Elasticsearch Master es-master-nodes
Elasticsearch Data es-data-nodes
Elasticsearch Client es-client-nodes
Kafka Connect apps-nodes
Search Service apps-nodes
Logging Service pod-logging-nodes
Kibana apps-nodes
Admin apps-nodes

13. Kubernetes Challenge:
EBS Read-Only Volume
•NVMe driver default timeout is 30 seconds
•driver fails the I/O and filesystem remounted as read-only
•Solutions:
–Update driver timeout (insufficient)
–Update driver (required new AMI)
13© 2019 Proofpoint. All rights reserved

14. Kubernetes Challenge:
CNI plugin does not scale
•Weave is unstable past 160 nodes
Solutions:
– Limit nodes
– Use different Container Network Interface
14© 2019 Proofpoint. All rights reserved

15. Kafka Challenge:
Parsing json records is slow
•API Gateway introduced unacceptable latency
•Secor routing extremely slow
Solutions:
– Incoming gateway uses header to determine message topic
– Secor modified to use Kafka header to determine partition
15© 2019 Proofpoint. All rights reserved

16. Elasticsearch Challenge:
Document Ingestion is slow
•Carefully tune index templates
– Map all fields
– Index only fields that are searched
– Use keyword type where possible
•Tune shard size
– Aim for 50G shards
– Add nodes as needed
16© 2019 Proofpoint. All rights reserved

17. Elasticsearch Challenge:
Document deletion is slow
•Drop entire index instead of individual documents
– Create index templates with schema
– Cron job to create daily indices
– Cron job to delete expired indices
17© 2019 Proofpoint. All rights reserved

18. Elasticsearch Challenge:
Document search is slow
•Only index fields that are searched
•Use keyword type where possible
•Limit search to appropriate daily indices
•Use shard routing and index.routing_partition_size to
limit queried shards
•Use index.routing.allocation.total_shards_per_node to
evenly distributed shards across data nodes
18© 2019 Proofpoint. All rights reserved

19. Elasticsearch Challenge:
Bursts and Resurrected Indices
•Email clusters may reconnect after extensive time offline
– Burst of incoming data
– Data may already be expired
Solutions:
– Network headroom handles bursts
– Hourly cron job deletes expired indices
19© 2019 Proofpoint. All rights reserved

20. Future Directions
•I3 EC2 instances
– Local SSD storage
– Decrease costs and increase ingestion speed
•Multiple retention periods
– 30, 60, 90, 365-day retentions
•Operators
– Graceful scaling
– In place software upgrade
•Additional meta-data
20© 2019 Proofpoint. All rights reserved

21. Questions? / Contacts
• chonton@proofpoint.com
• ahong@proofpoint.com
21© 2019 Proofpoint. All rights reserved

22. © 2019 Proofpoint. All rights reserved
22