What do you do when you've two different technologies on the upstream and the downstream that are both rapidly being adopted industrywide? How do you bridge them scalably and robustly? At Wework, the upstream data was being brokered by Kafka and the downstream consumers were highly scalable gRPC services. While Kafka was capable of efficiently channeling incoming events in near real-time from a variety of sensors that were used in select Wework spaces, the downstream gRPC services that were user-facing were exceptionally good at serving requests in a concurrent and robust manner. This was a formidable combination, if only there was a way to effectively bridge these two in an optimized way. Luckily, sink Connectors came to the rescue. However, there weren't any for gRPC sinks! So we wrote one. In this talk, we will briefly focus on the advantages of using Connectors, creating new Connectors, and specifically spend time on gRPC sink Connector and its impact on Wework's data pipeline.

1. Creating Connector to bridge the worlds of
Kafka and gRPC at Wework
-- Anoop Dixith

2. Agenda
❏ Role of Kafka at Wework
❏ - Why Kafka? Why gRPC?
❏ Life without Connect
❏ Why Connectors?
❏ gRPC sink Connector at WeWork
❏ - Configuration, implementation, monitoring

4. Architecture (publisher side)

5. Architecture (continued)
❏ Importance of the architecture
❏ Kafka-gRPC combination
❏ Java/Scala-Go
❏ Why gRPC?
❏ How the combination worked perfectly?

6. Architecture (continued)
Machine readable API contract, so
platform neutral
Throughput
Protobuf, so polyglot and faster,
smaller and simpler, safer payload
Distributed, HA
HTTP/2 - multiplexing, binary, safer,
future
Scalability
Streaming vs request/response Complementary components -
KStream, KTable, KSQL, Connectors
Community Community
gRPC Kafka

7. Life Without Connect
In the very beginning, there was no Connect
Kafka sources were connected to a variety of sinks at Wework- gRPC,
Elasticsearch etc. And hey, it worked well.
Yes, but what about
❏ Scalability?
❏ Security?
❏ Configurability?
❏ Error handling?
❏ Extendability?
Do not reinvent the wheel!

8. Architecture without Connect

9. Then there was Connect...
Very simply, Kafka Connect is a framework to stream data into and out of Kafka
Properties:
❏ Broad copying, scalability, streaming and batch apps, parallelism.
❏ Does one and one thing very well - copying data
❏ Extensible through Connectors
Models:
❏ Connector
❏ Worker
❏ Data

10. Connect components
❏ Connectors – abstraction that handles data streaming by managing “tasks”
❏ Tasks – the implementation of how data is copied to or from Kafka
❏ Workers – the running processes that execute connectors and tasks
❏ Converters – translates data between Connect and the source/sink
❏ Transforms – alters messages produced by or sent to a connector
❏ Dead Letter Queue – error handling

11. Types of Connectors
❏ Source Connector - Kinesis, Zendesk, Jira, Twitter, email
❏ Sink Connector - S3, MongoDB, HDFS, YouNameIt
❏ Connect Hub - https://www.confluent.io/hub/
❏ Also available are different Transform and Converter
❏ Availability in Confluent Cloud - fully managed
❏ Licenses, levels of verification

12. Connect flow

13. Writing your own Connectors. Yes, we can!
❏ Why?
❏ Oh, we don’t have that Connector!
❏ We have a Connector, but we need to customize it to our needs
❏ Complete control on how we want to move the data
❏ Give it back to the community
❏ How?
❏ Kafka Connect API to the rescue!
❏ Implement/extend your Connector, Task, Config interfaces/abstract classes

14. Custom Connectors

15. gRPC Sink Connector
❏ What is it intended to do?
❏ Why not directly sink to underlying databases?

16. Implementation
❏ SinkRecord
❏ SinkTask

17. GrpcService Interface

18. GrpcClient Interface
Crux of bulkSend() AKA how is gRPC Connector different from MySql Connector?
gRPC glossary
❏ stub: generated when protoc is run if a service declaration is in the proto file
❏ service class
❏ rpc name and args
❏ channel: provides a connection to a gRPC server on a specified host and port
❏ along with grpc server url and port

19. bulkSend()
bulkSend() forms the crux of the gRPC Sink Connector’s data copying
❏ Handles channel readiness (connectivity state)
❏ Manages security and all logic related to error-handling
❏ Controls the rate of data copying
❏ Potentially retry logic

20. Limitations of bulkSend()
Uses reflection to get stub classes and methods
What you can’t do with bulkSend() (yet)?

21. Configuration
Configs passed to GrpcSinkConnector object

22. Testing, Deployment, and Monitoring
❏ Testing Connector by mocking is hard and tricky as it involves two systems -
Kafka and the external source/sink
❏ Independent unit-testing is recommended for all Task and Connector classes
❏ End-to-end testing using gRPC servers created on the fly in Docker
containers of CircleCI testing plan
❏ Extremely difficult to test gRPC channel connectivity states and other error
scenarios

23. Testing, Deployment, and Monitoring (contd…)
❏ Packaging - for easy installing into into Kafka Connect installations
❏ By creating an Archive
❏ create a tarball or ZIP archive
❏ contains a single directory with unique name (name and version likely)
❏ all JAR files and other resource files needed by the connector are in tld
❏ doesn’t include Kafka Connect API or runtime libraries
❏ By creating an Uber JAR
❏ create an uber JAR that contains all JAR files and other resource files
❏ Installation -
❏ User simply unpacks the archive or places the uber JAR in a directory listed in Kafka
Connect’s plugin path

24. Monitoring Connectors
❏ Monitored via Connect’s extensive REST interface
❏ current status of a connector and its tasks
❏ worker ids to whom tasks are assigned
❏ pause/resume APIs
❏ active connectors, connector tasks, restart a connector, restart a task, update config, delete
connector
❏ Logging
❏ Connect comes with default Java-based logging utility Apache Log4j to collect runtime data
and record component events

25. Connect metrics and metrics using Prometheus
❏ Reports a variety of metrics through Java Management Extensions (JMX)
❏ task and worker metrics - status, running-ratio,
offset-commit-success-percentage, offset-commit-avg-time-ms, task-count,
connector-count, rebalancing metrics
❏ A variety of client metrics like connection-count, connection-close-rate,
network-io-rate, outgoing-byte-rate, request-rate etc
❏ gRPC Sink Connector metrics:
❏ sink-record-read-rate
❏ sink-record-active-count
❏ sink-record-read-total
❏ sink-record-send-rate

26. Connect metrics and metrics using Prometheus (cntd..)
❏ The monitoring tool Prometheus ingests metrics, makes them graphable, and
helps build alerts on top of metrics
❏ pulls metrics from HTTP endpoints added to the Prometheus configuration file
❏ provides JMX Exporter, a collector that can configurably scrape and expose
mBeans of a JMX target

27. Challenges and Lessons Learned
❏ Configuring more than one rpcs in a service
❏ Configuring rpcs with multiple arguments
❏ Testing the two components of the system
❏ Connectors have the capability to be extremely flexible, and can also hide
intricate logic when used off the shelf

28. Thank you!
Lets Connect over questions