Successfully reported this slideshow.
Your SlideShare is downloading. ×

Kafka Rest.pptx

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 14 Ad

More Related Content

Recently uploaded (20)

Advertisement

Kafka Rest.pptx

  1. 1. Intro Kafka Cluster incoming message - without kafka client library support (e.g. Android or IOS app, IOT devices, …) - send message using Rest APIs
  2. 2. Intro source : https://docs.confluent.io/platform/current/kafka-rest/index.html#confluent-rest-apis
  3. 3. License source : https://www.confluent.io/confluent-community-license-faq/
  4. 4. Installation • wget https://packages.confluent.io/archive/7.3/confluent-community-7.3.0.tar.gz • tar -zxvf confluent-community-7.3.0.tar.gz [root@master ~]# cd confluent-7.3.0/ [root@master confluent-7.3.0]# vi etc/kafka-rest/kafka-rest.properties #id=kafka-rest-test-server #schema.registry.url=http://localhost:8081 zookeeper.connect=master:2181,slave1:2181,slave2:2181 bootstrap.servers=master:9092,slave1:9092,slave2:9092 By default the server starts bound to port 8082, does not specify a unique instance ID (required to safely run multiple proxies concurrently) configuration settings : https://docs.confluent.io/platform/current/kafka-rest/production-deployment/rest-proxy/config.html#configuration-options-for-https
  5. 5. Run & Test • 1. Zookeeper Ensemble, Kafka Cluster Running • 2. execute shell at kafka rest home directory sh bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties default port : 8082
  6. 6. 1. List all the topics in a Kafka cluster • curl -X GET "http://localhost:8082/topics/"
  7. 7. 2. Query Kafka Topic • curl -X GET -H "Accept: application/vnd.kafka.v2+json" "http://localhost:8082/topics/kopo-topic"
  8. 8. 3. Sending data to Kafka Topic • curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -- data '{"records":[{"value":{"name": "testUser"}}]}' "http://localhost:8082/topics/kopo-topic" Producer Consumer
  9. 9. 4. Prepare Consumer • consumer id and group registration • curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"name": "my_consumer_instance", "format": "json", "auto.offset.reset": "earliest"}' http://localhost:8082/consumers/my_consumer_group
  10. 10. 4-1. Mapping topic with consumer • curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"topics":["kopo-topic"]}' http://localhost:8082/consumers/my_consumer_group/instances/my_consumer_instance /subscription
  11. 11. 4-2. Receiving data • curl -X GET -H "Accept: application/vnd.kafka.json.v2+json" http://localhost:8082/consumers/my_consumer_group/instances/my_consumer_instance/records
  12. 12. etc. • Idle consumer will be destroyed after 300 seconds by default • Delete curl -X DELETE -H "Content-Type: application/vnd.kafka.protobuf.v2+json" http://localhost:8082/consumers/[consumer group]/instances/[consumer instance id] source : https://docs.confluent.io/platform/current/kafka-rest/production-deployment/rest-proxy/config.html#configuration-options-for-https for more examples : https://docs.confluent.io/platform/current/kafka-rest/quickstart.html#produce-and-consume-json-schema-messages
  13. 13. I Think, • In environments where the Kafka-client is not available, Rest Proxy is a good option to consider. • Detailed options are not available compared to implementing directly with the Kafka-client. • The Rest Proxy shall be used after the performance test. • I think it is more appropriate to apply a reverse proxy using nginx, etc., and link it with a back-end application using the kafka-client.

Editor's Notes

  • listenersComma-separated list of listeners that listen for API requests over either HTTP or HTTPS. If a listener uses HTTPS, the appropriate SSL configuration parameters need to be set as well.
    Type: list
    Default: http://0.0.0.0:8082
    Importance: high

    < configuration example >
    #id=kafka-rest-test-server
    #schema.registry.url=http://localhost:8081
    zookeeper.connect=master:2181,slave1:2181,slave2:2181
    bootstrap.servers=master:9092,slave1:9092,slave2:9092
    listeners=http://0.0.0.0:8083

×