The document provides instructions on how to install, configure, and use various Kafka tools including kaf, kafkacat, and Node-RED. It shows how to produce messages to and consume messages from Kafka topics using the command line tools kaf and kafkacat. It also demonstrates integrating Kafka with Node-RED by adding Kafka nodes to consume and produce messages.

1. Useful tools

2. kaf

3. Install it
curl https://raw.githubusercontent.com/infinimesh/kaf/master/godownloader.sh
| BINDIR=$HOME/bin bash

4. Configure it
~/.kaf/config
current-cluster: dale
clusters:
- name: dale
brokers:
- bootstrap_host:port
SASL:
mechanism: PLAIN
username: token
password: API_KEY
TLS:
cafile: /Users/dalelane/dev/qp/auth/es-cert.pem
insecure: false
security-protocol: SASL_SSL

5. Configure it
$ kaf config use-cluster dale
Switched to cluster "dale".

6. Configure it
$ kaf config use-cluster dale
Switched to cluster "dale".
$ kaf config select-cluster
Use the arrow keys to navigate: ↓ ↑ → ←
? Select cluster:
▸ dale
ed
tim
nic
↓ matt

7. Produce with it
$ echo "Hello World" | kaf produce DEMO

8. Produce with it
$ echo "Hello World" | kaf produce DEMO
Sent record to partition 0 at offset 0.

9. Produce with it
$ echo "Hello World" | kaf produce DEMO
Sent record to partition 0 at offset 0.
$ echo -n "Bonjour le monde" | kaf produce DEMO
Sent record to partition 0 at offset 1.

10. Produce with it
$ echo "Hello World" | kaf produce DEMO
Sent record to partition 0 at offset 0.
$ echo -n "Bonjour le monde" | kaf produce DEMO
Sent record to partition 0 at offset 1.
$ cat README.md | kaf produce DEMO
Sent record to partition 0 at offset 2.

11. Consume with it
$ kaf consume DEMO

12. Consume with it
$ kaf consume DEMO
Partition: 0
Offset: 0
Timestamp: 2019-07-11 19:41:10.178 +0100 BST
Hello World
Partition: 0
Offset: 1
Timestamp: 2019-07-11 19:42:36.479 +0100 BST
Bonjour le monde
Partition: 0
Offset: 2
Timestamp: 2019-07-11 19:43:29.248 +0100 BST
# Ansible ICP Installer
Ansible scripts to install IBM Cloud Private and IBM Event

13. Consume with it
$ kaf consume DEMO -f

14. What else?
$ kaf topic describe DEMO
Name: DEMO
Internal: false
Compacted: false
Partitions:
Partition High Watermark Leader Replicas ISR
--------- -------------- ------ -------- ---
0 3 2 [0 1 2] [0 1 2]
Config:
Name Value ReadOnly Sensitive
---- ----- -------- ---------
min.insync.replicas 2 false false
retention.ms 604800000 false false
message.format.version 2.2-IV1 false false

15. What else?
$ kaf topic create ANOTHER
Created topic ANOTHER.
$ kaf topic ls
NAME PARTITIONS REPLICAS
ANOTHER 1 1
DEMO 1 3
$ kaf topic delete ANOTHER
Deleted topic ANOTHER.

16. kafkacat

17. Install it
$ brew install kafkacat

18. Configure it
~/.config/kafkacat.conf
bootstrap.servers=bootstrap_host:port
ssl.ca.location=/Users/dalelane/dev/qp/auth/es-cert.pem
security.protocol=sasl_ssl
sasl.mechanisms=PLAIN
sasl.username=token
sasl.password=API_KEY

19. Produce with it
$ kafkacat -P -t DEMO

20. Produce with it
$ kafkacat -P -t DEMO
Hello World

21. Produce with it
$ kafkacat -P -t DEMO
Hello World
Bonjour le monde

22. Produce with it
$ kafkacat -P -t DEMO
Hello World
Bonjour le monde
$ kafkacat -P -t DEMO README.md

23. Produce with it
$ kafkacat -P -t DEMO
Hello World
Bonjour le monde
$ kafkacat -P -t DEMO README.md
$ kafkacat -P -t DEMO -H mykey=myvalue README.md

24. Consume with it
$ kafkacat -C -t DEMO

25. Consume with it
$ kafkacat -C -t DEMO
Hello World
Bonjour le monde
% Reached end of topic DEMO [0] at offset 2

26. Consume with it
$ kafkacat -C -t DEMO -f
'NEXT MESSAGEn============nheaders:n%hnmessage:n%snn'

27. Consume with it
$ kafkacat -C -t DEMO -f
'NEXT MESSAGEn============nheaders:n%hnmessage:n%snn'
NEXT MESSAGE
============
headers:
message:
Hello World
NEXT MESSAGE
============
headers:
mykey=myvalue
message:
Bonjour le monde

28. What else?
$ kafkacat -L
Metadata for all topics (from broker -1:
sasl_ssl://9.30.249.220:31250/bootstrap):
3 brokers:
broker 0 at 9.30.249.220:31203 (controller)
broker 2 at 9.30.249.220:32599
broker 1 at 9.30.249.220:30797
1 topics:
topic "DEMO" with 1 partitions:
partition 0, leader 1, replicas: 1,2,0, isrs: 1,2,0

29. Node-RED

30. Install it
$ sudo npm install -g --unsafe-perm node-red

31. Run it
$ export CPPFLAGS=-I/usr/local/opt/openssl/include
$ export LDFLAGS=-L/usr/local/opt/openssl/lib
$ node-red

32. Add Kafka nodes to it

33. Consume with it

34. Consume with it

35. Consume with it

36. Consume with it

37. Consume with it

38. Consume with it

39. Produce with it

40. Produce with it