https://www.meetup.com/fr-FR/afsy-sfpot/events/258670432/

1. Kafka
Consume topics in PHP
from nightmare to performance

2. Loïc Doubinine

3. ~500 serveurs Paris / NY
1,5 To de logs par jour
~1B de requêtes / jour

4. KAFKA

5. What is Kafka ?

6. Key concepts
Messages Topic
Topic Partition
Partition Messages ordered
Partition Consumers
One partition
Consumer

7. How to use it in PHP

8. Kafka in php
librdkafka
$conf = new RdKafkaConf();
$conf->set('group.id', 'myConsumerGroup');
$conf->set('metadata.broker.list', '127.0.0.1');
$topicConf = new RdKafkaTopicConf();
$topicConf->set('auto.offset.reset', 'smallest');
$conf->setDefaultTopicConf($topicConf);
$consumer = new RdKafkaKafkaConsumer($conf);
$consumer->subscribe(['greatTopic']);
while (true) {
$message = $consumer->consume(120*1000);
switch ($message->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
var_dump($message);
break;
case RD_KAFKA_RESP_ERR__PARTITION_EOF:
echo "No more messages; will wait for moren";
break;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
echo "Timed outn";
break;
default:
throw new Exception($message->errstr(),
$message->err);
break;
}
}
php-rdkafka

9. How we did it

10. Reference topic for tests
↑30 K/
~60 K/
~1.3 bn/
80

11. What can we expect from kafka ?
Kafka-cat
30 K/

12. Naïve implementation
Topic
we did
we got
~3.2 m/ ~37 / ~1-2Gb /

13. Forks to process messages in parallel
Topic
we did
we got
0 / 0 / ~1-2Gb /

15. Back to naïve implementation + fork consumer
Topic
we did
we got
fork
Still
~3.2 m/ ~37 / ~1-2Gb /

16. Adjust librdkafka configuration
●
●
we did
we got
still
still
still
smaller memory footprint
~3.2 m/ ~37 / ~100mb /

17. Die them all
we did
we got
consumption stop for ~30s each time a fork die
~1.6 m/ ~15 / ~100mb /

19. ●
●
●
●
we did
Client-server

20. we got
fork process messages then die, no memory leak, no trace
~865 m/ ~10 K/ ~100mb + 100mb/
Client-server

22. How to increase processing rate ?
~865 m/ ~100mb + 100mb /~865 m/ ~10 K/ ~100mb + 100mb/

23. Start more consumers/fork
we did
we got
~865 m/ ~100mb + 100mb /~865 m/ ~10 K/ ~100mb + 100mb/

25. How to increase processing rate ?
~865 m/ ~100mb + 100mb /~865 m/ ~10 K/ ~100mb + 100mb/

26. Strace consumer

27. What is stracing ?

29. Force communication protocol
●
●
we did
we got
~1.6 bn/ ~30 K/ ~100mb + 100mb/

30. Finally
Kafka-cat
30 K/ /
Client-server
30 K/ /

31. Reference topic for tests
↑30 K/
~60 K/
~1.3 bn/
80

33. Takeaway

34. ●
●
●
●
●
Takeway

35. THANK YOU
35