The document discusses different technologies for real-time data collection and analysis, including Kafka for collecting and distributing streaming data, Storm for distributed real-time computation, and using PHP and FastCGI to parse real-time logs from Kafka in a Storm topology. It provides an overview of these technologies and their features, and proposes an architecture to collect logs with Kafka, process them with Storm and PHP, and output results through FastCGI.

1. PHP
|

2. PaaS SAE
Top3 SuperAPP

3. Pla$orm Services Group
Sam
AWS S3
CDN ATS
Biwing
1k
30+
10+
Monika
G o
Rambone
PHP Front-End
Services Panel
Skylee
C++ Go

4. Content Summary
"
# PHP
PHP
$ %
Case

5. Real 7me data collec7on
"

6. How to collect data in real 7me
?
…… ……

7. Ka<a ——
a high-throughput distributed messaging system source from LinkedIn
Producer
KaLa LinkedIn Apache
KaLa Producer
&
"
'
(
Model
Two models: queuing
and publish-subscribe.
the la<er is more
commonly used.
high-level abstracPon
Each Topic corresponds
to one or more log files.
Each Topic has one or
more par@@ons.
Feature
Scalability & Durability
& Reliability &
Performance & Fault-
tolerant
Ecosystem
Stream processing
systems, Hadoop
integraPon, monitoring,
and deployment tools.

8. Ka<a
should be no7ced when using Ka<a
)
KaLa offset
Consumer
Buffer
KaLa gzip snappy lz4
CPU
KaLa KaLa
KaLa
Topic Par@@on
Par@@on
Topic Par@@on
Zookeeper
KaLa Zookeeper
ZK ZK

9. Real-7me analysis
"

10. BackType nathanmarz Storm BackType
TwiXer Storm TwiXer TwiXer
Apache
' Feature
Storm ——
a distributed real-7me computa7on system source from TwiGer
Highly scalable
Fault-tolerant
Guarantees processing
Language agnosPc
* Topology of one Spout
* Topology of three Spout

11. * UI / Nimbus / ZK / Supervisor
Storm
Storm cluster organiza7on structure
Storm UI ZookeeperStorm Nimbus
Storm Supervisor
Topology

12. Storm
Storm mul7 language support
+
Python
PHP
JS
Shell
, Storm use STDIN & STDOUT
support multi language

13. PHP
Connect
Pool
Storm PHP
Connect Storm to PHP
+ PHP
STDIN
STDOUT
+
PHP
PHP-FPM
PHP
PHP
FastCGI

14. PHPphp analysis program
PHP

15. PHP
PHP parse a real 7me log
*
-> Kafka -> Storm -> Spout Bolt -> Bolt FCGI

16. PHP
PHP program for parse log
PHP
-
PHP-FPM
dynamic PHP
pm.max_children
PHP
DB Cache
Storm
web
RD
PHP
Input
PHP
PHP

17. Storm Bolt PHP
Storm bolt send a request by FastCgi protocol

18. Architecture design
$

19. Overall architecture design

20. Performance data

21. CaseShow Case
%

22. Sta7s7cal service pla$orm

23. Sta7s7cal service pla$orm

24. Real-7me log analyse service

25. Q & A
?