using Storm and PHP analysis big data real-time

1. PHP
|

2. PaaS SAE
Top3 SuperAPP

3. Pla$orm Services Group
Sam
AWS S3
CDN ATS
Biwing
1k
30+
10+
Monika
G o
Rambone
PHP Front-End
Services Panel
Skylee
C++ Go

4. Content Summary
"
# PHP
PHP
$ %
Case

5. Real 7me data collec7on
"

6. How to collect data in real 7me
?
…… ……

7. Ka<a ——
a high-throughput distributed messaging system source from LinkedIn
Producer
KaLa LinkedIn Apache
KaLa Producer
&
"
'
(
Model
Two models: queuing
and publish-subscribe.
the la<er is more
commonly used.
high-level abstracPon
Each Topic corresponds
to one or more log files.
Each Topic has one or
more par@@ons.
Feature
Scalability & Durability
& Reliability &
Performance & Fault-
tolerant
Ecosystem
Stream processing
systems, Hadoop
integraPon, monitoring,
and deployment tools.

8. Ka<a
should be no7ced when using Ka<a
)
KaLa offset
Consumer
Buffer
KaLa gzip snappy lz4
CPU
KaLa KaLa
KaLa
Topic Par@@on
Par@@on
Topic Par@@on
Zookeeper
KaLa Zookeeper
ZK ZK

9. Real-7me analysis
"

10. BackType nathanmarz Storm BackType
TwiXer Storm TwiXer TwiXer
Apache
' Feature
Storm ——
a distributed real-7me computa7on system source from TwiGer
Highly scalable
Fault-tolerant
Guarantees processing
Language agnosPc
* Topology of one Spout
* Topology of three Spout

11. * UI / Nimbus / ZK / Supervisor
Storm
Storm cluster organiza7on structure
Storm UI ZookeeperStorm Nimbus
Storm Supervisor
Topology

12. Storm
Storm mul7 language support
+
Python
PHP
JS
Shell
, Storm use STDIN & STDOUT
support multi language

13. PHP
Connect
Pool
Storm PHP
Connect Storm to PHP
+ PHP
STDIN
STDOUT
+
PHP
PHP-FPM
PHP
PHP
FastCGI

14. PHPphp analysis program
PHP

15. PHP
PHP parse a real 7me log
*
-> Kafka -> Storm -> Spout Bolt -> Bolt FCGI

16. PHP
PHP program for parse log
PHP
-
PHP-FPM
dynamic PHP
pm.max_children
PHP
DB Cache
Storm
web
RD
PHP
Input
PHP
PHP

17. Storm Bolt PHP
Storm bolt send a request by FastCgi protocol

18. Architecture design
$

19. Overall architecture design

20. Performance data

21. CaseShow Case
%

22. Sta7s7cal service pla$orm

23. Sta7s7cal service pla$orm

24. Real-7me log analyse service

25. Q & A
?