The application and architecture of the real-time computing platform in di di-flink_forward_2018
•
0 likes•60 views
The application and architecture of the real-time computing platform in didi-flink_forward_2018
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The application and architecture of the real-time computing platform in di di-flink_forward_2018
Similar to The application and architecture of the real-time computing platform in di di-flink_forward_2018 (20)
Recently uploaded
Recently uploaded (20)
The application and architecture of the real-time computing platform in di di-flink_forward_2018
- 1. ᄆᄆਫᦇᓒଘݣຝӨਫ᪢ The Application and Architecture of the Real-Time Computing Platform in DiDi ݪلғᄆᄆڊᤈ Company: DiDi Chuxing ᘳ֖ғದᎸᑪާ Title Director of BigData Infrastructure Department ᄍᦖᘏғᗔ Lecturer Luo Li
- 2. Agenda Ø ᙧว݊ሿᇫ Ø ਫᥢ܃ڞᯈਫ᪢ Ø 6WUHDP64/Օᕨ Ø ਫᦇᓒଘݣຝ Ø ᕮᥢښ Ø Background Ø The CEP practice Ø StreamSQL Ø Stream Platform Infrastructure Ø Future plans
- 4. Ø ᄆᄆฎӞӻਫԻฃක Ø ᄆᄆጱහഝ࣋วॠᆐฎਫጱ ள ӫ սՁ ڊᑄ ᨟ … ෭பහഝ ईᅩහഝ Իฃහഝ ᭃහഝ ਫᦇᓒ ਫᵞ Ø DiDi is a realtime Trade Engine Ø Didi's data and scenes are naturally real-time Fast Cars Special Cars Shared Cars Taxi /X[XU &DUV … Log Data Collection point Data Transaction Data Trace Data Real-time process Real-time ETC & Collection
- 5. ၞᦇᓒದᄍᬰ ᒫӞᴤྦྷғӱۓොᛔୌੜᵞᗭ ᒫԫᴤྦྷғᵞӾୗय़ᵞᗭ̵ଘݣ۸ ᒫӣᴤྦྷғၞᦇᓒݎ64/۸ Storm/Jstorm/ Spark/Samza ӱۓොᛔୌੜᵞᗭ Spark Streaming ๐ۓ۸ଘݣ۸ Flink Streaming ಭفଫአ चԭFlink CEPጱਫᥢ܃ڞ ᯈ๐ۓӤᕚ चԭFlinkጱStreamSQLӤᕚ Didi’s Real-time develop steps First Step A lot of some Clusters Second Step Single Big Shared Cluster Third Step SQL On Stream process Storm/Jstorm/ Spark/Samza some clusters Spark Streaming Flink Streaming Flink CEP StreamSQL On Flink
- 6. ଫአ࣋ว Scenarios - -- Ø Ø HBase Druid ES Ø Ø Ø ਫ̵ڭݎਫᷚഴ̵ ਫଉ༄ၥᒵ Ø හഝၾ௳ᴚکڜHBase Druid ESᒵਂؙ Ø Իฃፊഴ̵๐ۓፊ ഴ̵ૡܔፊഴᒵ Ø ਫԙਮᇙ̵ݪᇙ̵ ᦈܔᇙᒵ
- 7. ଫአᥢཛྷ ๐ۓӱۓᕚ ᵞᗭᥢཛྷ ၞᦇᓒձۓහ ྯ෭॒ቘහഝᰁ Data Scale Applications Cluster Scale Streaming Jobs Data Scale per day
- 9. ਫᥢ܃ڞᯈҁCEP҂ ӻ۸០ᲀ Ø ʼnउ૱ԙਮٛုݸᑁဌํӥܔŊ Ø ʼnउ૱ӥݸܔᑁဌՈളܔŊ Ø શ꧊ᥢڞ Pattern .begin(s1) .where(key=bubble city=166) .notFollowedBy(s2) .where(key=order_send city = 166) .timeEnd(s3) .timeShiftMore(s1, Time.ms(5000)) ଉૡܔ༄ၥ Ø ᦈܔᇫாᳩᳵړᰦ
- 12. CEPۑᚆ ᚆදᬰ
- 13. StreamSQL
- 14. StreamSQLୌᦡ ӿ8') ၞ-2,1 ᖌᤒ-2,1 64/ ,'( ಘ''/
- 15. ಘ''/ Ø ᭗ᄆᄆٖ᮱Ԇၞၾ௳ᴚڜਫਂؙ Ø ᭗ᬦಘ''/ݙൈᬿ6RXUFH6LQN Ø ٖᗝ-621̵ຽٵ෭ப̵3%໒ୗᥴຉ Kafka MQ StreamSQL Kafka Druid HBase Phoenix ElaticSearch MYSQL MQ KVStore
- 17. ඪ೮चԭTTLጱၞJoin topic1 topic2 source1 source2 topic3 JoinOperator TTLState1 TTLState2 ၾ௳ᴚڜ StreamSQL ၾ௳ᴚڜ Sink Ø inner join Ø Left join Ø Full join
- 18. source2 topic2 JoinOperator TTLState Cache ၾ௳ᴚڜ ၾ௳ᴚڜ topic1 ྍ IO Sink ᖌᤒਂؙ HBase/MySQL) StreamSQL ᖌᤒJOIN
- 21. ਫᦇᓒଘݣຝ ଘ੶ݣ ක੶ SQL IDE WEB IDE ձۓ ᓕഴ ෙ ֛ᔮ ᤅᖭ ىᔮ ፊഴ ಸᦄ ग़ᑄಁ ֛ᔮ ਫ ਂؙ Hadoop ବଷ ړୗկᔮᕹғHDFSᕹӞᩒრ᧣ଶғYARN Flink Streaming StreamSQL ਫᥢ܃ڞᯈ ҁCEP DataStream
- 25. Ø ᇿᒈୌᦡਫᦇᓒᵞᗭ҅عᐶᕚձۓଗಟ Ø ਫᵞᗭੱᰁᶌᬪӱ҅ۓᴳ֗ᗑᕶ Ø ਧ᧣ګଶ Ø ᧣ଶ࣐ᤍ Ø चԭ38᧣ଶ Ø चԭ1RGH /DEHO*URXS᯿ᵍᐶګ Ø ग़಄᮱ᗟ Router ਫ᧣ଶ RM1 NM1 NM2 NM3 Lable1 Lable1 Lable2 HDFS1 ਫ᧣ଶ RM2 NM4 NM5 NM6 Lable3 Lable4 Lable4 HDFS1 DC1 DC2 Hadoopବଷ
- 27. ᕮ Ø 6WUHDP64/ य़ଏᴳ֗ၞᦇᓒݎᳪད Ø ਫᥢ܃ڞᯈҁ(3҂ ࣁӻ۸០ᲀ̵ਫᷚഴᒵᶾऒ៧ࣈ Ø ਫᦇᓒଘݣ Ӟᒊୗၞᦇᓒݎଘݣ Summary Ø StreamSQL makes the Streaming develop much easier Ø Real-time CEP very useful in the risk control Personalized marketing Ø A Big Shared Stream Processing platform
- 28. Ø ਠ౮6SDUN 6WUHDPLQJک6WUHDP64/ጱද᭜ Ø ,R7ᶾऒێݎ Ø 6WUHDP0/ୌᦡ Ø Streaming Job Spark Streaming - StreamSQL upgrade Ø In IoT (Cars/bicye/Monitor in Cars) Ø StreamML )OLQNᥢښ Flink Plans