The application and architecture of the real-time computing platform in di di-flink_forward_2018
•
0 likes•63 views
The document discusses DiDi's real-time computing platform and its use of stream processing technologies. It covers DiDi's business characteristics requiring real-time capabilities, its steps to develop a real-time platform using Storm/Spark/Flink, its use of Flink CEP for complex event processing, StreamSQL for queries on streaming data, and its stream processing infrastructure involving Kafka, Druid, and other technologies. Future plans include expanding StreamSQL support, enhancing real-time CEP capabilities, and building a large shared stream processing platform.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The application and architecture of the real-time computing platform in di di-flink_forward_2018
Similar to The application and architecture of the real-time computing platform in di di-flink_forward_2018 (20)
Recently uploaded
Recently uploaded (20)
The application and architecture of the real-time computing platform in di di-flink_forward_2018
- 1. ᄆᄆਫᦇᓒଘݣຝӨਫ᪢ The Application and Architecture of the Real-Time Computing Platform in DiDi ݪلғᄆᄆڊᤈ Company: DiDi Chuxing ᘳ֖ғದᎸᑪާ Title Director of BigData Infrastructure Department ᄍᦖᘏғᗔ Lecturer Luo Li
- 2. Agenda Ø ᙧว݊ሿᇫ Ø ਫᥢ܃ڞᯈਫ᪢ Ø 6WUHDP64/Օᕨ Ø ਫᦇᓒଘݣຝ Ø ᕮᥢښ Ø Background Ø The CEP practice Ø StreamSQL Ø Stream Platform Infrastructure Ø Future plans
- 4. Ø ᄆᄆฎӞӻਫԻฃක Ø ᄆᄆጱහഝ࣋วॠᆐฎਫጱ ள ӫ սՁ ڊᑄ ᨟ … ෭பහഝ ईᅩහഝ Իฃහഝ ᭃහഝ ਫᦇᓒ ਫᵞ Ø DiDi is a realtime Trade Engine Ø Didi's data and scenes are naturally real-time Fast Cars Special Cars Shared Cars Taxi /X[XU &DUV … Log Data Collection point Data Transaction Data Trace Data Real-time process Real-time ETC & Collection
- 5. ၞᦇᓒದᄍᬰ ᒫӞᴤྦྷғӱۓොᛔୌੜᵞᗭ ᒫԫᴤྦྷғᵞӾୗय़ᵞᗭ̵ଘݣ۸ ᒫӣᴤྦྷғၞᦇᓒݎ64/۸ Storm/Jstorm/ Spark/Samza ӱۓොᛔୌੜᵞᗭ Spark Streaming ๐ۓ۸ଘݣ۸ Flink Streaming ಭفଫአ चԭFlink CEPጱਫᥢ܃ڞ ᯈ๐ۓӤᕚ चԭFlinkጱStreamSQLӤᕚ Didi’s Real-time develop steps First Step A lot of some Clusters Second Step Single Big Shared Cluster Third Step SQL On Stream process Storm/Jstorm/ Spark/Samza some clusters Spark Streaming Flink Streaming Flink CEP StreamSQL On Flink
- 6. ଫአ࣋ว Scenarios - -- Ø Ø HBase Druid ES Ø Ø Ø ਫ̵ڭݎਫᷚഴ̵ ਫଉ༄ၥᒵ Ø හഝၾ௳ᴚکڜHBase Druid ESᒵਂؙ Ø Իฃፊഴ̵๐ۓፊ ഴ̵ૡܔፊഴᒵ Ø ਫԙਮᇙ̵ݪᇙ̵ ᦈܔᇙᒵ
- 7. ଫአᥢཛྷ ๐ۓӱۓᕚ ᵞᗭᥢཛྷ ၞᦇᓒձۓහ ྯ෭॒ቘහഝᰁ Data Scale Applications Cluster Scale Streaming Jobs Data Scale per day
- 9. ਫᥢ܃ڞᯈҁCEP҂ ӻ۸០ᲀ Ø ʼnउ૱ԙਮٛုݸᑁဌํӥܔŊ Ø ʼnउ૱ӥݸܔᑁဌՈളܔŊ Ø શ꧊ᥢڞ Pattern .begin(s1) .where(key=bubble city=166) .notFollowedBy(s2) .where(key=order_send city = 166) .timeEnd(s3) .timeShiftMore(s1, Time.ms(5000)) ଉૡܔ༄ၥ Ø ᦈܔᇫாᳩᳵړᰦ
- 12. CEPۑᚆ ᚆදᬰ
- 13. StreamSQL
- 14. StreamSQLୌᦡ ӿ8') ၞ-2,1 ᖌᤒ-2,1 64/ ,'( ಘ''/
- 15. ಘ''/ Ø ᭗ᄆᄆٖ᮱Ԇၞၾ௳ᴚڜਫਂؙ Ø ᭗ᬦಘ''/ݙൈᬿ6RXUFH6LQN Ø ٖᗝ-621̵ຽٵ෭ப̵3%໒ୗᥴຉ Kafka MQ StreamSQL Kafka Druid HBase Phoenix ElaticSearch MYSQL MQ KVStore
- 17. ඪ೮चԭTTLጱၞJoin topic1 topic2 source1 source2 topic3 JoinOperator TTLState1 TTLState2 ၾ௳ᴚڜ StreamSQL ၾ௳ᴚڜ Sink Ø inner join Ø Left join Ø Full join
- 18. source2 topic2 JoinOperator TTLState Cache ၾ௳ᴚڜ ၾ௳ᴚڜ topic1 ྍ IO Sink ᖌᤒਂؙ HBase/MySQL) StreamSQL ᖌᤒJOIN
- 21. ਫᦇᓒଘݣຝ ଘ੶ݣ ක੶ SQL IDE WEB IDE ձۓ ᓕഴ ෙ ֛ᔮ ᤅᖭ ىᔮ ፊഴ ಸᦄ ग़ᑄಁ ֛ᔮ ਫ ਂؙ Hadoop ବଷ ړୗկᔮᕹғHDFSᕹӞᩒრ᧣ଶғYARN Flink Streaming StreamSQL ਫᥢ܃ڞᯈ ҁCEP DataStream
- 25. Ø ᇿᒈୌᦡਫᦇᓒᵞᗭ҅عᐶᕚձۓଗಟ Ø ਫᵞᗭੱᰁᶌᬪӱ҅ۓᴳ֗ᗑᕶ Ø ਧ᧣ګଶ Ø ᧣ଶ࣐ᤍ Ø चԭ38᧣ଶ Ø चԭ1RGH /DEHO*URXS᯿ᵍᐶګ Ø ग़಄᮱ᗟ Router ਫ᧣ଶ RM1 NM1 NM2 NM3 Lable1 Lable1 Lable2 HDFS1 ਫ᧣ଶ RM2 NM4 NM5 NM6 Lable3 Lable4 Lable4 HDFS1 DC1 DC2 Hadoopବଷ
- 27. ᕮ Ø 6WUHDP64/ य़ଏᴳ֗ၞᦇᓒݎᳪད Ø ਫᥢ܃ڞᯈҁ(3҂ ࣁӻ۸០ᲀ̵ਫᷚഴᒵᶾऒ៧ࣈ Ø ਫᦇᓒଘݣ Ӟᒊୗၞᦇᓒݎଘݣ Summary Ø StreamSQL makes the Streaming develop much easier Ø Real-time CEP very useful in the risk control Personalized marketing Ø A Big Shared Stream Processing platform
- 28. Ø ਠ౮6SDUN 6WUHDPLQJک6WUHDP64/ጱද᭜ Ø ,R7ᶾऒێݎ Ø 6WUHDP0/ୌᦡ Ø Streaming Job Spark Streaming - StreamSQL upgrade Ø In IoT (Cars/bicye/Monitor in Cars) Ø StreamML )OLQNᥢښ Flink Plans