Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Realtime Risk Management Using Kafka, Python, and Spark Streaming
Similar to Realtime Risk Management Using Kafka, Python, and Spark Streaming (20)
Recently uploaded
Recently uploaded (20)
Realtime Risk Management Using Kafka, Python, and Spark Streaming
- 2. Realtime Risk ManagementUSING KAFKA, PYTHON, AND SPARK STREAMING
- 3. 3 UNDERWRITING CREDIT CARD TRANSACTIONS I S RISKY
- 4. 4 WE NEED TO BE QUICK AT UNDERWRIT ING
- 5. 5 WE ALSO NEED TO AVOID LOSIN G MONE Y
- 6. 6 Some Numbers $10BnC U M U L AT I V E P R O C E S S E D 175k+M E R C H A N T S 14kE V E N T S / S E C O N D 5R I S K A N A LYS T S
- 8. 8 WE NEEDED TO BU ILD SOMETHING THAT STOPS THE HOPE FACTOR
- 9. 9
- 10. 10 SPA RK STREA MI NG ALLOWS US TO DO REAL-TIME DATA P ROC ESSING. WE CAN DECIDE WHICH EVENTS NEED A CLOSER LOOK
- 11. 11 Intro to Kafka, Zookeeper, and Spark Streaming
- 12. 12 Apache Kafka Image taken from Kafka docs
- 13. 13 Apache Zookeeper Image taken from Zookeeper docs
- 14. 14 Apache Spark Streaming Image taken from mapr.com
- 15. 15 OLD WAY: REC EI VERS
- 16. 16 Kafka Receiver Spark Engine t0 t1 t2 t3 Event Event Event Event Event Event Build RDD with Events from t0 to t1 Build RDD with Events from t1 to t2 Process RDD Process RDD
- 17. 17 Problems w/ Receivers • The only way to get at-least once delivery makes it hard to deploy new code • Zookeeper is updated with which offsets to start from when data is received, not when it is processed • We’re actually duplicating Kafka
- 18. 18 NEW WAY: RECEI VERLESS
- 19. 19 Kafka Spark Engine t0 t1 t2 t3 Event Event Event Event Event Event Process RDD with Events from t0 to t1 Process RDD with Events from t1 to t2
- 20. 20 General Structure • Load Kafka offsets from Zookeeper • Tell Spark Streaming to create a DStream that consumes from Kafka, starting at the specified offsets • Define your processing step (ie. filter out non-risky events) • Define your output step (ie. POST the data to the case management software) • Save Kafka offset of most recently processed event to Zookeeper • Start your streaming application, and grab some popcorn!
- 21. 21 Example Filtering: Risky Products hair extensions pharmacy vaporizer gateway card wifi pineapple iPhone gucci cannabis travel package
- 22. 22 Risky Products hair extensions gucci cannabis sweet bag nice shoes taylor swift t-shirt RDD for Time 0 Filter hair extensions gucci cannabis Map {“title”: “hair exten…} {“title”: “gucci”…} {“title”: “cannab…} HTTPS Post Case Management Software
- 23. 23
- 24. 24 TIM E-WI NDOWE D AGGREG ATIO NS
- 25. 25 The Future • Time-Windowed Functions – A necessity for most of the non-trivial jobs • Performance Tweaks – We haven’t spent any time on this, so lots of potential for gains • Machine Learning – We could use the Risk Analyst decisions to build a ML model • Improved Monitoring – We are only monitoring the basics right now • Apache Cassandra – Others use it as a fast key/value store for their jobs • Improved Receiverless API – An API to access Kafka / Zookeeper without hard work
- 26. 26 Icon Credits Credit Card by Rediffusion from the Noun Project Money Bag by icon 54 from the Noun Project
- 27. 27 WE’RE HI RI NG SHOPI FY.COM /CAREE RS