One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool
Speaker: Nitin Lamba and Suhas Gogate, Ampool Big Data Applications Meetup, 09/14/2016 Palo Alto, CA More info here: http://www.meetup.com/BigDataApps/ Link to video: https://youtu.be/tGfPKYizZWY About the talk: Anomaly detection is a very common pattern used not only in financial transactions but also in finding abnormal behavior in health monitoring and IoT. What’s even more common is multiple analytical tools used in data science (Python, R, Apache Spark, to name a few) especially in large multi-tenant environments. Enterprises spend a lot of time moving & copying data to cater to these needs. Instead of having disparate back-end systems feed these tools, a simpler approach is to separate the concerns for compute and fast data serving. In this talk, we will walk through such an anomaly detection use-case, where an in-memory data service layer serves hot, high-value data to different tools from a single, scalable cluster. This not only reduces data copies but also mitigates operational complexity (less number of moving parts). We illustrate how a single data flow can use these multiple engines, making timely actionable insights a reality, and run concurrent analytics workloads at in-memory speeds.
Recommended
Recommended
More Related Content
More from Cask Data
More from Cask Data (10)
Recently uploaded
Recently uploaded (20)
One size doesn’t fit all: Federated Data Science using Ampool - by Nitin Lamba, Suhas Gogate, Ampool
- 1. Prepared for: BDAM One Size doesn’t fit all Making a case for Federated Data Science Nitin Lamba & Suhas Gogate (Riya Makhijani, Ruta Joshi, Shantanu Tulshibagwale)
- 2. Prepared for: BDAM Data Science is a multi-framework environment ! Data Scientists ! ! Business Analysts !! Data Engineers ! language tool
- 3. Prepared for: BDAM Data Processing is an iterative, multi-step process …" " " # # & # ModelExplore ServeFlatten
- 4. Prepared for: BDAM ModelExplore ServeFlatten " " " # # & # … What if fast object access is available across stages?
- 5. Prepared for: BDAM ModelExplore ServeFlatten " " " # # # … & Reduce time to insights, build real-time models ✅ Object-based data exchange ✅ No Data Copies ✅ End-to-end speedup ✅ Increased Concurrency
- 6. Prepared for: BDAM Ampool is a memory-oriented Active Data Store... A primary store for ALL data processing Store ALL active data & update it, as reqd. Serves data concurrently to multiple stages & tenants Data Persistence
- 7. Prepared for: BDAM ...offers choice for different analytical needs... ' ( ) ) … MPP DatabaseLocal FileSystem Software-Defined Memory * Experimental; waiting feedback from design partners … …
- 8. Prepared for: BDAM …and delivers value to all types of data users No change in application logic Make config. changes only No change in existing user tools Get memory speeds No hassle deployment Use current mgmt. tools ! Data Architect Data Engineers ! ! Business Analysts Data Scientists ! ! Data Admins Infra/ Sys Admins !
- 9. Prepared for: BDAM In-Memory Distributed Sys Low-latency Comms. Key-Value Store Function Pushdown + High Throughput Table Store Native InterfacePluggable Store Manager Java API MASH (CLI Ext) Java API It is built on a proven in-memory technology Smart Data Tiering Mature Event Model Tunable Consistency Metadata/ Catalog Security AuthZ
- 10. Prepared for: BDAM Card Holder Init Purchase Pay bill Merchant Scans Card Processes Tx ABC Bank Offers Cards Serves Bills Anomaly Detection: An illustrative use-case… Transaction Logs Tx Tx Tx… Tx Tx Tx … Acquirer Intermediates Tx Issuer Process request Monitors activity Billing Events Network *++ Aggregates Tx 1 2 3 4 5 6 7 8 9 0
- 11. Prepared for: BDAM …with Kafka, Spark, R & Ampool… Ingest Rep at the Issuing Bank Financial Transactions Stream & Batch Processing Monitoring/ App Tx Validate User Profile $ impact Explore & Model
- 12. Prepared for: BDAM R Studio …supporting multiple analytical tools KafkaData Generator Ingest Streaming Explore/ Model Cross Valid Tx Log Outliers User/ App Data Fin Tx Logs Spark Streaming Fin_Txn Tx Confirm. User Updates Dstream RDD In_DF Model Spark Batch/ ML Batch_DF Batch
- 14. Prepared for: BDAM Questions? , /company/ampool-inc- -. /AmpoolIO@AmpoolIO/ www.ampool.io