Speaker: Nitin Lamba and Suhas Gogate, Ampool Big Data Applications Meetup, 09/14/2016 Palo Alto, CA More info here: http://www.meetup.com/BigDataApps/ Link to video: https://youtu.be/tGfPKYizZWY About the talk: Anomaly detection is a very common pattern used not only in financial transactions but also in finding abnormal behavior in health monitoring and IoT. What’s even more common is multiple analytical tools used in data science (Python, R, Apache Spark, to name a few) especially in large multi-tenant environments. Enterprises spend a lot of time moving & copying data to cater to these needs. Instead of having disparate back-end systems feed these tools, a simpler approach is to separate the concerns for compute and fast data serving. In this talk, we will walk through such an anomaly detection use-case, where an in-memory data service layer serves hot, high-value data to different tools from a single, scalable cluster. This not only reduces data copies but also mitigates operational complexity (less number of moving parts). We illustrate how a single data flow can use these multiple engines, making timely actionable insights a reality, and run concurrent analytics workloads at in-memory speeds.

1. Prepared for: BDAM
One Size doesn’t fit all
Making a case for Federated Data Science
Nitin Lamba &
Suhas Gogate
(Riya Makhijani, Ruta Joshi, Shantanu Tulshibagwale)

2. Prepared for: BDAM
Data Science is a multi-framework environment
! Data
Scientists
! ! Business
Analysts
!! Data
Engineers
!
language
tool

3. Prepared for: BDAM
Data Processing is an iterative, multi-step process
…" " "
# # 

&
#
ModelExplore ServeFlatten

4. Prepared for: BDAM
ModelExplore ServeFlatten
" " "
# #


&
#




…
What if fast object access is available across stages?

5. Prepared for: BDAM
ModelExplore ServeFlatten
" " "
# #

#




…
&
Reduce time to insights, build real-time models
✅ Object-based data
exchange
✅ No Data Copies
✅ End-to-end speedup
✅ Increased
Concurrency

6. Prepared for: BDAM
Ampool is a memory-oriented Active Data Store...
A primary store for ALL
data processing
Store ALL active data &
update it, as reqd.
Serves data concurrently
to multiple stages &
tenants
Data
Persistence


7. Prepared for: BDAM
...offers choice for different analytical needs...
'
( ) ) …
MPP DatabaseLocal FileSystem
Software-Defined
Memory
* Experimental; waiting feedback from design partners
…
…

8. Prepared for: BDAM
…and delivers value to all types of data users
No change in application
logic
Make config. changes only
No change in existing
user tools
Get memory speeds
No hassle deployment
Use current mgmt. tools
! Data Architect
Data Engineers
!
! Business Analysts
Data Scientists
!
! Data Admins
Infra/ Sys Admins
!

9. Prepared for: BDAM
In-Memory Distributed Sys
Low-latency Comms.
Key-Value Store
Function Pushdown
+
High Throughput
Table Store
Native InterfacePluggable Store Manager
Java API
MASH (CLI Ext)
Java API
It is built on a proven in-memory technology
Smart Data Tiering
Mature Event Model
Tunable Consistency
Metadata/ Catalog
Security AuthZ

10. Prepared for: BDAM
Card Holder
Init Purchase
Pay bill
Merchant
Scans Card
Processes Tx
ABC Bank
Offers Cards
Serves Bills
Anomaly Detection: An illustrative use-case…
Transaction Logs
Tx Tx Tx…
Tx Tx Tx …
Acquirer
Intermediates
Tx
Issuer
Process request
Monitors activity
Billing
Events
Network
*++
Aggregates Tx
1 2 3
4 5 6
7 8 9
0

11. Prepared for: BDAM
…with Kafka, Spark, R & Ampool…
Ingest
Rep at the
Issuing Bank Financial
Transactions
Stream & Batch
Processing
Monitoring/ App
Tx Validate
User Profile
$ impact
Explore
& Model

12. Prepared for: BDAM
R Studio
…supporting multiple analytical tools
KafkaData Generator
Ingest Streaming Explore/ Model
Cross Valid Tx Log Outliers
User/ App Data
Fin Tx Logs
Spark Streaming
Fin_Txn
Tx Confirm.
User Updates
Dstream RDD
In_DF
Model
Spark Batch/ ML
Batch_DF
Batch

13. Prepared for: BDAM
Demo

14. Prepared for: BDAM
Questions?
, /company/ampool-inc- -. /AmpoolIO@AmpoolIO/ www.ampool.io