Strata London Talk 2019. This presentation covers the architecture of key components of our AI based Master Data Management System for any entity, any format, any scale using Cassandra, Elastic, Spark and Machine Learning

1. Mastering Data with Spark
and ML
Strata London 2019

2. About Me
IIT Delhi, 1998
Founder and CEO, Nube Technologies
Strata Data San Jose Program Committee
Speaker at Spark Summit, Strata, GIDS etc

3. Nube
India based startup
Deep technical problems with an enterprise solution
ML, Big Data, UX

4. This talk today
Problem Statement
Our Approach

5. Simple business asks
Customer LTV
Best supplier for a part
Supplier payment terms
Householding
Cross Sell Opportunities
M&A

6. Actual Data

8. Actual data - real facts
Silos
Data Quality
Volumes

9. Challenges
Variety of sources
Scale
Capturing rules for matching and merging for different business entities

10. Wishlist
Any source and format
Any entity type
Any volume

11. Reifier
AI powered data management, matching and merging different data sources to
build a holistic view.
- MDM
- Fraud and Analytics
- Sales and Marketing
- Customer AML/KYC/cross and Upsell
- Data Enrichment
- Reference data Management
- Data Quality

12. Our stack

13. Wishlist
Any source and format
Any entity type
Any scale

14. Any source, Any format
Based on RDDs
RDD = fault-tolerant collection of elements that can be operated on in parallel.
Custom source and sink formats written by us/borrowed from community

15. Any source, Any format
Elastic:
import org.elasticsearch.spark.rdd.EsSpark
EsSpark.saveToEs(rdd, "spark/docs")
Cassandra:
val rdd = sc.cassandraTable("test", "kv")
rdd.saveToCassandra("test", "kv", SomeColumns("key", "value"))

16. Problems with RDDs
Record wise reading was good, but adding structure to the data was left to us.
reifier.Tuple - indexed data structure
Development and maintenance nightmare

17. Reifier 2.0
- Datasets
- Pipe abstraction
public class Pipe implements Serializable{
String name;
Format format;
Map<String, String> props;
StructType schema;
}

18. Building Dataset through Pipe
DataFrameReader reader = spark.read();
reader = reader.format(p.getFormat().type());
reader = reader.schema(p.getSchema());
for (String key: p.getProps().keySet()) {
reader = reader.option(key, p.get(key));
}
Dataset<Row> input = reader.load();

19. A word of caution
Abstraction is good, but

20. Wishlist
Any source and format
Any entity type
Any scale

21. Any entity type
Traditional rule based system fails
AI to the rescue
Also Cassandra

22. Reifier Interactive Learner

23. Reifier Interactive Learner

24. Wishlist
Any source and format
Any entity type
Any scale

25. Any scale
Add Spark to the mix
Ouch, cartesian join - 1 million records = Order of a trillion comparisons
Learn what to join

26. AutoML
Build multiple models based on the training data
Optimize for accuracy and performance
Use Spark to train and assess different models

27. Spark Integration
Livy
Spark Job Server

30. Spark Integration
Two ways in which we integrate.
-Local SparkContext.
-Programmatically through hidden Spark REST service

31. Cassandra
Any Entity
Any Scale

32. Cassandra Training
Primary Key - Cluster Id, Record Id
Secondary Index - r_isMatch

33. Cassandra Entity
Primary Key - Record Id
Secondary Index - Cluster Id

34. Elastic
View over Cassandra
Free flowing search
Ad hoc analytics
Realtime Plugin

35. Wishlist
Any source and format
Any entity type
Any scale

36. Thank You!
www.nubetech.co
sonal@nubetech.co