Apache MADlib is an open-source library for scalable in-database analytics and machine learning in SQL, supporting both structured and unstructured data. It features a layered architecture involving Python drivers, C++ implementations, and a database abstraction layer, enabling various statistical and machine learning functions including deep learning and time series analysis. Additionally, it provides utilities for data transformation and model selection, and is compatible with databases like PostgreSQL and Greenplum.

1. What Is Apache MADlib ?
● For scalable in-database analytics
● Open source Apache 2.0 license
● For machine learning in SQL
● At big data scale
● Offers graph, statistics, analytics, deep learning
● Provides data-parallel implementations
● For structured and unstructured data

2. MADlib Prerequisites
● Currently supports databases
– PostgreSQL
● Needs Python extension specified
– Greenplum (distributed db)
– Apache Hawq ( v1.12+ ) (distributed db)
● Requires the GNU M4 Unix macro processor
● Works with Python 2.6 and 2.7

3. MADlib Architecture

4. MADlib Architecture
● MADlib has three main layers
● Python driver functions
– Main entry point from user input
– Largely responsible for algorithm flow control
– Validating input parameters
– Executing SQL statements
– Evaluating the results
– Potentially looping to execute more SQL statements
● Until some convergence criteria has been hit

5. MADlib Architecture
● MADlib has three main layers
● C++ implementations functions
– C++ definitions of the core functions/aggregates
● Needed for particular algorithms
– Implemented in C++ rather than Python
● For performance reasons

6. MADlib Architecture
● MADlib has three main layers
● C++ database abstraction layer
– Provide a programming interface
– Abstracts all the Postgres internal details
– Provides support for different back end platforms
– Focuses on the internal functionality
● Rather than the platform integration logic

7. MADlib Data Types and Transformations
● Arrays and Matrices
● Encoding Categorical Variables
● Path
● Pivot
● Sessionize
● Stemming

8. MADlib Graph Functionality
● All Pairs Shortest Path
● Breadth-First Search
● HITS
● Measures
● PageRank
● Single Source Shortest Path
● Weakly Connected Components

9. MADlib Model Selection / Sampling
● Model Selection
– Cross Validation
– Prediction Metrics
– Train-Test Split
● Sampling
– Balanced Sampling
– Stratified Sampling

10. MADlib Statistics / Supervised Learning
● Statistics
– Descriptive Statistics
– Inferential Statistics
– Probability Functions
● Supervised Learning
– Conditional Random Field
– k-Nearest Neighbors
– Neural Network
– Regression Models
– Support Vector Machines
– Tree Methods

11. MADlib Time Series / Unsupervised Learning
● Time Series Analysis
– ARIMA
● Unsupervised Learning
– Association Rules
– Clustering
– Dimensionality Reduction
– Topic Modelling

12. MADlib Utilities
● Columns to Vector
● Database Functions
● Linear Solvers
● Mini-Batch Preprocessor
● PMML Export
● Term Frequency
● Vector to Columns

13. MADlib Deep Learning Example SQL
● First define the model configurations to train
● Meaning either model architectures or hyperparameters
● Load them into a model selection table
● The combination of model architectures and hyperparameters
● Constitutes the model configurations to train
● In the picture there are three model configurations
● Represented by the three different purple shapes

14. MADlib Deep Learning Example SQL

15. MADlib Deep Learning Example SQL
● Once we have model combinations
● In the model selection table
● Call the fit function to train the models
– In parallel.
● In the picture the three orange shapes
● Represent the three models that have been trained

16. MADlib Deep Learning Example SQL

17. Available Books
● See “Big Data Made Easy”
– Apress Jan 2015
●
See “Mastering Apache Spark”
– Packt Oct 2015
●
See “Complete Guide to Open Source Big Data Stack
– “Apress Jan 2018”
● Find the author on Amazon
– www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
●
Connect on LinkedIn
– www.linkedin.com/in/mike-frampton-38563020

18. Connect
● Feel free to connect on LinkedIn
– www.linkedin.com/in/mike-frampton-38563020
● See my open source blog at
– open-source-systems.blogspot.com/
● I am always interested in
– New technology
– Opportunities
– Technology based issues
– Big data integration