Knowing how to quickly create advanced analytical models and deploy them efficiently is the true calling card of the data-driven enterprise. Join us for the Pivotal Analytics Innovation Roadshow, a day-long event for data professionals, where you will learn how to roll out successful analytics initiatives while avoiding traps that lead to delays and setbacks. Participants will get practical advice and best practices for building a data-driven enterprise from a seasoned team of data and analytics experts.

1. Pivotal Data Science
Technical Workshop
Scott Hajek
shajek@pivotal.io
April Song
asong@pivotal.io

2. Workshop Agenda
1. Data Science and Parallelism
2. Pivotal Greenplum Fundamental
Concepts
3. Airline Optimization Use Case

3. Parallelism: Crucial for Analytics at Scale
• Explicit Parallelism
• Problems that are easy to break up into a number of parallel tasks
• No dependency (or communication) between those parallel tasks
• Examples
• Have each person in this room weigh themselves
• Count a deck of cards by dividing it up between people in this room
• map() function in Python
• apply() family of functions in R

4. Pivotal Big Data Technology: Pivotal Greenplum & HDB
• Performance through massive
parallelism
• Automatic parallelization
• Load and query like any database
• Automatically distributed tables across
nodes
• Analytics-oriented query optimization
• Scalable MPP architecture
• All nodes can scan and process in
parallel
• Linear scalability by adding nodes

5. Pivotal Big Data Technology: Pivotal Greenplum & HDB
Think of it as multiple PostgreSQL servers
Segments (Workers)
Master
Rows are distributed across segments by a particular field (or randomly)

6. Pivotal Greenplum
Fundamental Concepts

7. Shared-Nothing Massively Parallel
Processing Architecture
External
Sources
Loading,
streaming
Segment
Servers
Query processing
and data storage
Interconnect
Master Servers
Query planning
and dispatch

8. Key Features and Benefits of Pivotal Greenplum
PRODUCT
FEATURES
CLIENT ACCESS
AND TOOLS
CORE MPP
ARCHITECTURE
GPDB ADAPTIVE
SERVICES
Workload Management
CLIENT ACCESS
ODBC, JDBC, OLEDB,
MapReduce, etc.
3rd PARTY TOOLS
BI Tools, ETL Tools
Data Mining, etc.
ADMIN TOOLS
Command Center
Package Manager
LOADING AND
EXTERNAL ACCESS
Petabyte-Scale Loading
Hadoop Integration
Trickle Micro-Batching
Anywhere Data Access
STORAGE/DATA ACCESS
Hybrid Storage and
Execution
In-Database Compression
Multi-Level Partitioning
Indexes – B-tree, Bitmap
External Table Support
LANGUAGE SUPPORT
Comprehensive SQL
Native MapReduce
SQL 2003 OLAP
Extensions
Programmable Analytics
Package Support
Multi-Level Fault
Tolerance
Online System
Expansion
Shared-Nothing MPP
Parallel Query Optimizer
Polymorphic Data Storage™
Parallel Dataflow Engine
gNet™ Software Interconnect
MPP Scatter/Gather Streaming™

9. Benefits of Greenplum
Faster
performance
Analytics where
the data lives
Flexibility and
control
Centralized
management
Enterprise class
reliability
Linear
scalability

10. Airline Optimization Use Case

11. Revenue Management
Solution for Airlines
Industry
Data Analytics Training
Machine Learning, Optimization, MADlib, PL/R

12. Revenue Management
What is Revenue Management ?
• Charge different prices to different segments to maximize revenue
When is it used?
• There is a fixed amount of resources available for sale
• The resources to sell are perishable
• Customers are willing to pay a different price for using the same resources.
What is the current state of Revenue Management?
• Revenue Management is in use since early 1980s
• Technology is dispersed
• Competitor prices are now available
What we will demonstrate
• We wanted to combine machine learning and optimization in the same flow to
highlight the benefits of having a single address for analytics.
• Highlight two technologies: MADlib and PL/R

13. Data Generation - Airports
We imported airport information from Wikipedia

14. Route Creation: Hub & Spoke Model

15. Route Creation: Hub & Spoke Model

16. Route Creation: Point to Point Model

17. Data Generation
Airports:
• Serve only airports with annual enplanement
larger than 1 M
• 65 Airports
Routes:
• Picked the Hub & Spoke Model
• From any airport to any other airport through one
of the 5 hubs
• From/to any airport to/from any hubs non-stop
routes
• No two connection routes allowed!
• 4180 Routes (640 are nonstop others through a
hub)
Daily Flights:
• Up to 5 times for a flight
• 2200 flights/day

18. How realistic is 2200 flights/day?
According to Bureau of Transportation Statistics the average number of
flights per day in 2011 by Airlines
• American Airlines ~1,500
• Delta ~2000
• JetBlue ~ 600
• SouthWest ~ 3200
• United ~ 910
Note: Numbers presented here are averages. There are small seasonal deviations.

19. Data Generation
Sales History:
• Close to two years of history
• Own Price + Competitors Prices
• Flight Date, Month, Weekday, Holiday Indicators
• Flights are available starting only 20 days before the flight date
• Sales for each sales date is captured.
• 3 Classes: First Class, Business, and Economy
4180
Routes
x
665
Flight
Dates
Up to 5
Flight
Times
20
Sales
Dates
3
Classes
x x x
> 500 Million observations ( > 150 GB)
Not big data but big enough that will not to fit in memory
Note: With networks data grows exponentially! Remember we started with only 65 airports.

20. Problem Statement
• Decision Variables
pt
where t is Number of Days to Flight
• Assume linear relationship between Demand and Price
– Dt
(pt
) is demand on day t when price is set to pt
• Demand depends also on
– Competitor Prices
– Trend
– Seasonality
▪ Day of the Week, Month, Holiday Indicator
Once you determine D(pi
)s - (ai
pi
+bi
) the problem is a Quadratic
Programming problem with Linear Constraints
Max Σi
D(pi
) pi
st. Σi
D(pi
) <= Capacity, D(pi
) >= 0

21. Linear Regression: Other possible features
Own Price
Comp 1 Price
Comp 2 Price
Comp 3 Price
Comp 4 Price
Price Gaps
Minimum Price
Month
Week day
Holiday Ind
Individual
Holidays
Linear Trend
Projected BTS
Trends
Unemployment
Consumer
Confidence
Long Overlays
Short Connections
No of Connections
Distribution
Frequent Flyer
Promos
Delay Statistics
Weather
School Schedules
Marketing Spend
Season

22. Solution
Sales History
~500 Million
Model Results
627 K Models
MADlib
To Be Priced Routes
86 Million
Scoring
86 Million
Input for QP
4.3 Million
Optimal Prices
4.3*20 Million
PL/R
SQL+MADlib
SQL
~60 secs
~13 secs
~10 secs
~45 secs
• Get insight from sales history
• Optimize the pricing decisions
for 4.3 Million flights
Step 1: Linear Regression Step 2: Scoring Step 3: Aggregate Step 4: Optimize
• Approximately 2 Minutes
• With 4 Select Statements
• Without data ever leaving the DB

23. Next Steps to Continue your Learning Journey
• Download the Greenplum Sandbox with MADlib
– http://greenplum.org/
• Learn more about Pivotal Data Science
– https://content.pivotal.io/data-science
• Pivotal Academy
– http://academy.pivotal.io

24. Thank You.