1. How Bellhops Leverages Amazon Redshift UDFs
for Massively Parallel Data Science
Ian Eaves, Bellhops
May 12th, 2016

2. Today’s Speakers
Chartio
AJ Welch
Chartio.com
Bellhops
Ian Eaves
GetBellhops.com
AWS
Brandon Chavis
aws.amazon.com

3. • The recording will be sent to all webinar participants after the event.
• Questions? Type them in the chat box & we will answer at the end
• Posting to social? Use #AWSandChartio
Housekeeping

4. Relational data warehouse
Massively parallel; Petabyte scale
Fully managed
HDD and SSD Platforms
$1,000/TB/Year; starts at $0.25/hour
Amazon
Redshift
What is Amazon Redshift?

5. Amazon Redshift is easy to use
• Provision in minutes
• Monitor query performance
• Point and click resize
• Built in security
• Automatic backups

6. Amazon Redshift System Architecture
10 GigE
(HPC)
Ingestion
Backup
Restore
JDBC/ODBC

7. The Amazon Redshift view of data warehousing
• 10x cheaper
• Easy to provision
• Higher DBA productivity
• 10x faster
• No programming
• Easily leverage BI tools, Hadoop,
Machine Learning, Streaming
• Analysis in-line with process
flows
• Pay as you go, grow as you need
• Managed availability & DR
Enterprise Big Data SaaS

8. The legacy view of data warehousing ...
Global 2,000 companies
Sell to central IT
Multi-year commitment
Multi-year deployments
Multi-million dollar deals

9. … Leads to dark data
This is a narrow view
Small companies also have big data
(mobile, social, gaming, adtech, IoT)
Long cycles, high costs, administrative
complexity all stifle innovation
0
200
400
600
800
1000
1200
Enterprise Data Data in Warehouse

10. New SQL Functions
We add SQL functions regularly to expand Amazon Redshift’s query capabilities
Added 25+ window and aggregate functions since launch, including:
• LISTAGG
• [APPROXIMATE] COUNT
• DROP IF EXISTS, CREATE IF NOT EXISTS
• REGEXP_SUBSTR, _COUNT, _INSTR, _REPLACE
• PERCENTILE_CONT, _DISC, MEDIAN
• PERCENT_RANK, RATIO_TO_REPORT
We’ll continue iterating but also want to enable you to write your own

11. Scalar User Defined Functions
You can write UDFs using Python 2.7
• Syntax is largely identical to PostgreSQL UDF syntax
• System and network calls within UDFs are prohibited
Comes with Pandas, NumPy, and SciPy pre-installed
• You’ll also be able import your own libraries for even more flexibility

12. Scalar UDF Example
CREATE FUNCTION f_hostname (VARCHAR url)
RETURNS varchar
IMMUTABLE AS $$
import urlparse
return urlparse.urlparse(url).hostname
$$ LANGUAGE plpythonu;

13. Analytics for Everyone
The best platform for everyone to explore and visualize data

14. The Smartest Companies Use Chartio

15. Legacy BI
• Expensive to set up
• Expensive to maintain
• Requires technical skills
• Creates a bottleneck
• Limits your ability to
make decisions

16. Modern BI
• Faster time to value
• Easier to maintain
• Modes for both technical
and non-technical users
• Alleviates bottlenecks
• Enhances your ability to
make decisions

17. Chartio’s Modern Architecture
Schema/
Business
Rules
Interactive
Mode
SQL Mode
Data Stores
TV Screens
Scheduled
Emails
Data
Exploration
Dashboards
Embedded
Data
Pipeline/
Data
Blending
Data Caching
Security

18. Chartio’s Modern Architecture
Schema/
Business
Rules
Interactive
Mode
SQL Mode
Data Stores
TV Screens
Scheduled
Emails
Data
Exploration
Dashboards
Embedded
Data
Pipeline/
Data
Blending
Data Caching
Security

19. Chartio’s Modern Architecture

20. The Chartio Schema Editor
• Team-specific
schemas
• Rename
tables/columns
• Hide tables/columns
• Define custom
tables/columns
• Define data types and
foreign keys

21. Schema Editor Live Demo

22. UDF’s (A Brave New World)
Using UDF’s in the Real World
Ian Eaves - Data Scientist

23. The Land Between SQL and Analysis

24. The Land Between SQL and Scripts
UDF

25. A Little About Bellhops
• On-demand moving and labor
company
• Self-scheduling capacity (a la Uber)
• Located in 83 markets

26. Herfindahl Index (H)
N = number of market actors
si = market share of ith actor

27. Herfindahl Index

28. Market Health Feedback Process
Monthly
Data Warehouse

29. Market Health Feedback Process
Monthly
Data Warehouse

30. Market Health Feedback Process
Monthly
UDF’s
Data Warehouse

31. Market Health Feedback Process
Monthly
Users
UDF’s
Data Warehouse

32. T-test UDF
create function f_t_test (val float, mean float, stddev float, n_samps float, alpha float)
returns varchar
stable
as $$
from scipy.stats import t
df = n_samps - 1
tval = (mean - val) / stddev
p = t.sf(abs(tval), df) * 2 # two sided
if p <= alpha:
return 'Better' if tval > 0 else 'Worse'
else:
return 'No Change'
$$LANGUAGE plpythonu;

33. Example Table Schema
market_month_dimension
———————————
id
date
herfindahl_index
~
~
~
market_fact
————————-
id
month_key
market_name
~
~

34. Our Query
WITH market_stats AS (
SELECT market_name, date, mmd.herfindahl_index,
avg(herfindahl_index) OVER (PARTITION BY market_name ORDER BY date ROWS BETWEEN 6 PRECEDING AND 1 PRECEDING) as avg,
stddev_samp(herfindahl_index) OVER (PARTITION BY market_name ORDER BY date ROWS BETWEEN 6 PRECEDING AND 1 PRECEDING) as
FROM market_fact
LEFT JOIN(
SELECT herfindahl_index, month_key as join_key, date
FROM market_month_dimension
) AS mmd
ON join_key = market_fact.month_key
GROUP BY market_name, date, month_key, mmd.herfindahl_index
ORDER BY date
)
SELECT market_name, date, herfindahl_index, avg, stddev, f_t_test(herfindahl_index, avg, stddev, 6, .2)
FROM market_stats
WHERE market_name = 'Atlanta'
ORDER BY market_name, date;

35. Sample Result

36. General Thoughts
UDF Use Cases

37. General Thoughts
UDF Use Cases
Complicated Conditional Logic

38. General Thoughts
UDF Use Cases
Complicated Conditional Logic
Text Processing

39. General Thoughts
UDF Use Cases
Complicated Conditional Logic
Text Processing
Basic Statistical Analysis

40. Thank You

41. Questions?
Chartio
AJ Welch
aj@chartio.com
Chartio.com
Bellhops
Ian Eaves
ian.eaves@getbellhops.com
GetBellhops.com
AWS
Brandon Chavis
chavisb@amazon.com
aws.amazon.com