1.
WIFI SSID:SparkAISummit | Password: UnifiedAnalytics

2.
Josh McNutt and Keria Bermúdez-Hernández
Showtime Networks Inc.
Data-Driven Transformation:
Leveraging Big Data at
SHOWTIME with Apache Spark
#UnifiedAnalytics #SparkAISummit

3.
Premium cable network known for bold and wholly
unique programming
In 2018 SHOWTIME had four of the top six scripted
hour-long series on premium cable: HOMELAND,
SHAMELESS, RAY DONOVAN and BILLIONS
Launch of SHOWTIME standalone streaming
service in July 2015 made available an
unprecedented level of user-level data
Showtime Networks Inc.

4.
- Viewing History
- Customer Journey Information
- CRM interactions
...
Billions of records
Extremely high level of granularity
Big Data at SHOWTIME
Direct relationship with our
customers
SHOWTIME standalone streaming service
Rich data captured on
every customer interaction:

5.
QuestionsBig Data at SHOWTIME

6.
What is the lifetime value of a subscriber who
signs up to watch Ray Donovan?
How many free trial signups have been
generated by Shameless?
What is the probability a subscriber will begin
watching Billions for the first time in the next 7
days?

7.
Questions
Capacity to Answer
Big Data at SHOWTIME

8.
Big Data at SHOWTIME
Strong data science skills needed to
interact directly with data lake
Research/Business
Analyst
Basic questions answered via
operational reporting
Complex questions required bespoke
analyses, sometimes taking weeks or
longer to develop
Raw
Data

9.
Questions
Capacity to Answer
How to reduce this
gap?
Big Data at SHOWTIME

10.
Small team of data scientists working to…
Democratize data and analytics
Understand and predict subscriber behaviors
Support data-driven programming & scheduling
Data Strategy Team

11.
We’re working to bring the data closer to the users and the users closer to the data
Business/Research
Analyst
Raw
Data
Augmenting Data
Supply Chain
Making the data
more suitable for
analysis
Democratizing Data and Analytics

12.
Master Viewers Table
Foundational customer data
representation
Captures 1000s of metrics and
behaviors in Free Trial, Paid Month 1,
lifetime
Tracks relationship between each
user and series
Supports several use cases:
• machine learning
• reporting/dashboarding
• ad hoc analysis
Democratizing Data and Analytics

13.
We’re working to bring the data closer to the users and the users closer to the data
Business/Research
Analyst
Raw
Data
Augmenting Data
Supply Chain
Data Skills
Training
Creating conditions
leading to a chain reaction
of curiosity, exploration,
analysis and insight
Making the data
more suitable for
analysis
Tableau
SQL
Democratizing Data and Analytics

14.
We build actionable machine learning models to gain an intimate understanding of
how our users behave and how to strengthen their relationship to SHOWTIME
XGBoost
Random
Forests
Churn Propensity
Resubscription Propensity
Future Customer Value
Series Viewership Propensity
$
Understanding and Predicting Subscriber Behaviors
Put into production:

15.
We are employing data and analytics to attribute revenue and subscribers to content
Free trial signups
Resubscribers
Reduction in Churn
among viewers
Renewal
Decisions
Schedule
Optimization
Subscriber
Growth
Revenue Growth
1 3 5 7 9 11
Churn % by Paid
Month
Mean subscriber
lifetime value (LTV)
Supporting Data-Driven Programming and Scheduling

16.
How to stitch
everything
together?
FLASHBACK: In order to build this capability, we needed to
get basic infrastructure in place

17.
• Team comprised exclusively of data scientists without a dedicated DevOps engineer
• Lot of time spent troubleshooting cluster configuration
• Launching clusters was slow and clumsy
• Debugging our bootstrap script (to install python libraries) was nontrivial
• Any software or hardware upgrades were scary and generally avoided for as long as
possible
#struggle
Configuration Challenges

18.
Back to focusing on data science!
• Booting and managing clusters is now easy, fast and
reliable
• Simple to attach/detach notebooks to clusters
• Databricks handles installation of python libraries
• Easy to connect data sources to Tableau
• We can confidently experiment with new software or
hardware in an effort to improve our workflows
Databricks Unified Analytics Platform

19.
Data Pipeline
19#UnifiedAnalytics #SparkAISummit

20.
Optimizing Pipeline
Solutions
• Apache Airflow
• Code optimization tactics:
– Replacing RDD and Pandas
transformations with Pyspark
dataframe transformations
– Multithreading using futures
module
20#UnifiedAnalytics #SparkAISummit

21.
Apache Airflow
• Produced different tables by using
dbutils.notebook API
– While technically you can run notebooks
concurrently, complex dependencies and
concurrent jobs are harder to do in a
single notebook and job
job1
job2
job3
job4
job5
job6
Workflow in a single Notebook Workflow in Apache Airflow
job1
job2
job3
job4
job5
job6
dbutils.notebook.run("notebook-name", 60,
{"argument": "data", "argument2": "data2",
...})
notebook_task = DatabricksSubmitRunOperator(
task_id='notebook_task’,
dag=dag,
json=notebook_task_params)
• Apache Airflow was the solution for
scheduling and managing pipelines
– Airflow and Databricks integration
– Manages dependencies
– Task job can be run in a different cluster

22.
Code Optimization Tactics
22
• User level RDD with subscriber status and viewership records
• Combinations of methods from custom classes

23.
Code Optimization Tactics
23

24.
Code Optimization Tactics
24

25.
Code Optimization Tactics
25
• To reduce tasks complexity we saved intermediate
tables

26.
Different Levels of Concurrency
26
BEFORE AFTER

27.
Tracking Data Quality
27#UnifiedAnalytics #SparkAISummit

28.
28#UnifiedAnalytics #SparkAISummit
Tracking Data Quality

29.
Using Delta to Update Tables
• Without Delta:
– Ingestion daily
data
– Updating fields
– Rewriting data
3.03
0.47
0
0.5
1
1.5
2
2.5
3
3.5
Without Delta With Delta
Hours
29#UnifiedAnalytics #SparkAISummit
MERGE INTO viewership AS hv
USING temp_tc AS t
ON hv.title = t.title
WHEN MATCHED AND hv.category IS NULL OR
hv.category != t.category THEN
UPDATE SET
hv.category = t.category
• With Delta
– Ingestion daily
data
– Updating fields
where necessary
– Rewriting data

30.
Optimization Summary
• Apache Airflow
• Code optimizations
• MLflow for tracking data quality
• Delta for updating tables
30#UnifiedAnalytics #SparkAISummit

31.
Unified platform allows us to democratize data and
analytics
We continue to engage in ongoing innovation and
optimization
Exciting cultural shift
Final Thoughts

32.
Questions?
32#UnifiedAnalytics #SparkAISummit

33.
DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT