ACMS TV Ratings Midterm Angelini

Television Show Cancelation Model
Purposeful Selection of Covariates
Brandon Angelini
Topics in Statistics – Logistic Modeling
ACMS 40950
March 4th
2016

Introduction
The model’s goal is to predict if a TV show will be canceled by observing a variety of
publically available information about the show. The data set includes a binary output of
cancelled or renewed, and has continuous covariates 18-49 demographic viewership,
previous year 18-49 demo viewership, overall viewership, previous year overall viewership,
as well as categorical variable Network (ABC, CBS, NBC, etc.), and binary variables
Scripted and Broadcast. The data is an aggregation of Nielsen TV ratings, as archived by
TVSeriesFinale.com, and compiled by the researcher (Brandon Angelini).
Data Set Variables:
Title – Show Title
Cancel – 1 if cancelled, 0 if renewed/other
ID – Arbitrary numerical ID assigned to show
Demo – Ratings for adults 18-49 demographic
PrevDemo – Ratings for previous year for adults 18-49 demographic (0 for new shows)
Viewers – overall viewership
PrevViewers – overall viewership for previous year (0 for new shows)
Scripted – 0 if unscripted show (reality), 1 if scripted show (drama, comedy, etc.)
Broadcast – 0 if cable, 1 if broadcast network (ABC, CBS, FOX, and NBC typically have
higher distribution because they’re broadcast networks)
Network – Network show airs on
ABC – 1
CBS – 2
CW – 3
FOX – 4
Freeform – 5
FX – 6
MTV – 7
NBC – 8
SyFy – 9
TNT – 10
Often the cancelling and renewing of TV shows is looked at as a black box in which TV
executives choose which shows live and die for a variety of reasons, but this model attempts
to simplify the cancelation decisions to ratings data, and the categories a show occupies.
Production companies, media companies, and advertisers could use the model to understand
how to allocate resources among shows, allowing them to predict what production decisions
executives may make.

Purposeful Selection of Covariates
Step 1:
Create a univariable logistic regression model for each covariate
Covariate Coeff. Std. Err. Odds Ratio G p
Demo -1.389 0.238 0.24935 46.57 0.00
PrevDemo -2.592 0.429 0.07487 91.10 0.000
Viewers -0.293 0.052 0.74602 42.92 0.000
PrevViewers -0.568 0.101 0.56666 87.32 0.00
Script 1.999 0.612 7.3817 17.51 0.001
Broadcast -0.013 0.310 0.98708 0.00179 0.966
as.factor(Network)2 -.581 0.427 0.55934 6.89 0.174
as.factor(Network)3 -0.899 0.812 0.40698 6.89 0.268
as.factor(Network)5 0.112 0.905 1.1185 6.89 0.901
All appear to be significant at this stage except for Broadcast, so a model will be fit with all
variables except ‘Broadcast’
Step 2:
Fit a multivariable model that contains all covariates that are significant in univariable
analysis at the 25% level.
Covariate Coeff. Std. Err. Z Pr(>|z|)
Demo -2.230 1.112 -2.006 0.04489*
PrevDemo -2.641 1.564 -1.689 0.09120.
Viewers -0.771 0.302 -2.554 0.01064*
PrevViewers 0.025 0.347 0.072 0.94257
Script 2.646 0.836 3.163 0.00156**
as.factor(Network)2 2.054 0.953 2.154 0.03121*
as.factor(Network)3 -6.661 1.331 -5.004 5.62e-07***
as.factor(Network)4 -1.636 0.837 -1.954 0.05073.
as.factor(Network)8 -0.730 0.6582 -1.110 0.26698
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
PrevDemo, PrevViewers, and Network 8 don’t appear to be significant, so we must use the
likelihood ratio test. For PrevDemo the significance is .0923 (>.05), PrevViewers is .94275
(>.05) so both should be excluded from the model at this point. Excluding Network results
in a significance of 1.589e-12, so Network be kept in the model.

The new reduced model doesn’t include data points for previous year (insignificant in
multivariable model in step 2) or if the show is on broadcast TV or not (insignificant in
univariable model is step 1).
Reduced Model
Demo -2.7532 0.8836 -3.116 0.00183**
Viewers -0.8748 0.2209 -3.959 7.51e-05***
Script 3.0433 0.7953 3.827 0.00013***
as.factor(Network)4 -2.1870 0.7657 -2.856 0.00429**
Step 3:
Check to see if covariates removed from the model in step 2 confound or are needed to
adjust the effects of the covariates remaining in the model.
With the removal of PrevDemo, coefficients for Demo (21%), Network 4 (33%), and
Network 8 (29%) change by >20%, and the removal of PrevViewers results in the
coefficients for Viewers (58%), and Script (24%) changing by >20%, meaning that both
PrevDemo and PrevViewers are adjusters, and should be left in the model. Additionally, it
makes sense intuitively that a show may not only be judged on that year’s performance, but
also on the previous year, so both will be kept in the model.
Step 4:
In univariable analysis (step 1) the Broadcast covariate was deemed to be insignificant
(p=0.966). The addition of Broadcast to the current model results in a small change to some
of the Network categorical coefficients, but because broadcast is a classification for network
variables (ABC, CBS, Fox, and NBC are ‘broadcast channels’, others are not) the effects of
Broadcast are likely captured through the inclusion of the network categorical variables, so
Broadcast will be left out of the model.
Preliminary Main Effects Model:
Demo -2.230 1.112 -2.006 0.04489*
PrevDemo -2.641 1.564 -1.689 0.09120.
Viewers -0.771 0.302 -2.554 0.01064*
PrevViewers 0.025 0.347 0.072 0.94257
Script 2.646 0.836 3.163 0.00156**

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Step 5:
Check the linearity of the remaining continuous covariates (Demo, PrevDemo, Viewers, and
PrevViewers) through the use of lowess plots, and account for any nonlinearlity.
Lowess Plots
Demo PrevDemo
Viewers PrevViewers
The lowess plots for all 4 continuous covariates appear to be non-linear, so it appears that it
will be necessary to use quartile design variables, fractional polynomials, or linear splines.
In this case, we’ll attempt to use linear splines to try to get the best possible fit, and test the
fit. Using linear splines resulted in a model that appeared to be over fit and yielded huge
standard errors, so we’ll try fractional polynomials instead.

Rejected linear splines model
Covariate Estimate Std. Err. Z value P values
Demo.splinesx.l1 -2.411e+15 9.919e+07 -24305729 <2e-16***
Demo.splinesx.l3 2.843e+15 4.082e+07 69649357 <2e-16***
PrevDemo.splinesx.l1 1.982e+15 7.734e+07 25626005 <2e-16***
PrevDemo.splinesx.l2 -8.001e+14 2.453e+07 -32612936 <2e-16***
Viewers.splinesx.l1 NA NA NA NA
Viewers.splinesx.l2 1.401e+13 8.712e+06 1608118 <2e-16***
Viewers.splinesx.l3 -7.136e+14 5.992e+06 -119077111 <2e-16***
Viewers.splinesx.l4 -8.728e+14 2.426e+07 -35975439 <2e-16***
PrevViewers.splinesx.l1 -1.852e+15 1.407e+08 -13159211 <2e-16***
PrevViewers.splinesx.l2 -1.942e+14 7.531e+06 -25786173 <2e-16***
PrevViewers.splinesx.l3 1.676e+14 6.477e+06 25883316 <2e-16***
PrevViewers.splinesx.l4 1.424e+15 2.930e+07 48600026 <2e-16***
Script 1.468e+15 1.268e+07 115751505 <2e-16***
as.factor(Network)2 9.463e+14 1.464e+07 64632365 <2e-16***
as.factor(Network)3 -2.331e+15 2.670e+07 -87309082 <2e-16***
To determine the fractional polynomials, we’ll attempt to fit using the mfp package in R
fit1 <-mfp(Cancel~ fp(Demo) + fp(PrevDemo) +fp(Viewers) +fp(PrevViewers) +Script
+as.factor(Network), family=binomial, data=data1, verbose=T)
Upon trying fractional polynomials, none of the 4 variables result in p-values that merit the
addition of any degree of fractional polynomial, so the continuous covariates could likely
remain assumed to be linear relationships (determined through G-Stat/p-value testing).
However, the plots appear not linear, so we’ll try adding 1 term fractional polynomials as a
sort of compromise between a more complex but statistically insignificant way, and the
linear. The output of the fractional polynomial analysis leads us to conclude p1 the value for
Demo=0.5, PrevDemo=1, Viewers=1, and PrevViewers=-2.
Covariate Estimate Std. Error z value Pr(>|z|)
Demo 7.4589 4.4412 1.679 0.093061.
DemoSQRT -23.7592 9.9072 -2.398 0.016477*
PrevDemo -0.1829 1.9413 -0.094 0.924937

Viewers 2.3239 1.3430 1.730 0.083550.
ViewersSQ -0.2950 0.1372 -2.151 0.031488*
PrevViewers -0.5409 0.5159 -1.048 0.294423
Script 19.1208 1137.0261 0.017 0.986583
as.factor(Network)3 -5.5146 1.6626 -3.317 0.000911***
as.factor(Network)5 -4.4888 1.7850 -2.515 0.011913*
as.factor(Network)7 -4.9793 1.8046 -2.759 0.005793**
The introduction of fractional polynomials has changed the coefficient on Script and
PrevDemo substantially, and made them statistically insignificant, so at this point it seems
rational to conclude that the addition of fractional polynomials is over complicating the
model. A linear splines model was attempted, but similar results occurred, so as a result,
while we’d like to account of the apparent non-linearity illustrated in the Lowess plots, we
cannot introduce a way of accounting for that without severely altering the model or
coefficients. The fractional polynomials will be removed from the model, and the
continuous covariates will remain assumed to be linear.
Step 6:
Explore possible interactions among main effects
15 models were fit with all combinations of interactions between covariates, and none were
significant at the 5% level. The highest significance was between Demo and Network [6.7%]
and between PrevViewers and Network [5.2%], but neither makes sense in context. If demo
viewership and network interact, that should happen every year, and we don’t see that for
network and PrevDemo (and we don’t see that for Viewers and Network like the interaction
between PrevViewers and Network would suggest). As a result of intuition and relatively
low significance levels, we won’t include any interactions in the model.
Preliminary final model
Demo -2.230 1.112 -2.006 0.04489*
PrevDemo -2.641 1.564 -1.689 0.09120.
Viewers -0.771 0.302 -2.554 0.01064*
PrevViewers 0.025 0.347 0.072 0.94257
Script 2.646 0.836 3.163 0.00156**

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Diagnostics and Results
Hosmer-Lemeshow Statistic
Test of goodness of fit based on fitted values and predicted values. The H-L stat has a null
hypothesis that the model fits, and alternative that there is a poor fit, and the null can be
rejected at typical p-value levels (5% typically).
H-L Values: X-squared = 4.2516, df = 8, p-value = 0.8337
Fail to reject the null, and conclude that there is adequate agreement between the fitted
values and the predicted values.
Cook’s Distances
A way to determine the effect on the coefficients created by “outlier” data points. Cook’s
distances are computed, a maximum distance is determined, and the relative effects on the
model’s coefficients are determined. If any data point has >20% effect, it is removed and the
model is fit again.
Observation 115 is the max, and upon removal none of the changes have the required 20%
effect, with the max being 3.93% for PrevViewers.
K-Folds Validation
To evaluate the generalizability of the model we’ll test it on a variety of subsets of the data.
To do so, we’ll start by splitting the data set into 5 subsets or ‘folds’, of sizes 56, 56, 56, 57,
and 57. The mean error rate over the five ‘folds’ is 0.3196115, which is acceptable for this
type of model.
Fold 1 [1:56] Fold 2 [57:112] Fold 3 [113:168] Fold 4 [169:225] Fold 5 [225:282]
0.3035714 0.1428571 0.6428571 0.3333333 0.1754386
AUC
Performing an Area Under the Curve analysis yields a value of 0.8687, which is very good
discrimination between subjects classified correctly and incorrectly.

Conclusions
The final model seems to describe the TV cancelations for this data set well, but the
model may not be generalizable, because there may be groupings to the data points that
weren’t taken into account. However, the favorable AUC and 5-fold validation are positive
signs that it may still discriminate well in outside data sets.
Potential issues with grouping in the model arise in that it may matter what other
shows are in competition in any given year, as it’s unclear if shows are fighting for a fixed
share of viewers in a year. It seems possible that shows have the potential to increase
viewership overall and fight for ratings independently of other shows, but more likely
ratings depend on other shows in the same year, and shows are fighting for a ‘slice of the
pie’ of viewers.
Additionally, I would’ve liked to add some type of variable for genre of a show, as
that’d help tie similarities between types of shows in addition to types of channels as the
categorical network variable does. Currently the scripted variable starts to do this, but there
are more classifications like ‘drama’ and ‘comedy’ that may have expected viewership
levels, that if they don’t achieve they’ll be canceled. Finally, show air time seems like it
would be an interesting statistic to keep on the shows, as it would help normalize ratings and
create a “good rating for that time” type of idea, so that shows that air at times that are
traditionally associated with lower viewership are evaluated properly.
Overall, the model fit is very successful, and has applications that could prove useful
to the general television production community.
Applications
The television cancelation model’s coefficients can help us draw some interesting
conclusions that come from the effects that a coefficient have in the model
(positive/negative, and overall magnitudes).
1. Scripted shows are more likely to be canceled than unscripted shows, as shown by
the positive coefficient on Script (2.64558). This could likely be related to the fact
that there are more scripted shows that air in a given year, but the reality remains,
scripted shows are more likely to be canceled.
2. Previous year demographic ratings are the most significant ratings effect on
cancelation (PrevDemo -2.64102), followed by current year demographic ratings
(Demo -2.22992), overall viewers (Viewers -0.77149) and previous year overall
viewers (PrevView 0.02498). This is interesting, as it points to executives caring
more about attracting the 18-49 demographic that advertisers often target than
overall viewership. This could have broader implications for television, and provide
reasoning for the common criticism of television that ‘minorities are
underrepresented’ by showing that studios have to appeal to the majority voices in
the 18-49 demographic, as dictated by advertisers.
3. It matters what network a show airs on. Some shows may be able to avoid
cancelation by airing on a specific network, as it appears each network has it’s own
standards for ratings resulting in cancelation, shown by the significance of the
network categorical variable.

Appendix
Load Data
data1 <- read.delim(file.choose(),header=T)
attach(data1)
Final Model
model <- glm(Cancel~Demo+PrevDemo+Viewers+PrevViewers+
Script+as.factor(Network),family=binomial)
summary(model)
Step 1:
mod.demo <-glm(Cancel~Demo,family=binomial)
mod.prevdemo <-glm(Cancel~PrevDemo,family=binomial)
mod.viewers <-glm(Cancel~Viewers,family=binomial)
mod.prevviewers <-glm(Cancel~PrevViewers,family=binomial)
mod.script <-glm(Cancel~Script,family=binomial)
mod.broadcast <-glm(Cancel~Broadcast,family=binomial)
mod.network <-glm(Cancel~as.factor(Network),family=binomial)
#Examinep p-values
round(coef(summary(mod.demo)),3)
round(coef(summary(mod.prevdemo)),3)
round(coef(summary(mod.viewers)),3)
round(coef(summary(mod.prevviewers)),3)
round(coef(summary(mod.script)),3)
round(coef(summary(mod.broadcast)),3)
round(coef(summary(mod.network)),3)
G.demo <- mod.demo$null.deviance-mod.demo$deviance
G.prevdemo <- mod.prevdemo$null.deviance-mod.prevdemo$deviance
G.viewers <- mod.viewers$null.deviance-mod.viewers$deviance
G.prevviewers <- mod.prevviewers$null.deviance-mod.prevviewers$deviance
G.script <- mod.script$null.deviance-mod.script$deviance
G.broadcast <- mod.broadcast$null.deviance-mod.broadcast$deviance
G.network <- mod.network$null.deviance-mod.network$deviance
Step 2:
mod.1.reduce<- glm(Cancel~Demo+PrevDemo+Viewers+PrevViewers+Script,
family=binomial)
summary(mod.1.reduce)
Step 3:
mod.1.reduce<- glm(Cancel~Demo+Viewers+Script+as.factor(Network), family=binomial)
betas.withPrevDemo <- mod.1.reduce$coefficients[-c(1,4,7)]
betas.withPrevDemo <- mod.1.reduce$coefficients[-1]
mod.1.test<- glm(Cancel~Demo+PrevDemo+Viewers+Script+as.factor(Network),
family=binomial)

betas.withPrevDemo <- mod.1.test$coefficients[-1]
betas.wo.PrevDemo <-mod.1.reduce$coefficients[-1]
Step 5:
##taken from http://thestatsgeek.com/2014/09/13/checking-functional-form-in-logistic-
regression-using-loess/
logitloess <- function(x, y, s) {
logit <- function(pr) {
log(pr/(1-pr))
}
if (missing(s)) {
locspan <- 0.7
} else {
locspan <- s
}
loessfit <- predict(loess(y~x,span=locspan))
pi <- pmax(pmin(loessfit,0.9999),0.0001)
logitfitted <- logit(pi)
plot(x, logitfitted, ylab="logit")
}
logitloess(Demo,Cancel,0.8)
logitloess(PrevDemo,Cancel,0.8)
logitloess(Viewers,Cancel,0.8)
logitloess(PrevViewers,Cancel,0.8)
Linear Splines
#Failed method – not used in model
#Create knots for both demo and viewers
knotsdem <- c(.5,2.2,3.5)
knotsview <- c(.2,5.5,13)
Demo.splines <- my.4splines(Demo,knotsdemo)
Demo.splines <- my.4splines(Demo,knotsdem)
PrevDemo.splines <- my.4splines(PrevDemo,knotsdem)
Viewers.splines <- my.4splines(Viewers,knotsview)
PrevViewers.splines <- my.4splines(PrevViewers,knotsview)
mod.linearsplines <-
glm(Cancel~Demo.splines+PrevDemo.splines+Viewers.splines+PrevViewers.splines+Scrip
t+ as.factor(Network),family=binomial)
Fractional Polynomials
mod<- mfp(Cancel~fp(Demo)+fp(PrevDemo)+fp(Viewers)+fp(PrevViewers)+Script+
as.factor(Network), family=binomial, data=data1, verbose=T)
mod<- glm(Cancel~Demo+DemoSQRT+PrevDemo+Viewers+ViewersSQ+PrevViewers
+Script+ as.factor(Network), family=binomial)

Step 6:
#Fit Models for all possible interactions
mod.int01 <- glm(Cancel~Demo+PrevDemo+Viewers+PrevViewers+Script+
as.factor(Network)+Demo*PrevDemo, family=binomial)
as.factor(Network)+Demo*Viewers, family=binomial)
as.factor(Network)+Demo*PrevViewers, family=binomial)
as.factor(Network)+Demo*Script, family=binomial)
as.factor(Network)+Demo*as.factor(Network), family=binomial)
as.factor(Network)+PrevDemo*Viewers, family=binomial)
as.factor(Network)+PrevDemo*PrevViewers, family=binomial)
as.factor(Network)+PrevDemo*Script, family=binomial)
as.factor(Network)+PrevDemo*as.factor(Network), family=binomial)
as.factor(Network)+Viewers*PrevViewers, family=binomial)
as.factor(Network)+PrevDemo*as.factor(Network), family=binomial)
as.factor(Network)+Viewers*PrevViewers, family=binomial)
as.factor(Network)+Viewers*Script, family=binomial)
as.factor(Network)+Viewers*as.factor(Network), family=binomial)
as.factor(Network)+PrevViewers*Script, family=binomial)
as.factor(Network)+PrevViewers*as.factor(Network), family=binomial)
as.factor(Network)+Script*as.factor(Network), family=binomial)
coef(summary(mod.int01))[16,4]

Diagnostics
Hosmer-Lemeshow
#Perform Hosmer-Lemeshow (HL) test via R package ResourceSelection
require(ResourceSelection)
mod<- glm(Cancel~Demo+PrevDemo+Viewers+PrevViewers+Script+as.factor(Network),
family=binomial)
hoslem.test(Cancel,mod$fitted.values,g=10)
Cook’s distances
X <-cbind(rep(1,n),Demo,PrevDemo,Viewers,PrevViewers,Script,as.factor(Network))
p.hats <- mod$fitted.values[-1]
V <- diag(p.hats*(1-p.hats))
hs <- diag(sqrt(V)%*%X%*%solve(t(X)%*%V%*%X)%*%t(X)%*%sqrt(V))
rs <- (Cancel-p.hats)/sqrt(p.hats*(1-p.hats))
delta.chis <- rs^2/(1-hs)
#obtain delta betas (cook's distances)
delta.beta <- rs^2*hs/(1-hs)^2
##obtain delta deviances
#first obtain deviance residuals
ds <- resid(mod)
delta.D <- ds^2/(1-hs)
#Examine observations with large values of diagnostics which.max(delta.beta)
delta.beta[114]
#For some reason 114 identified 115 as max (somehow things got offset by 1)
mod2 <-glm(Cancel[-115]~Demo[-115]+PrevDemo[-115]+Viewers[-115]+PrevViewers[-
115]+Script[-115]+as.factor(Network)[-115], family=binomial)
100*(mod2$coefficients-mod$coefficients)/mod$coefficients
K-fold cross validation
##Create folds from original data
#First k=5 folds, i.e., cut X and y into fifths
X.f1 <- as.matrix(X[1:56,1:6])
X.f2 <- as.matrix(X[57:112,1:6])
X.f3 <- as.matrix(X[113:168,1:6])
X.f4 <- as.matrix(X[169:225,1:6])
X.f5 <- as.matrix(X[226:282,1:6])
y.f1 <- Cancel[1:56]
y.f2 <- Cancel[57:112]
y.f3 <- Cancel[113:168]
y.f4 <- Cancel[169:225]
y.f5 <- Cancel[226:282]
##Next, create training sets. When using Fold 1 as validation set (X.f1 and y.f1), then all
other folds combined are training set, and so on. X.t1 <- rbind(X.f2,X.f3,X.f4,X.f5) X.t2 <-
rbind(X.f1,X.f3,X.f4,X.f5) X.t3 <- rbind(X.f1,X.f2,X.f4,X.f5) X.t4 <-

rbind(X.f1,X.f2,X.f3,X.f5) X.t5 <- rbind(X.f1,X.f2,X.f3,X.f4) y.t1 <-c(y.f2,y.f3,y.f4,y.f5)
y.t2 <-c(y.f1,y.f3,y.f4,y.f5) y.t3 <-c(y.f1,y.f2,y.f4,y.f5) y.t4 <-c(y.f1,y.f2,y.f3,y.f5) y.t5 <-
c(y.f1,y.f2,y.f3,y.f4)
###Now, use each training set to fit a regression model and each Fold as a validation set,
recording the error rate each time
##Fold 1 as validation mod1<-glm(y.t1~X.t1,family=binomial)
#Compute fitted values for for validation set data uisng coefficients from training model
pi1 <- plogis(cbind(1,X.f1)%*%mod1$coefficients)
yhat1 <- round(pi1)
#Compute error rate, i.e., agreement between predictions and actual y values in validation
set
err.rate1 <- length(which (y.f1 != yhat1))/length(y.f1)
err.rate1
##Fold 2 as validation
mod2<-glm(y.t2~X.t2,family=binomial)
yhat2 <- round(pi2)
set
err.rate2
yhat3 <- round(pi3)
set
err.rate3
##Fold 4 as validation
mod4<-glm(y.t4~X.t4,family=binomial)
yhat4 <- round(pi4) #Compute error rate, i.e., agreement between predictions and actual y
values in validation set
err.rate4
yhat5 <- round(pi5) #Compute error rate, i.e., agreement between predictions and actual y
values in validation set
err.rate5 #compute mean error rate over five folds
mean(c(err.rate1,err.rate2,err.rate3,err.rate4,err.rate5))

ACMS TV Ratings Midterm Angelini

ACMS TV Ratings Midterm Angelini

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (10)

Similar to ACMS TV Ratings Midterm Angelini

Similar to ACMS TV Ratings Midterm Angelini (20)

ACMS TV Ratings Midterm Angelini