I studied global box office of major movies, and analyzed the key attributes to be considered in Oscar nominations. I acquired the dataset of IMDB 5000 from kaggle.com, the movies and ratings from grouplens, and the Academy Award records. The data set size is around 510MB. I predicted the global box office by using Linear Regression model and Gradient Boosting Tree model. For Oscar nominations, I used Decision Tree and Logistic Regression as classification methods. I ran these models on both Azure ML and Spark ML, then compared them using cross validation to find out which algorithm would produce the most accurate forecast.

1. Trend Forecasting of
Movie Box Office & Oscar Nominations
Group A
Tianyi Deng, Biran Li, Jiyang Liu, Yu-lin Shih

2. Agenda
● Hardware & Tools
● Data Size and Sources
● Movie Box Office Forecast
● Oscar Nominations Forecast
● Conclusion & Observations

3. Hardware
● Databricks File System
● Spark 2.1 Scala 2.10
● Memory - 6GB
● IBM data science experience
● Cores: 0.88
● 1 Driver, 1 DBU

4. Tools
● Azure Machine Learning
● IBM Data Science Experiences
● DataBricks
● Jupyter(python)
● Spark(python)
● Github

5. Data Size: 510 MB
Data Source:
GroupLens Research for Movielens Lab Dataset:
https://grouplens.org/datasets/movielens/
IMDB 5000 Movie Dataset:
https://www.kaggle.com/deepmatrix/imdb-5000-movie-dataset
The Academy Awards(1927-2015) Dataset:
https://www.kaggle.com/theacademy/academy-awards

6. Movie Box Office Forecast
Algorithm
● Linear Regression(LR)
● Gradient boost regression(GBT)
Platform
● Jupyter Notebook
● AzureML, Databricks, IBM

7. Data preparation
Jupyter Notebook
Replace column from word to integer
Change and add columns

8. Data preparation AzureML
Upload file to AzureML
Join Data

9. New CSV file

10. Data Understanding
●Target column: globalgross
●min/max: $423/$2,783,918,982
●Numeric Features: “budget, domesticgross, globalgross,
duration, imdb_score”
● Categorical Features: “language, country, majorgenres”

11. AzureML - Regression
Algorithms: Linear Regression(LR), Gradient Boost Tree
Regression(GBT)
Split: 0.7/0.3
Coefficient of Determination: 90.6%

12. AzureML - Evaluation
Linear Regression Gradient Boosted Regression

13. Importance Scores of Feature Variables

14. Importance Scores of Feature Variables Cont.
Global Gross vs. Domestic Gross Global Gross vs. Budget Global Gross vs. Movie Rating

15. Azure ML - Mapping

16. Spark Machine Learning
IBM: Linear Regression
DATABRICKS: Gradient Boosted Trees Regression

17. Spark ML(Load & Prepare Data)

18. Spark ML(Split Data)
Linear Regression in IBM
GBT in Databricks

19. Spark ML(Pipeline)
Linear Regression in IBM
GBT in Databricks

20. Spark ML(Train)
Linear Regression in IBM
GBT in Databricks
● ParamGrid, Cross Validation

21. Spark ML(Test - Linear Regression)

22. Spark ML(Test - Linear Regression)

23. Spark ML(Test - GBT Regression)

24. Spark ML(Test - GBT Regression)

25. Spark ML (RMSE)
Linear Regression in IBM
GBT in Databricks
RMSE: 52,699,899.3262
RMSE:60,409,700.0000

26. AzureML vs SparkML
RMSE AzureML SparkML
Linear Regression 52,050,094.7637 57,188,471.1788
GBT Regression 63,194,986.7556 60,409,700.0000
Movie Box Office Forecast
The RMSE is big but reasonable, as the max movie box
office is around 3 billion, which makes it 1.6% of the label.

27. Oscar Best Picture Forecast
Classification
Decision Tree Logistic
Regression

28. Data preparation

29. Preprocessed data

30. Azure(Mapping)

31. Azure(Evaluate)
Two-Class Decision Tree Two-Class Logistic Regression

32. Azure(Score)

33. Preprocessed data
Features
★ Gross
★ IMDB_score
★ Director_facebook_likes

34. Databricks(Load & Prepare Data)

35. Databricks(Split Data)

36. Databricks(Pipeline)

37. Databricks(Train & Test)

38. Databricks(Accuracy)

39. Conclusion
Movie Box Office
●Linear regression is better as it has lower RMSE
Oscar Nominations
●both models produced similar precision and accuracy rate

40. Observation
●Azure is better for cleaning missing data, has many built-in
graphics, and is easier to use.
●Spark(DB and IBM) is better for advanced graphics.
●Databricks has faster processing time.
●IBM provides detailed error log.

41. Github Link
https://github.com/biranlicis/Movie-Box-Office-Machine-Learning

42. References1. Microsoft DAT202.3x Implementing Predictive Analytics with Spark in Azure
HDInsight
2. Microsoft's DAT203x, Data Science and Machine Learning Essentials
3. Gautam, Geetika, and Divakar Yadav. Sentiment Analysis of Twitter Data
Using Machine Learning Approaches and Semantic Analysis - IEEE Xplore
Document. N.p., n.d. Web. 02 May 2017.
4. "MovieLens." GroupLens. N.p., 18 Oct. 2016. Web. 02 May 2017.
5. Concetta A., and Constanc H. McLaren. "Movie Data." Journal of Statistics
Education, V17n1. N.p., 2009. Web. 02 May 2017.
6. Chuansun7. "IMDB 5000 Movie Dataset." IMDB 5000 Movie Dataset | Kaggle.
N.p., n.d. Web. 02 May 2017.
7. "The Academy Awards, 1927-2015." Academy of Motion Picture Arts and
Sciences, n.d. Web. 02 May 2017.

43. Thank you
Any questions?