Recommended
More Related Content
Viewers also liked
Viewers also liked (15)
Similar to Dancing Machine to Doesn't Really Matter: Analysis of pop music
Similar to Dancing Machine to Doesn't Really Matter: Analysis of pop music (20)
Recently uploaded
Recently uploaded (20)
Dancing Machine to Doesn't Really Matter: Analysis of pop music
- 1. Cynthia Antonio Jayalakshmi Nair Reshma J Palangat Taylor Stevak
- 2. INTRODUCTION Originated in the 1950s Derived from “popular” music Mass audience appeal Catchy rhythm and lyrics Major form of entertainment for all ages Medium of expression
- 3. WE FOCUS ON Analyze lyrical patterns of hit pop songs From 1970 to 2009 Observe trends in terms of frequent words, artists, length of the track etc. Notice variations in styles
- 4. HOW DID WE DO THIS? Data from multiple sources Data summarization Text mining and modeling Combined into a large data set SAS Sentiment studio and modeling
- 6. BENEFITS Monetary paybacks – Higher salaries Profits – Chart topping songs Higher revenues – Increase knowledge Cultivate an identifiable trend of lyrics Recognitions, awards, competi tive advantage
- 7. COSTS No direct costs internet, software and other resources were provided by Oklahoma State University. Intangible costs – 100 labor hours
- 8. COSTS SOFTWARE TOOLS COST SAS® Enterprise Miner $ 25,570 Microsoft® Office $ 400 SAS® JMP $ 5,892 Tableau $ 1,999 IBM® Cognos $ 14,700 SAS® Enterprise Guide $ 2,035 Text Miner add-on $ 16,800
- 9. “It is a capital mistake to theorize before one has data.”
- 10. DATA PREPARATION Data access: COLUMN DESCRIPTION SOURCE Year of song release Position Artist Name http://www.bobborst.com/popculture/ top-100-songs-of-the-year/?year=1970 Song Name Lyrics http://www.metrolyrics.com/azlyrics.html Gender http://www.lyricsfreak.com/ http://en.wikipedia.org/wiki/ State http://en.wikipedia.org/wiki/ Length http://en.wikipedia.org/wiki/
- 11. DATA CONSOLIDATION 4 separate excel workbooks created with 201 rows each. Consolidate function on Excel was used to merge the datasets.
- 12. DATA CONSOLIDATION Preview of rows in an excel workbook Conversion of data Conversion to standard format Importing data into environment Steps involved in data consolidation
- 13. DATA CLEANING Unwanted ads between lyrics were discarded. E.g.: “www.metrolyrics.com” The excel workbook
- 14. DATA TRANSFORMATION Only adjectives, nouns and verbs were considered. A Synonym list was created to filter words with similar meanings.
- 15. DATA DICTIONARY Attribute Year Position Artist Song Gender Description Year of appearance on list Rank of song on list Name of the singer Title of the track Gender of the artist Field type Num(5) Num(3) VarChar(50) VarChar(100) Char(10) Source http://www.bobborst.com http://www.bobborst.com http://www.bobborst.com http://www.bobborst.com http://www.wikipedia.com Example 2004 4 Maroon 5 This love Male Lyrics State Lyrics of the song Name of the US state of origin, else NA Length of the track in seconds Specifies theme of the song as rap/religion/men/women VarChar(20000) Char(50) http://www.azlyrics.com http://www.wikipedia.com I was so high... California Num(10) http://www.wikipedia.com 207 Varchar(20) Manually coded Love Length Theme* * We used themes such as Happiness, Love, Heartbreak, Optimism etc. as a categorical variable to signify the theme of
- 16. DATA UNDERSTANDING Artists/groups by location California: 76 artists New York: 68 artists
- 17. DATA UNDERSTANDING Distribution of songs according to themes: Heartbreak: 188 Love: 162 Happiness: 89 Dance: 82 Sorrow: 81 Optimism: 54 Women: 46 Men: 13 Hate: 13 Religion: 8 Instrumental: 5
- 18. DATA UNDERSTANDING Average length of songs by year The average length of songs peaked between the late 80s to the 90s. The current trend is towards shorter songs.
- 19. DATA UNDERSTANDING Length versus position on chart Shortest song: 1:40 minutes Longest song: 8:57 Songs with lengths less than 2:30 minutes minutes and beyond 7:30 minutes never made it to the Overall Mean: 4:02
- 20. DATA UNDERSTANDING Gender – by decade For the years 1970 through 1979, 75.5% of the entries were by male singers and 24.5% were female singers. In the next decade from 1980 to 1989, the male entries reduced to 71% and female entries increased to 29%.
- 21. DATA UNDERSTANDING Gender – by decade During the 90s, there was a change in trends and almost equal entries were observed in both cases. The male entries dropped to 53% while the female entries rose to 47%. During the following decade spanning from 2000 to 2009, male entries
- 22. TEXT MINING PARTITION training: validation: testing 50:30:20 TEXT CLUSTERING TEXT PARSING TEXT FILTERING Flow diagram for Data preparation and modeling
- 23. MODELING Expectation maximization algorithm: 13 clusters Love-88 Rap- 31 Heartbreak- 59
- 24. TEXT MINING Hierarchical clustering: 30 clusters
- 25. MODELING Aim: Predictive modeling to predict themes Regression- Logistic regression with stepwise selection method Text topics as input variable Model Themes as target variable Misclassification rate Logistic regression 0.07625 (stepwise) Logistic Regression Average squared error 0.070996
- 26. MODELING Aim: Predictive modeling to predict themes Regression- Logistic regression with stepwise selection method Topics 2 and 5 were identified and considered significant inputs
- 27. SENTIMENT ANALYSIS Aim: Categorize the songs into positive and negative themes Three models were developed • Statistical • Rule-based • Hybrid
- 28. MODELING
- 29. MODELING A rule based model was built using specified rules by us and were classified into positive, negative and neutral categories.
- 30. MODELING Testing positive folder Statistical model Rule based model Hybrid model
- 31. MODELING Testing negative folder Statistical model Rule based model Hybrid model
- 32. MODELING Model Precision (%) Accuracy (%) Recall (%) Statistical 72 62 75 Rule-based 87 70 70 Hybrid 88 65 64 Precision, accuracy and recall for m
- 34. WORD CLOUD Word cloud depicting the most freque
- 35. CONCLUSIONS THEMES Themes of the 70’s (Heartbreak 18.5% Love 17.5% Sorrow 16.5%) Themes of the 80’s (Love 31% Heartbreak 18% Dance 11.5%)
- 36. CONCLUSIONS THEMES Themes of the 90’s (Heartbreak 36.5% Love 20% Dance 11%) Themes of the 00’s (Heartbreak and Rap 21% each; Love 12.5%)
- 38. CONCLUSIONS – POPULAR TRENDS 79 songs by the top ten most recurring artists were further analyzed. Six female entries and four male entries were observed. 47 songs were sung by females as opposed to 32 by males. Top themes were heartbreak (36.7%), love (22.8%) and
- 39. CONCLUSIONS – POPULAR TRENDS There was no pattern in the place of origin. 3 out of 10 entries belonged to countries other than the U.S. Most of these singers made it to the top hits in multiple years spanning decades.
- 40. CONCLUSIONS – POPULAR York 12 entries New TRENDS 1 11 entries Texas 2 3 9 entries Indiana 8 entries 4 8 entries California 7 entries 5 6 entries Texas 6 entries Pennsylvania 6 6 entries Michigan 6 entries Barbados
- 41. THANK YOU
Editor's Notes
- This is an interesting trend because there are more male singers than females in general according to our dataset.
- This is an interesting trend because there are more male singers than females in general according to our dataset.