Recommended
More Related Content
What's hot
What's hot (20)
Similar to Language Empowered Recommendations
Similar to Language Empowered Recommendations (20)
Recently uploaded
Recently uploaded (20)
Language Empowered Recommendations
- 1. DataScience@SMU Language Empowered Recommendations Albert Asuncion, Peter Kouvaris, Ekaterina Pirogova, Hari Sanadhya and Arun Rajagopal 1
- 3. DataScience@SMU Recommendation Systems Types 3 Earliest Modern New generation Item, User, Rating Items, Users, Context, Time, Location, Rating Algorithm Recommendation
- 4. DataScience@SMU Recommendation Systems Types 4 Earliest Modern New generation + Item, User, Rating Items, Users, Context, Time, Location, Rating Everything in modern + Reviews text data Algorithm Recommendation Algorithm Recommendation
- 5. DataScience@SMU Not All Recommendation Data is Equal 5
- 6. DataScience@SMU Yelp – Some Data Limitations 6 Most of non- mobile app users are not registered Recommendation success/failure is difficult to measure User time of visit is unknown Ideal data structure -?
- 7. DataScience@SMU Yelp's Primary Data Point - The Review 7
- 8. DataScience@SMU Yelp Star Rating What Does it Mean? • Ambiguity • A 2-star rating is greater than 1-star, but by how much? 8
- 9. DataScience@SMU Star Bias 9 Current Stars Distribution Ideal Shape
- 10. DataScience@SMU But What If We Could Create a Super Star? 10
- 11. DataScience@SMU The Information Within Text 11
- 12. DataScience@SMU 12 I. Make ordinal star ratings more meaningful II. Improve the modern generation recommendation system with the value from the text Objectives
- 13. DataScience@SMU High Level View Of Our Strategy 13
- 14. DataScience@SMU Our Super Star Factory 14
- 16. DataScience@SMU How We Measure Each Method 16
- 17. DataScience@SMU How We Measure Each Method 17
- 18. DataScience@SMU Overview of Our New Star Results 18 Basic Stars
- 19. DataScience@SMU Overview of Super Star Results 19 Basic Stars Sentiment Stars
- 20. DataScience@SMU Overview of Super Star Results 20 Basic Stars Sentiment Stars Deep Learning Stars
- 21. DataScience@SMU How to Think About The Deep Learning Approach 21
- 22. DataScience@SMU Recommendation with Super Stars 22 Basic NLP 0.4307 0.5289 *22% FCP improvement (Fraction of Concordant Pairs)
- 23. DataScience@SMU Where Yelp's Recommendations Are Today • Yelp aims for scalable solutions. 23
- 24. DataScience@SMU Limitations of this Approach • Spark API is limited. Must remain distributed. • Preprocessing spam is too harsh 24
- 25. DataScience@SMU Why this Way? Yelp's Incentive: 25 Revenue Sources (2017)
- 26. DataScience@SMU Features in 3rd Generation Recommendations 26 New Generation Data Example for Random User
- 27. DataScience@SMU Methods in 3rd Generation Recommendations 27 New generation + Algorithm Recommendation • Text processing • Image processing • Feature creation
- 28. DataScience@SMU Thinking About Recommendation and Ethics Search rank in recommendation systems becomes economically valuable as the product gains popularity. 28
- 29. DataScience@SMU Lawsuits Due to Bad Reviews 29
- 30. DataScience@SMU The Effects of Unsafe Recommendation Products • Attackers gain the economic benefit. • Good products or companies lose market share. 30 Are users’ livelihoods directly tied to our system?
- 31. DataScience@SMU Privacy for Users 31 • Anonymity is important in cases where recommendations hold value. • e.g. Businesses pursuing positive reviews
- 32. DataScience@SMU Weaknesses with Reliance on Text Data Adversarial agents that were trained in the same way can get to the top! 32
- 33. DataScience@SMU Conclusions • The use of text data makes star ratings more meaningful. • The text of review improves the recommendation system quality, even with small sample sizes. • Fake reviews penalty is too harsh in Yelp system. 33
Editor's Notes
- Recommendations are ubiquitous in today's technology products. They save us time so we don’t have to continuously search. On LinkedIn we are recommended to connect with individuals closely related to us and on YouTube we see an endless array of quality data science videos. On Amazon we are recommended products we may have never even heard of but would like to buy. Yelp we are shown popular restaurants we may not have heard of. Let's take a look at how these systems are built today.
- What started out simple as a simple item, user, and rating algorithm has now evolved. In the current age, these systems have evolved to include not just this recommendation layer but also preprocessing on inputs so that the recommendations can leverage all new types of data. New data points are now included to add value.
- As data science methods related to natural language processing evolve, a new layer is being introduced to this stack. The power of language is being added in what experts are calling an ontology model, where a user is no longer just a rating tied to an item, but instead a complex web of interests and opinions.
- From this added complexity we can conclude that while data may be valuable for a business purpose, not all of these valuable data points naturally translate over to being used in recommendations. We want to know our data is representative and can be trusted for these algorithms!
- Looking at our Venn Diagram, Yelp is missing a majority of these dimensions. Most non-mobile users are not registered, so no additional data is captured. Measuring the success and failure is generally difficult,,, and finally we rarely know when a user visited. __________________ 1. No need to have user logged in to view the ratings and reviews. This makes most visitors of yelp as unregistered users thus making personalized user based recommendations less useful. 2. Without user information who was provided the recommendation, it's not possible to verify if the user actually went to the restaurant recommended and get users feedback on the same. If this was possible then the accuracy of recommendation system could have been improved over time. 3. Reviewers time of visit to the business is not recorded in the system making the calculation of delay in time elapsed between the visit to the restaurant and review writing time unavailable. This information would have been critical as its more likely as the more is this delay, more chances that the rating provided by user would be a little lower. "Almost everyone remembers negative things more strongly and in more detail - Clifford Nass, a professor of communication at Stanford University".
- Yelp's primary data comes in the form of a business review like this. User's post these with a star rating, review text, and can add images or check-in if they want. There also is some meta-data about the actual reviews. For recommendations, the 1 to 5 star rating seems like the clear starting point.
- The Yelp the star rating is an ordinal data point. It is naturally ambiguous, where we can’t be sure exactly how different each rank is from the other. This distance is key, because the algorithms that allow us to provide useful recommendations will leverage this distance in a very literal mathematical way.
- This ambiguous nature of star ratings is clear with a closer look. We noted that star distributions were heavily skewed, which, becomes a large issue when training the recommender algorithm. A better shape would be what we see on the right, a distribution based around an average experience.
- So what if we can transform these stars into something better? Infusing each with the text of a review, perhaps we can achieve a more normal distribution and then increase the recommendations our model gives off. _________________________________ Recommendation algorithms at a high level provide you with information based on others who look similarly. If we cannot clearly pin down what is bad, average, and great for a user in a clean numeric format then we are not leveraging these tools as best we could. Our data looks like the left here out of the box, but what we would actually prefer is something closer to the right. So what if we could create this better star rating? We would just want something that has this clear information in a numeric format and our recommendations would benefit heavily, increasing the relevance of what were before noise that was confusing our algorithm. Our recommendations will greatly benefit from this data! https://docs.google.com/drawings/d/1bLuT9jKVSbOXag8GFGljIWhTSUHFeR6m7bKhvVMSV7o/edit?usp=sharing
- The information in the text of a review is very personal. Different individuals use different word choices, word order, grammar, and more. Using these pieces of information we will attempt to redistribute stars so that they hold more information for our recommender models.
- Our strategy, at a high level, is to focus our efforts on the star creation aspect of the pipeline, taking the yelp data and creating new stars. We then will build recommendation models and review the effect of these stars to get a sense of the functionality added by our methods. https://docs.google.com/drawings/d/1NV41F5ZzstL9TcgRneOyTnQNcnTV5dBfrjiT0VbeFkA/edit?usp=sharing
- Our star factory has two sides. One neural network learns the normal review style for each business, and then another learns it for each user. The combined star output considers text regarding both entities. Our final result is therefore a new star rating. https://docs.google.com/drawings/d/131waFzcnjVKWB8CWjPlOG_jQNOZQiGgjT6RkBbSrxQw/edit?usp=sharing
- Building a recommendation algorithm and making sure that the format does not distort the results is key. We used an algorithm in the same family as Yelp's called the Collaborative Filtering Algorithm family. These methods will make much better use of our adjusted stars compared to the previous type. Drawings https://docs.google.com/drawings/d/1tBXzUZKp_1Item7H6sb3TRpWPXC4rLrwXe_6rxn5aZo/edit?usp=sharing
- We use measurement to verify if our methods will provide an improvement to the final recommendation model. To achieve this we look at the adjusted star distribution on a very large level, simply a histogram of all the new star ratings. https://docs.google.com/drawings/d/1TzaYyh4KV2ywfXKenQH00MIa1mEI2uuITfr-ZF50tq8/edit?usp=sharing
- We also measure the skewness of all user star distributions to ensure its having an effect across all our users. We will explain this in more detail in a moment, but you can think of our second metric as a way to average all the histograms for each user's specific star distribution. https://docs.google.com/drawings/d/1TzaYyh4KV2ywfXKenQH00MIa1mEI2uuITfr-ZF50tq8/edit?usp=sharing
- Our basic star method is, as we would expect, weak along the dimensions we are using to measure success. It's distribution is heavily skewed, and the tails for the skew plot extend from –3 to 2. Simply put, our standard format isn’t ideal for recommendation.
- If we use a basic sentiment score as our new stars, one of the simpler NLP applications, we notice a change in the overall distribution. Looking at the skewness of users as a whole however, we note that the user skewness plot is almost an exact match. This means that our method reshapes the stars overall but does not help user distributions.
- Our final combined star method achieves both of our goals, producing a normal distribution on the top plot and reducing the tails of our users' skewness by a fair margin. ________________ Looking at the plot on the bottom right, we see the tails have shortened as well. This highlights that our user's star data now looks closer to a bell curve on average, and we have begun to successfully change star ratings on the user level, adding a great depth of information.
- ???? Change the New label to Super Star The best way to think about our Deep Learning approach is to understand the basic goal of the architecture. The goal is to learn what a user's and business' historic reviews look like, then analyze the current text and use that respective value to adjust the star rating. __________________ In this manner derivations from normal behavior, like a very strict user giving a positive and flowery review for the first time will be noted as very important compared to the same review text from another user who is generally very positive. https://docs.google.com/drawings/d/10JTmP1x3o3Cq1yFVtayavczlcE8YmRCwzz_J2xfEJIs/edit?usp=sharing
- Our new stars are able to perform well on an out of sample dataset using the FCP score, a common metric for these methods where higher is better, although the ultimate validation for this method would have to be delivering users our recommendations via a product and reviewing the response. __________________ In this manner derivations from normal behavior, like a very strict user giving a positive and flowery review for the first time will be noted as very important compared to the same review text from another user who is generally very positive. https://docs.google.com/drawings/d/10JTmP1x3o3Cq1yFVtayavczlcE8YmRCwzz_J2xfEJIs/edit?usp=sharing
- Let's compare where Yelp is today. Why aren't they doing this? Yelp is implements their recommendation model in Spark, sandwiching it between two layers of filtering and machine learning. Within these layers they use things like NLP and Deep Learning to remove bad (maybe “fake” is better word?) reviews. https://docs.google.com/drawings/d/1IGARr_gqf1CMmqGEfoHrs3sKxkn_a45bbWhCGL_hpDM/edit?usp=sharing
- Yelp's implementation has a few limitations. The Spark API while scalable, is difficult to extend. Yelp's preprocessing is often considered clunky, where reviews that are valid and useful are randomly discarded. This can cascade into some odd suggestions for users. https://docs.google.com/drawings/d/1BuM0HodwdqTQpdMcnZi9S8DejneYFqZnITRaj2tQ4VU/edit?usp=sharing
- While Yelp is focused on providing a valuable service, it's incentive is somewhat contradictory to providing the best recommendations. Making most of its revenue from advertising businesses near the top of search results; adjusting what is shown to users would damage revenues. Should recommendation results be mixed between paid and unpaid? Ethics discussion. http://www.yelp-ir.com/news-releases/news-release-details/yelp-reports-fourth-quarter-and-full-year-2017-financial-results
- New methods extend the data past just what's in the data set
- 3rd generation recommender systems are not just about the collaborative filtering algorithm anymore. Building additional components to amplify accuracy or increase robustness against noise is the key to building these products in the coming age. As we have shown in this presentation so far, this algorithm layer of the recommendation stack is open for data scientists to do whatever they feel could improve the end result.
- Thinking about the results of a heavily relied upon recommendation system becomes important as the product grows. Google has had growing pains in this space as it grew important enough to cause damage, helping criminals finding protected persons or destroying businesses that relied on their search rank due to an unfair automated ban.
- These cascading effects can be determined by asking oneself what type of value does the platform deliver, and what additional benefit is there to being ranked near the top? For Yelp, the additional free publicity on one of the most trafficked restaurant review websites is highly valuable. Not only do these attackers benefit, but also good products and companies lose market share. We must ask ourselves, are user's livelihoods directly tied to our system?
- When these systems become economically important, privacy becomes an issue of user safety. If an Elite yelp user has heavy sway in the success of a business, what expectations are there of technologists to maintain anonymity for users without sacraficing quality.
- There are some weaknesses when relying on these methods. Adversarial agents that have been trained in a similar way can dupe our system and get ranked highly. Adding a large volume of fake reviews to a system like this would cause a large restructuring of the results delivered, which would have some cascading effects.
- 1. We can improve ordinal star ratings by translating them with text data to create a new star rating. 2. These new star ratings can them simply be passed to a Yelp recommendation algorithm and will test highly on out-off-sample data.