bigdata_datascience_methodology
•
0 likes•58 views
Data science methodology document by Sitaraman Ramachandran dated July 31, 2015. The document likely outlines Ramachandran's approach and process for conducting data science projects and analyses. Without seeing the full document content, no other key details can be summarized in 3 sentences or less.
Report
Share
Report
Share
Download to read offline
Recommended
YELP Data Set Challenge
This document provides details about a project to analyze Yelp restaurant review data to identify factors that contribute to high restaurant ratings. It outlines the project objectives, data to be used, potential benefits, and limitations. Key aspects of the project plan include data cleaning, model building to predict highly rated restaurants, and delivering findings in multiple reports and presentations.
Eat, Rate, Love -- Presentation Slides v3
The document proposes adding a sub-rating to Yelp reviews to help users choose authentic ethnic restaurants. It analyzes adding a rating that gives more weight to reviewers who have reviewed other restaurants of the same cuisine ("Method 1") or a rating based on user names matching the cuisine's ethnicity ("Method 2"). Both methods reduced clustered ratings for Indian restaurants in various cities to 1 or a few options. Method 2 had a greater impact on ratings and was better at reducing options, so the document recommends it. It proposes adding an "Authenticity Badge" and filter for restaurants that score highly on the sub-rating.
Delivering Insights: Building the DataScience Web Application
DataScience presents insights to customers in an easily digestible, interactive format via a collaborative web application. This presentation outlines the technology behind the DataScience application, as well as future plans to enhance it.
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
The business cases for Hadoop can be made on the tremendous operational cost savings that it affords. But why stop there? The integration of R-powered analytics in Hadoop presents a totally new value proposition. Organizations can write R code and deploy it natively in Hadoop without data movement or the need to write their own MapReduce. Bringing R-powered predictive analytics into Hadoop will accelerate Hadoop’s value to organizations by allowing them to break through performance and scalability challenges and solve new analytic problems. Use all the data in Hadoop to discover more, grow more quickly, and operate more efficiently. Ask bigger questions. Ask new questions. Get better, faster results and share them.
Data mining on yelp dataset
This project involves predictive analysis on user reviews, review count, neighborhoods of businesses etc. and deriving star rating for different businesses in the YELP application.
Technologies used: Machine Learning Models (Navie Bayes, KNN, Decision Tree), Weka, Hadoop, Hive, Java
You can also find the blog at
https://sjsubigdata.wordpress.com/2014/12/19/data-mining-on-yelp-dataset-2/
Yelp final
This document provides an analysis of the online local business ratings and reviews industry, with a focus on Yelp. It discusses Yelp's business model, revenue streams, key activities, partnerships, and competitive advantages. Yelp connects people with local businesses by allowing users to write and read reviews. It makes money primarily from selling ads to local businesses. With over 100 million monthly users generating reviews, Yelp provides a valuable user-generated content resource for finding and evaluating local companies.
Yelp Academic Dataset
This document describes a project analyzing Yelp business data using an HDInsight Hadoop cluster on Azure. The project involves downloading Yelp data, converting it to CSV, loading it onto the cluster, and using HiveQL to query and visualize the data. Key aspects analyzed include business locations, categories, ratings over time, and reviews. Visualizations were created using PowerBI. The document outlines the cluster configuration, tools used, data processing flow, sample queries, and potential extensions like natural language processing.
Yelp Data Challenge - Discovering Latent Factors using Ratings and Reviews
A restaurant's average rating and reviews on Yelp in influence customers to an incredible degree. An extra half-star rating causes restaurants to sell out 19 percentage points (49%) more frequently. Despite the impact on the restaurant's business, achieving a better overall rating is not straightforward. A user may give only one star to the restaurant just because he or she found the quality of service to be abysmal even though the food and the restaurant's location were up to his or her standard. These facts may have been mentioned in the review in detail but the final rating would just reflect the poor quality of service. The user rating alone does not provide any additional details, and as a result, the restaurant may not be able to understand which aspects create a negative impact on user experience. Another case may be that a certain popular dish will make users give the restaurant five star ratings, but they would not be satisfied with another aspect of the restaurant such as the dessert. The high user ratings may hide the fact that some aspects of the user experience was negative and that the restaurant has room to improve. Traditional recommender systems usually use only the aggregated ratings without considering the hidden factors in the preference of the users and the properties of the restaurants. For the restaurant domain, this could mean main cuisine, dessert, service, staff friendliness, knowledge of staff, location, ambiance, price and many more aspects. Without considering the ratings for individual aspects, it is likely that the recommendation systems will give inaccurate predictions to restaurants as well as users.
In this project, we aim to uncover hidden details about the users' preferences with respect to restaurant properties. With this information, we can provide precise recommendations to the restaurants regarding what aspects they should concentrate on to improve user experience. Since we are backed by more meaningful information about users' preferences we can provide better recommendations to users as to which restaurants they would prefer and why. To summarize, from the results of this project, we can answer the following questions: "what does a particular user care about when dining from a restaurant?", "which aspect should the restaurant improve in order to effectively increase the rating?", and "which restaurant is the best for a particular user?"
Recommended
YELP Data Set Challenge
This document provides details about a project to analyze Yelp restaurant review data to identify factors that contribute to high restaurant ratings. It outlines the project objectives, data to be used, potential benefits, and limitations. Key aspects of the project plan include data cleaning, model building to predict highly rated restaurants, and delivering findings in multiple reports and presentations.
Eat, Rate, Love -- Presentation Slides v3
The document proposes adding a sub-rating to Yelp reviews to help users choose authentic ethnic restaurants. It analyzes adding a rating that gives more weight to reviewers who have reviewed other restaurants of the same cuisine ("Method 1") or a rating based on user names matching the cuisine's ethnicity ("Method 2"). Both methods reduced clustered ratings for Indian restaurants in various cities to 1 or a few options. Method 2 had a greater impact on ratings and was better at reducing options, so the document recommends it. It proposes adding an "Authenticity Badge" and filter for restaurants that score highly on the sub-rating.
Delivering Insights: Building the DataScience Web Application
DataScience presents insights to customers in an easily digestible, interactive format via a collaborative web application. This presentation outlines the technology behind the DataScience application, as well as future plans to enhance it.
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
The business cases for Hadoop can be made on the tremendous operational cost savings that it affords. But why stop there? The integration of R-powered analytics in Hadoop presents a totally new value proposition. Organizations can write R code and deploy it natively in Hadoop without data movement or the need to write their own MapReduce. Bringing R-powered predictive analytics into Hadoop will accelerate Hadoop’s value to organizations by allowing them to break through performance and scalability challenges and solve new analytic problems. Use all the data in Hadoop to discover more, grow more quickly, and operate more efficiently. Ask bigger questions. Ask new questions. Get better, faster results and share them.
Data mining on yelp dataset
This project involves predictive analysis on user reviews, review count, neighborhoods of businesses etc. and deriving star rating for different businesses in the YELP application.
Technologies used: Machine Learning Models (Navie Bayes, KNN, Decision Tree), Weka, Hadoop, Hive, Java
You can also find the blog at
https://sjsubigdata.wordpress.com/2014/12/19/data-mining-on-yelp-dataset-2/
Yelp final
This document provides an analysis of the online local business ratings and reviews industry, with a focus on Yelp. It discusses Yelp's business model, revenue streams, key activities, partnerships, and competitive advantages. Yelp connects people with local businesses by allowing users to write and read reviews. It makes money primarily from selling ads to local businesses. With over 100 million monthly users generating reviews, Yelp provides a valuable user-generated content resource for finding and evaluating local companies.
Yelp Academic Dataset
This document describes a project analyzing Yelp business data using an HDInsight Hadoop cluster on Azure. The project involves downloading Yelp data, converting it to CSV, loading it onto the cluster, and using HiveQL to query and visualize the data. Key aspects analyzed include business locations, categories, ratings over time, and reviews. Visualizations were created using PowerBI. The document outlines the cluster configuration, tools used, data processing flow, sample queries, and potential extensions like natural language processing.
Yelp Data Challenge - Discovering Latent Factors using Ratings and Reviews
A restaurant's average rating and reviews on Yelp in influence customers to an incredible degree. An extra half-star rating causes restaurants to sell out 19 percentage points (49%) more frequently. Despite the impact on the restaurant's business, achieving a better overall rating is not straightforward. A user may give only one star to the restaurant just because he or she found the quality of service to be abysmal even though the food and the restaurant's location were up to his or her standard. These facts may have been mentioned in the review in detail but the final rating would just reflect the poor quality of service. The user rating alone does not provide any additional details, and as a result, the restaurant may not be able to understand which aspects create a negative impact on user experience. Another case may be that a certain popular dish will make users give the restaurant five star ratings, but they would not be satisfied with another aspect of the restaurant such as the dessert. The high user ratings may hide the fact that some aspects of the user experience was negative and that the restaurant has room to improve. Traditional recommender systems usually use only the aggregated ratings without considering the hidden factors in the preference of the users and the properties of the restaurants. For the restaurant domain, this could mean main cuisine, dessert, service, staff friendliness, knowledge of staff, location, ambiance, price and many more aspects. Without considering the ratings for individual aspects, it is likely that the recommendation systems will give inaccurate predictions to restaurants as well as users.
In this project, we aim to uncover hidden details about the users' preferences with respect to restaurant properties. With this information, we can provide precise recommendations to the restaurants regarding what aspects they should concentrate on to improve user experience. Since we are backed by more meaningful information about users' preferences we can provide better recommendations to users as to which restaurants they would prefer and why. To summarize, from the results of this project, we can answer the following questions: "what does a particular user care about when dining from a restaurant?", "which aspect should the restaurant improve in order to effectively increase the rating?", and "which restaurant is the best for a particular user?"
Integrating R & Hadoop - Text Mining & Sentiment Analysis
The document discusses integrating R and Hadoop for big data analytics. It notes that existing statistical applications like R are incapable of handling big data, while data management tools lack analytical capabilities. Integrating R with Hadoop bridges this gap by leveraging R's analytics and statistics functionality with Hadoop's ability to process and store distributed data. RHadoop is introduced as an open source project that allows R programmers to directly use MapReduce functionality in R code. Specific RHadoop packages like rhdfs and rmr2 are described that enable interacting with HDFS and performing statistical analysis via MapReduce on Hadoop clusters. Text analytics use cases with R and Hadoop like sentiment analysis are also briefly outlined.
TextMining with R
Goal: Demonstrate popular practices when mining big dissimilar texts
Object of mining: texts from site: http://blogs.korrespondent.net
Tool: R
Text Mining with R -- an Analysis of Twitter Data
This document discusses analyzing Twitter data using text mining techniques in R. It outlines extracting tweets from Twitter and cleaning the text by removing punctuation, numbers, URLs, and stopwords. It then analyzes the cleaned text by finding frequent words, word associations, and creating a word cloud visualization. It performs text clustering on the tweets using hierarchical and k-means clustering. Finally, it models topics in the tweets using partitioning around medoids clustering. The overall goal is to demonstrate various text mining and natural language processing techniques for analyzing Twitter data in R.
R by example: mining Twitter for consumer attitudes towards airlines
This document describes analyzing sentiment towards airlines on Twitter. It searches Twitter for mentions of airlines, collects the tweets, scores the sentiment of each tweet using a simple word counting algorithm, and summarizes the results for each airline. It then compares the Twitter sentiment scores to customer satisfaction scores from the American Customer Satisfaction Index. A linear regression shows a relationship between the Twitter and ACSI scores, suggesting Twitter sentiment analysis can provide insights into customer satisfaction.
How Sentiment Analysis works
Sentiment analysis software uses natural language processing and artificial intelligence to analyze text such as reviews and identify whether the opinions and sentiments expressed are positive or negative. It can help businesses understand customer perceptions of products and brands. While sentiment analysis works reasonably well for classifying simple positive and negative sentiments, it faces challenges in dealing with ambiguity and nuance in human language. The accuracy of sentiment analysis depends on factors such as the complexity of the language analyzed and how finely sentiments are classified.
Tutorial of Sentiment Analysis
This document provides an overview of sentiment analysis techniques including AFINN-111, SentiWordNet, and document classification. It describes analyzing sentiment at the word level using lexicons and at the document level. Key steps are outlined such as tokenization, part-of-speech tagging, word sense disambiguation, and assigning sentiment scores. Issues with analyzing short texts like tweets are also discussed. The document provides references and links to related projects and APIs.
Introduction to Sentiment Analysis
This document provides an introduction to sentiment analysis. It begins with an overview of sentiment analysis and what it aims to do, which is to automatically extract subjective content like opinions from digital text and classify the sentiment as positive or negative. It then discusses the components of sentiment analysis like subjectivity and sources of subjective text. Different approaches to sentiment analysis are presented like lexicon-based, supervised learning, and unsupervised learning. Challenges in sentiment analysis are also outlined, such as dealing with language, domain, spam, and identifying reliable content. The document concludes with references for further reading.
Noah's Ark
This certificate certifies that Pratik Pal successfully attained an E0 competency level in Big Data and Hadoop Ecosystem Fundamentals through Noah's Ark Digital Learning Program in October 2015. The certification is signed by Sitaraman R, Global Head of Retail and CPG.
bigdata_pig
Pig is a platform for analyzing large datasets that operates on Hadoop. It provides a high-level language called Pig Latin for expressing data analysis programs, as well as infrastructure for evaluating these programs. Pig Latin scripts are compiled into sequences of MapReduce jobs for execution on Hadoop clusters.
bigdata_nosql_DBaaS
NoSQL and DBaaS are database technologies. Sitaraman Ramachandran will give an introduction to NoSQL and DBaaS on August 5, 2015 to discuss these database technologies. The document provides information about an upcoming presentation on NoSQL and DBaaS databases.
bigdata_mapreduce
MapReduce is a programming model and framework for processing large datasets in a distributed computing environment. It was originally developed by Google to process vast amounts of data across thousands of commodity servers. The MapReduce programming model consists of two functions - map and reduce - that process key/value pairs to perform filtering and sorting operations in parallel.
bigdata_hive
Hive provides a mechanism for easy data summarization, ad-hoc querying, and analysis of large datasets stored in Hadoop. It presents a SQL-like interface to query data stored in Hadoop and relieves users from writing MapReduce programs directly. The document discusses how Hive can be used to access and analyze Hadoop data through a simple SQL-like interface.
bigdata_hadoop_fundamentals
Hadoop Fundamentals I is a course taught by Sitaraman Ramachandran on July 31, 2015. The course provides an introduction to the fundamentals of Hadoop, including its architecture and core components. Students learn the basics of Hadoop necessary to work with big data on Hadoop platforms.
More Related Content
Viewers also liked
Integrating R & Hadoop - Text Mining & Sentiment Analysis
The document discusses integrating R and Hadoop for big data analytics. It notes that existing statistical applications like R are incapable of handling big data, while data management tools lack analytical capabilities. Integrating R with Hadoop bridges this gap by leveraging R's analytics and statistics functionality with Hadoop's ability to process and store distributed data. RHadoop is introduced as an open source project that allows R programmers to directly use MapReduce functionality in R code. Specific RHadoop packages like rhdfs and rmr2 are described that enable interacting with HDFS and performing statistical analysis via MapReduce on Hadoop clusters. Text analytics use cases with R and Hadoop like sentiment analysis are also briefly outlined.
TextMining with R
Goal: Demonstrate popular practices when mining big dissimilar texts
Object of mining: texts from site: http://blogs.korrespondent.net
Tool: R
Text Mining with R -- an Analysis of Twitter Data
This document discusses analyzing Twitter data using text mining techniques in R. It outlines extracting tweets from Twitter and cleaning the text by removing punctuation, numbers, URLs, and stopwords. It then analyzes the cleaned text by finding frequent words, word associations, and creating a word cloud visualization. It performs text clustering on the tweets using hierarchical and k-means clustering. Finally, it models topics in the tweets using partitioning around medoids clustering. The overall goal is to demonstrate various text mining and natural language processing techniques for analyzing Twitter data in R.
R by example: mining Twitter for consumer attitudes towards airlines
This document describes analyzing sentiment towards airlines on Twitter. It searches Twitter for mentions of airlines, collects the tweets, scores the sentiment of each tweet using a simple word counting algorithm, and summarizes the results for each airline. It then compares the Twitter sentiment scores to customer satisfaction scores from the American Customer Satisfaction Index. A linear regression shows a relationship between the Twitter and ACSI scores, suggesting Twitter sentiment analysis can provide insights into customer satisfaction.
How Sentiment Analysis works
Sentiment analysis software uses natural language processing and artificial intelligence to analyze text such as reviews and identify whether the opinions and sentiments expressed are positive or negative. It can help businesses understand customer perceptions of products and brands. While sentiment analysis works reasonably well for classifying simple positive and negative sentiments, it faces challenges in dealing with ambiguity and nuance in human language. The accuracy of sentiment analysis depends on factors such as the complexity of the language analyzed and how finely sentiments are classified.
Tutorial of Sentiment Analysis
This document provides an overview of sentiment analysis techniques including AFINN-111, SentiWordNet, and document classification. It describes analyzing sentiment at the word level using lexicons and at the document level. Key steps are outlined such as tokenization, part-of-speech tagging, word sense disambiguation, and assigning sentiment scores. Issues with analyzing short texts like tweets are also discussed. The document provides references and links to related projects and APIs.
Introduction to Sentiment Analysis
This document provides an introduction to sentiment analysis. It begins with an overview of sentiment analysis and what it aims to do, which is to automatically extract subjective content like opinions from digital text and classify the sentiment as positive or negative. It then discusses the components of sentiment analysis like subjectivity and sources of subjective text. Different approaches to sentiment analysis are presented like lexicon-based, supervised learning, and unsupervised learning. Challenges in sentiment analysis are also outlined, such as dealing with language, domain, spam, and identifying reliable content. The document concludes with references for further reading.
Viewers also liked (7)
Integrating R & Hadoop - Text Mining & Sentiment Analysis
Integrating R & Hadoop - Text Mining & Sentiment Analysis
R by example: mining Twitter for consumer attitudes towards airlines
R by example: mining Twitter for consumer attitudes towards airlines
More from Sitaraman Ramachandran
Noah's Ark
This certificate certifies that Pratik Pal successfully attained an E0 competency level in Big Data and Hadoop Ecosystem Fundamentals through Noah's Ark Digital Learning Program in October 2015. The certification is signed by Sitaraman R, Global Head of Retail and CPG.
bigdata_pig
Pig is a platform for analyzing large datasets that operates on Hadoop. It provides a high-level language called Pig Latin for expressing data analysis programs, as well as infrastructure for evaluating these programs. Pig Latin scripts are compiled into sequences of MapReduce jobs for execution on Hadoop clusters.
bigdata_nosql_DBaaS
NoSQL and DBaaS are database technologies. Sitaraman Ramachandran will give an introduction to NoSQL and DBaaS on August 5, 2015 to discuss these database technologies. The document provides information about an upcoming presentation on NoSQL and DBaaS databases.
bigdata_mapreduce
MapReduce is a programming model and framework for processing large datasets in a distributed computing environment. It was originally developed by Google to process vast amounts of data across thousands of commodity servers. The MapReduce programming model consists of two functions - map and reduce - that process key/value pairs to perform filtering and sorting operations in parallel.
bigdata_hive
Hive provides a mechanism for easy data summarization, ad-hoc querying, and analysis of large datasets stored in Hadoop. It presents a SQL-like interface to query data stored in Hadoop and relieves users from writing MapReduce programs directly. The document discusses how Hive can be used to access and analyze Hadoop data through a simple SQL-like interface.
bigdata_hadoop_fundamentals
Hadoop Fundamentals I is a course taught by Sitaraman Ramachandran on July 31, 2015. The course provides an introduction to the fundamentals of Hadoop, including its architecture and core components. Students learn the basics of Hadoop necessary to work with big data on Hadoop platforms.
More from Sitaraman Ramachandran (13)
bigdata_datascience_methodology
- 1. Sitaraman Ramachandran Data Science Methodology July 31, 2015