Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Centipede: Analyzing Web Crawl data for context of a location Vikas Bansal Primal Pappachan Abhishek Sethi
Introduction
Introduction
Description A web service that presents the context associated with a location
Context of a location 1. Weather 2. Healthcare 3. Crime 4. Employment 5. ……
Customers 1. Moving/Travelling into a new place 2. Policy Makers 3. Journalists 4. Researchers
Scenario
Related Services ● Yelp ● Google news ● http://bestplaces.net/ ● http://www.nycgo.com/events/ ● http://www.stubhub.com/
Technical Description of Service ● Analyze the web crawl data ● Create a list of locations ● Filter top 100 words from the...
System Architecture
Data Sources •Common Crawl Data from Amazon S3 –Contains information on billions of web pages –Search through the contents...
Technologies and Resources ● Hadoop Cluster on Bluegrit System ● Apache Pig ○ Python for UDF’s ● Java/PHP for front end de...
MapReduce Job Splitter ● Sentence ● Paragraph ● Article
Elastic Search ● Distributed restful search and analytics. ● Has near real-time search. ● Resilient clusters - detect and ...
Challenges and Limitations •Amount of HDD space available. •Learning new technologies such as Apache Pig, WSDL etc. •Creat...
Timeline
References ● Data set ● Common Crawl Web data ● Elastic Search ● Apache Pig ● Elastic Search for Term Filter lookup ● Hado...
Upcoming SlideShare
Loading in …5
×

Cenitpede: Analyzing Webcrawl

1,971 views

Published on

Published in: Technology, Education
no profile picture user

  • Be the first to comment

  • Be the first to like this

Cenitpede: Analyzing Webcrawl

  1. 1. Centipede: Analyzing Web Crawl data for context of a location Vikas Bansal Primal Pappachan Abhishek Sethi
  2. 2. Introduction
  3. 3. Introduction
  4. 4. Description A web service that presents the context associated with a location
  5. 5. Context of a location 1. Weather 2. Healthcare 3. Crime 4. Employment 5. ……
  6. 6. Customers 1. Moving/Travelling into a new place 2. Policy Makers 3. Journalists 4. Researchers
  7. 7. Scenario
  8. 8. Related Services ● Yelp ● Google news ● http://bestplaces.net/ ● http://www.nycgo.com/events/ ● http://www.stubhub.com/
  9. 9. Technical Description of Service ● Analyze the web crawl data ● Create a list of locations ● Filter top 100 words from the files that mention a location from the list ● Build an index of location against list of words corresponding to that location
  10. 10. System Architecture
  11. 11. Data Sources •Common Crawl Data from Amazon S3 –Contains information on billions of web pages –Search through the contents –Use ARC and Text files
  12. 12. Technologies and Resources ● Hadoop Cluster on Bluegrit System ● Apache Pig ○ Python for UDF’s ● Java/PHP for front end development ○ Use a Jboss container for Java, Xampp for PHP ● Elastic Search ● Map Reduce ● SQL/NoSQL database ● REST ● WSDL 2.0 ● AWS - RDS, R53, EC2
  13. 13. MapReduce Job Splitter ● Sentence ● Paragraph ● Article
  14. 14. Elastic Search ● Distributed restful search and analytics. ● Has near real-time search. ● Resilient clusters - detect and remove failed nodes.
  15. 15. Challenges and Limitations •Amount of HDD space available. •Learning new technologies such as Apache Pig, WSDL etc. •Creating special UDF’s in Python.
  16. 16. Timeline
  17. 17. References ● Data set ● Common Crawl Web data ● Elastic Search ● Apache Pig ● Elastic Search for Term Filter lookup ● Hadoop Tutorial ● Dean, Jeffrey, and Sanjay Ghemawat. "MapReduce: simplified data processing on large clusters." Communications of the ACM 51.1 (2008): 107-113. ● Blei, David M., Andrew Y. Ng, and Michael I. Jordan. "Latent dirichlet allocation." the Journal of machine Learning research 3 (2003): 993-1022.

×