• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Web Search - Lecture 10 - Web Information Systems (4011474FNR)
 

Web Search - Lecture 10 - Web Information Systems (4011474FNR)

on

  • 3,552 views

This lecture is part of a Web Information Systems course given at the Vrije Universiteit Brussel.

This lecture is part of a Web Information Systems course given at the Vrije Universiteit Brussel.

Statistics

Views

Total Views
3,552
Views on SlideShare
3,504
Embed Views
48

Actions

Likes
2
Downloads
152
Comments
0

4 Embeds 48

http://wise.vub.ac.be 34
http://www.inf.ethz.ch 12
http://www.slideshare.net 1
http://translate.googleusercontent.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Web Search - Lecture 10 - Web Information Systems (4011474FNR) Web Search - Lecture 10 - Web Information Systems (4011474FNR) Presentation Transcript

    • Web Information Systems Web Search Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel http://www.beatsigner.com 2 December 2005
    • Search Engine Result Pages (SERP) December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2
    • Search Engine Result Pages (SERP) ... December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3
    • Vertical Search Result Pages December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4
    • Search Engine Market Share (2013) December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5
    • Search Engine Result Page  There is a variety of information shown on a search engine result page (SERP)        organic search results non-organic search results meta-information about the result (e.g. number of result pages) vertical navigation advanced search options query refinement suggestions ... December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6
    • Search Engine History  Early "search engines" include various systems starting with Bush's Memex  Archie (1990)   first Internet search engine indexing of files on FTP servers  W3Catalog (September 1993)   first "web search engine" mirroring and integration of manually maintained catalogues  JumpStation (December 1993)  first web search engine combining crawling, indexing and searching December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7
    • Search Engine History ...  In the following two years (1994/1995) many new search engines appeared  AltaVista, Infoseek, Excite, Inktomi, Yahoo!, ...  Two categories of early Web search solutions  full text search - based on an index that is automatically created by a web crawler in combination with an indexer - e.g. AltaVista or InfoSeek  manually maintained classification (hierarchy) of webpages - significant human editing effort - e.g. Yahoo December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8
    • Information Retrieval  Precision and recall can be used to measure the performance of different information retrieval algorithms relevant documents  retrieved documents precision  retrieved documents relevant documents  retrieved documents recall  relevant documents D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 December 6, 2013 D1 D3 D9 query D10 D8 precision  3  0.6 5 3 recall   0.75 4 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9
    • Information Retrieval ...  Often a combination of precision and recall, the so-called F-score (harmonic mean) is used as a single measure F - score  2  precision  recall precision  recall D1 D2 D3 D4 D5 D6 D7 D8 D9 D1 D2 D3 D6 D7 D8 December 6, 2013 D1 D3 D10 D9 D10 D4 D5 D1 D2 D3 D9 D10 D8 D9 D10 precision  0.6 recall  0.75 F - score  0.67 query query D8 D5 precision  0.57 recall  1 F - score  0.73 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10
    • Boolean Model  Based on set theory and boolean logic  Exact matching of documents to a user query  Uses the boolean AND, OR and NOT operators D1 D2 D3 D4 D5 D6 Bank 1 1 0 0 1 1 Delhaize 1 1 1 0 0 0 Ghent 1 0 0 1 1 1 Metro 0 0 1 0 0 0 Shopping 1 0 1 1 1 0 Train 1 1 0 1 0 0 ... ... ... ... ... ... ... inverted index    query: Shopping AND Ghent AND NOT Delhaize computation: 101110 AND 100111 AND 000111 = 000110 result: document set {D4,D5} December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11
    • Boolean Model ...  Advantages   relatively easy to implement and scalable fast query processing based on parallel scanning of indexes  Disadvantages  does not pay attention to synonymy - different words with similar meaning  does not pay attention to polysemy - a single word with different meanings   no ranking of output often the user has to learn a special syntax such as the use of double quotes to search for phrases  Variants of the boolean model form the basis of many search engines December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12
    • Vector Space Model  Algebraic model representing text documents and queries as vectors based on the index terms  one dimension for each term  Compute the similarity (angle) between the query vector and the document vectors  Advantages    simple model based on linear algebra partial matching with relevance scoring for results potenial query reevaluation based on user relevance feedback  Disadvantages   computationally expensive (similarity measures for each query) limited scalability December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13
    • Web Search Engines  Most web search engines are based on traditional information retrieval techniques but they have to be adapted to deal with the characteristics of the Web     immense amount of web resources (>50 billion webpages) hyperlinked resources dynamic content with frequent updates self-organised web resources  Evaluation of performance   no standard collections often based on user studies (satisfaction)  Of course not only the precision and recall but also the query answer time is an important issue December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14
    • Web Search Engine Architecture WWW Storage Manager Crawler content already added? Page Repository URL Pool Ranking filter URL Handler Indexers normalisation and duplicate elimination Query Handler URL Repository Client December 6, 2013 Document Index Special Indexes inverted index Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15
    • Web Crawler  A web crawler or spider is used to create an index of webpages to be used by a web search engine  any web search is then based on this index  Web crawler has to deal with the following issues  freshness - the index should be updated regularly (based on webpage update frequency)  quality - since not all webpages can be indexed, the crawler should give priority to "high quality" pages  scalabilty - it should be possible to increase the crawl rate by just adding additional servers (modular architecture) - e.g. the estimated number of Google servers in 2007 was 1'000'000 (including not only the crawler but the entire Google platform) December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16
    • Web Crawler ...  distribution - the crawler should be able to run in a distributed manner (computer centers all over the world)  robustness - the Web contains a lot of pages with errors and a crawler has to deal with these problems - e.g. deal with a web server that creates an unlimited number of "virtual web pages" (crawler trap)  efficiency - resources (e.g. network bandwidth) should be used in a most efficient way  crawl rates - the crawler should pay attention to existing web server policies (e.g. revisit-after HTML meta tag or robots.txt file) User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ December 6, 2013 robots.txt Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17
    • Pre-1998 Web Search  Find all documents for a given query term  use information retrieval (IR) solutions - boolean model - vector space model - ...  ranking based on "on-page factors"  problem: poor quality of search results (order)  Larry Page and Sergey Brin proposed to compute the absolute quality of a page called PageRank   based on the number and quality of pages linking to a page (votes) query-independent December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18
    • Origins of PageRank  Developed as part of an academic project at Stanford University   research platform to aid understanding of large-scale web data and enable researchers to easily experiment with new search Sergey Brin Larry Page technologies Larry Page and Sergey Brin worked on the project about a new kind of search engine (1995-1998) which finally led to a functional prototype called Google December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19
    • PageRank P1 P5 P7 P8 R1 R5 R7 R8 P2 P3 P4 P6 R2 R3 R4 R6  A page Pi has a high PageRank Ri if   there are many pages linking to it or, if there are some pages with a high PageRank linking to it  Total score = IR score × PageRank December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20
    • Basic PageRank Algorithm R( Pi )   Pj Bi R( Pj ) P1 P2 Lj 1.5 1 1.5 1  where   Bi is the set of pages that link to page Pi Lj is the number of outgoing links for page Pj December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be P3 0.75 1 21
    • Matrix Representation  Let us define a hyperlink matrix H 1 L j H ij    0 P1 P2 if Pj  Bi otherwise 0 1 2 1  H  1 0 0   0 1 2 0    and R  RPi   R  HR P3 R is an eigenvector of H with eigenvalue 1 December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22
    • Matrix Representation ...  We can use the power method to find R  sparse matrix H with 40 billion columns and rows but only an average of 10 non-zero entries in each colum Rt 1  HR t 0 1 2 1    For our example H  1 0 0   0 1 2 0    this results in R  2 December 6, 2013 2 1 or 0.4 0.4 0.2 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23
    • Dangling Pages (Rank Sink)  Problem with pages that have no outgoing links (e.g. P2) P1 C 0 0  H  and R  0 0 1 0 P2 C  Stochastic adjustment  if page Pj has no outgoing links then replace column j with 1/Lj 0 1 2 0 1 2  C  and S  H  C  1 1 2   0 1 2   New stochastic matrix S always has a stationary vector R  can also be interpreted as a markov chain December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24
    • Strongly Connected Pages (Graph)  Add new transition proba- 1-d P1 P2 P4 P3 bilities between all pages    with probability d we follow the hyperlink structure S with probability 1-d we choose a random page matrix G becomes irreducible  Google matrix G reflects a random surfer  no modelling of back button 1 G  d S  1  d  1 n December 6, 2013 1-d P5 1-d R  GR Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25
    • 1 G  d S  1  d  1 n Examples A2 0.37 A1 A3 0.26 0.37 December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26
    • 1 G  d S  1  d  1 n Examples ... A2 B2 0.185 0.185 A1 A3 B1 B3 0.13 0.185 0.13 0.185 P A  0.5 December 6, 2013 PB   0.5 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27
    • 1 G  d S  1  d  1 n Examples A2 B2 0.14 0.20 A1 A3 B1 B3 0.10 0.14 0.22 0.20 P A  0.38  PB   0.62 PageRank leakage December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28
    • 1 G  d S  1  d  1 n Examples ... A2 B2 0.23 0.095 A1 A3 B1 B3 0.3 0.18 0.10 0.095 P A  0.71 December 6, 2013 PB   0.29 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29
    • 1 G  d S  1  d  1 n Examples A2 B2 0.24 0.07 A1 A3 B1 B3 0.35 0.18 0.09 0.07 PB   0.23 P A  0.77  PageRank feedback December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30
    • 1 G  d S  1  d  1 n Examples ... A2 B2 0.17 0.06 A1 A3 B1 B3 0.33 0.175 0.08 0.06 A4 P A  0.80 December 6, 2013 PB   0.20 0.125 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31
    • Google Webmaster Tools  Various services and information about a website  Site configuration     submission of sitemap crawler access URLs of indexed pages settings - e.g. preferred domain  Your site on the web    search queries keywords internal and external links December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32
    • Google Webmaster Tools ...  Diagnostics   crawl rates and errors HTML suggestions  Use HTML suggestions for on-page factor optimisation  meta description - duplicate meta descriptions - too long meta descriptions  title tag - missing or duplicate title tags - too long or too short title tags  non-indexable content  Similar tools offered by other search engines  e.g. Bing Webmaster Tools December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33
    • XML Sitemaps <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.example.com/sitemap/0.9"> <url> <loc>http://www.tenera.ch/zyliss-trommelreibe.html</loc> <lastmod>2013-07-06</lastmod> <changefreq>weekly</changefreq> <priority>0.4</priority> </url> <url> <loc>http://www.tenera.ch/tenera-universalmesser.html</loc> <lastmod>2012-12-05</lastmod> <changefreq>weekly</changefreq> <priority>0.1</priority> </url> ... </urlset>  List of URLs that should be crawled and indexed December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34
    • XML Sitemaps ...  All major search engines support the sitemap format  The URLs of sitemap are not guaranteed to be added to a search engine's index  helps search engine to find pages that are not yet indexed  Additional metadata might be provided to search engines    relative page relevance (priority) date of last modififaction (lastmod) update frequency (changefreq) December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35
    • Questions  Is PageRank fair?  What about Google's power and influence?  What about Web 2.0 or Web 3.0 and web search?   "non-existent" webpages such as offered by Rich Internet Applications (e.g. using AJAX) may bring problems for traditional search engines (hidden web) new forms of social search - Delicious - ...  social marketing December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36
    • The Google Effect  A recent study by Sparrow et al. shows that people less likely remember things that they believe to be accessible online  Internet as a transactive memory  Does our memory work differently in the age of Google?  What implications will the future of the Internet and new search have? December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37
    • Search Engine Marketing (SEM)  For many companies Internet marketing has become a big business  Search engine marketing (SEM) aims to increase the visibility of a website    search engine optimisation (SEO) paid search advertising (non-organic search) social media marketing  SEO should not be decoupled from a website's content, structure, design and used technologies  SEO has to be seen as an continuous process in a rapidly changing environment  different search engines with regular changes in ranking December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38
    • Structural Choices  Keep the website structure as flat a possible   minimise link depth avoid pages with much more than 100 links  Think about your website's internal link structure    which pages are directly linked from the homepage? create many internal links for important pages be "careful" about where to put outgoing links - PageRank leakage   use keyword-rich anchor texts dynamically create links between related content - e.g. "customer who bought this also bought ..." or "visitors who viewed this also viewed ..."  Increase the number of pages December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39
    • Technological Choices  Use SEO-friendly content management system (CMS)  Dynamic URLs vs. static URLs   avoid session IDs and parameters in URL use URL rewriting to get descriptive URLs containing keywords  Think carefully about the use of dynamic content   Rich Internet Applications (RIAs) based on AJAX etc. content hidden behind pull-down menus etc.  Address webpages consistently  http://www.vub.ac.be  http://www.vub.ac.be/index.php  Some notes about the Google toolbar   shows logarithmic PageRank value (from 0 to 10) information not frequently updated (google dance) December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40
    • Consistent Addressing of Webpages December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41
    • Search Engine Optimisations  Different things can be optimised   on-page factors off-page factors  It is assumed that some search engines use more than 200 on-page and off-page factors for their ranking  Difference between optimisation and breaking the "search engine rules"  white hat and black hat optimisations  A bad ranking or removal from index can cost a company a lot of money or even mark the end of the company  e.g. supplemental index ("Google hell") December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42
    • Positive On-Page Factors  Use of keywords at relevant places      in title tag (preferably one of the first words) in URL in domain name in header tags (e.g. <h1>) multiple times in body text  Provide metadata  e.g. <meta name="description"> also used by search engines to create the text snippets on the SERPs  Quality of HTML code  Uniqueness of content across the website  Page freshness (changes from time to time) December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43
    • Negative On-Page Factors  Links to "bad neighbourhood"  Link selling      in 2007 Google announced a campaign against paid links that transfer PageRank Over optimisation penalty (keyword stuffing) Text with same colour as background (hidden content) Automatic redirect via the refresh meta tag Cloaking  different pages for spider and user  Malware being hosted on the page December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44
    • Negative On-Page Factors ...      Duplicate or similar content Duplicate page titles or meta tags Slow page load time Any copyright violations ... December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45
    • Positive Off-Page Factors     Links from pages with a high PageRank Keywords in anchor text of inbound links Links from topically relevant sites High clickthrough rate (CTR) from search engine for a given keyword  Listed in DMOZ / Open Directory Project (ODP) and Yahoo directories  High number of shares on social networks  e.g. Facebook, Google+ or Twitter December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46
    • Positive Off-Page Factors ...  Site age (stability)  Google sandbox?  Domain expiration date  High PageRank  ... December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47
    • Negative Off-Page Factors  Site often not accessible to crawlers  e.g. server problem  High bounce rate  users immediately press the back button  Link buying      rapidly increasing number of inbound links Use of link farms Participation in link sharing programmes Links from bad neighbourhood? Competitor attack (e.g. via duplicate content)? December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 48
    • Black Hat Optimisations (Don'ts)  Link farms  Spamdexing in guestbooks, Wikipedia etc.  "solution": <a rel="nofollow" href="...">...</a>  Keyword Stuffing  overuse of keywords - content keyword stuffing - image keyword stuffing - keywords in meta tags - invisible text with keywords  Selling/buying links   "big" business until 2007 costs based on the PageRank of the linking site December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 49
    • Black Hat Optimisations (Don'ts) ...  Doorway pages (cloaking)  doorway pages are normally just designed for search engines - user is automatically redirected to the target page  e.g. BMW Germany and Ricoh Germany banned in February 2006 December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 50
    • Nofollow Link Example  Nofollow value for hyperlinks introduced by Google in 2005 to avoid spamdexing  <a rel="nofollow" href="...">...</a>  Links with a nofollow value were not counted in the PageRank computation   division by number of outgoing links e.g. page with 9 outgoing links and 3 of them are nofollow links - PageRank divided by 6 and distributed across the 6 "really linked pages"  SEO experts started to use (misuse) the nofollow links for PageRank sculpting  control flow of PageRank within a website December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 51
    • Nofollow Link Example ...  In June 2009 Google decided to treat nofollow links differently to avoid PageRank sculpting   division by total number of outgoing links e.g. page with 9 outgoing links and 3 of them are nofollow links - PageRank divided by 9 and distributed across the 6 "really linked pages"  no longer a good solution to prevent Spamdexing since we loose (diffuse) some PageRank  SEO experts start to use alternative techniques to replace nofollow links  e.g. obfuscated JavaScript links December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 52
    • Product Search  Various shopping and price comparison sites import product data  some of them are free, for others one has to pay  Google Product Search    started as Froogle, became Google Products and now Google Product Search product data uploaded to Google Base very effective vertical search December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 53
    • Non-Organic Search  In addition to the so-called organic search, websites can also participate in non-organic web search   cost per impression (CPI) cost- per-click (CPC)  The non-organic web search should be treated independently from the organic web search  Quality of the landing page can have an impact on the non-organic web search performance!  The Google AdWords programme is an example of a commercial non-organic web search service  other services include Yahoo! Advertising Solutions, Facebook Ads, ... December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 54
    • Google AdWords  pay-per-click (PPC) or cost-per-thousand (CPM)  Campains and ad groups  Two types of advertising   search content network - Google Adsense  Highly customisable ads     region language daytime ... December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 55
    • Google AdWords ...  Excellent control and monitoring for AdWords users  cost per conversion  In 2012 Google's total advertising revenues were 44 billion USD December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 56
    • Conclusions  Web information retrieval techniques have to deal with the specific characteristics of the Web  PageRank algorithm    absolute quality of a page based on incoming links based on random surfer model computed as eigenvector of Google matrix G  PageRank is just one (important) factor  Various implications for website development and SEO December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 57
    • Exercise 10  Web Search and PageRank December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 58
    • References  L. Page, S. Brin, R. Motwani and T. Winograd, The PageRank Citation Ranking: Bringing Order to the Web, January 1998  S. Brin and L. Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine, Computer Networks and ISDN Systems, 30(1-7), April 1998  Amy N. Langville and Carl D. Meyer, Google's PageRank and Beyond – The Science of Search Engine Rankings, Princeton University Press, July 2006  PageRank Calculator  http://www.webworkshop.net/pagerank_calculator.php December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 59
    • References …  B. Sparrow, J. Liu and D.M. Wegner, Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips, Science, July 2011  Google Webmaster Tools  http://www.google.com/webmasters/  The W3C Markup Validation Service  http://validator.w3.org  SEOmoz  http://moz.com December 6, 2013 Beat Signer - Department of Computer Science - bsigner@vub.ac.be 60
    • Next Lecture Security, Privacy and Trust 2 December 2005