• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Analytics and Access to the UK web archive
 

Analytics and Access to the UK web archive

on

  • 511 views

 

Statistics

Views

Total Views
511
Views on SlideShare
505
Embed Views
6

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 6

http://www.linkedin.com 5
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page

Analytics and Access to the UK web archive Analytics and Access to the UK web archive Presentation Transcript

  • Access and Analytics to the UK Web Archive Lewis Crawford, Web Archive Technical Lead The British Library
  • Introduction
    • This talk will cover:
    • Background of the UK Web Archive
    • Traditional access methods to Web Archives
    • Full text search for resource discovery
    • Problems of scale – needles and haystacks
  • Web Archiving: the basics
    • What Selecting, capturing, storing, preserving and managing access to snapshots of websites over time
    • How Use crawler software to download websites automatically
        • Selective or domain archiving
        • Provide access in a Web Archive
    • When Since mid 1990s
    • Who Heritage and memory organisations, eg (IIPC)
        • University libraries
        • Not-for-profit and commercial organisations, eg Internet Archive
        • Individual researchers
    • Why Global information resource
        • Artefact of cultural and technology change
        • Representative sample of the web: historical and sociological data that may not be found elsewhere
        • Part of national digital heritage - legal requirements
  • UK Web Archive:
  • Web archive as historical documents
  • Multimedia based content
  • 3D visualisation wall
  • Full text search
  • N-gram visualisation
  • N-gram visualisation
  • Media based results
  • Semantic analysis
  • Scale: needle and haystack
    • Google: “seen 1 trillion unique URLs”
    • more than a billion new pages are added to the web every day
    • The UK web domain
      • 9 million .uk domain names registered in December 2010
      • ~ 1 million using other domain names
      • Growing at 11% - 14% per year
      • 40% estimated to be in scope for Legal Deposit
      • Estimated ~110TB each UK domain crawl
    • Subject hierarchy visualisation UK Web Archive
    • ~ 10,000 websites collected since 2004
    • ~ 40,000 instances
  • The value of the haystacks – content visualisation
  • Big Data analytics
    • Java Map/Reduce to use Tika to extract text and generate XML files for Solr ingest
    • Hive & Pig for ad hoc query analysis
  • Search indexing process SOLR Dedicated Indexer SOLR Dedicated Search Hadoop Node 1 Node 50 (w)arcs Document Meta Service Meta Database XML Document store Web Access Replication WCT Crawlers Generate (w)arcs Insert meta information Retrieve (w)arcs and meta information Generate xml files DIH Indexes new xml SOLR Dedicated Indexer XML Image store DIH Indexes new xml SOLR Dedicated Indexer XML Media store DIH Indexes new xml SOLR Dedicated Search SOLR Dedicated Search Replication Replication
  • Tag cloud analysis – General Election 2005
    • Special Collection 2005 general election
        • 147 websites archived during and immediately after the UK general election campaign of 2005.
      • Tag clouds (or weighted lists) generated for websites belonging to key political parties
      • Shows the most frequently used words in the websites during the 2005 election campaign
    • Special collection 2010 general election now available
  • The value of the haystacks – postcode-based access
  • 1: Blue 2-5: Green 5+ Purple 50+ Yellow 100+ Red
  • Questions?
    • Thank you.
    • http://www.webarchive.org.uk
    • [email_address]
    • @relephantdata