Your SlideShare is downloading. ×
Analytics and Access to the UK web archive
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Analytics and Access to the UK web archive

464

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
464
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page Footer text here...
  • Header text here... Footer text here... Page
  • Header text here... Footer text here... Page
  • Transcript

    • 1. Access and Analytics to the UK Web Archive Lewis Crawford, Web Archive Technical Lead The British Library
    • 2. Introduction
      • This talk will cover:
      • Background of the UK Web Archive
      • Traditional access methods to Web Archives
      • Full text search for resource discovery
      • Problems of scale – needles and haystacks
    • 3. Web Archiving: the basics
      • What Selecting, capturing, storing, preserving and managing access to snapshots of websites over time
      • How Use crawler software to download websites automatically
          • Selective or domain archiving
          • Provide access in a Web Archive
      • When Since mid 1990s
      • Who Heritage and memory organisations, eg (IIPC)
          • University libraries
          • Not-for-profit and commercial organisations, eg Internet Archive
          • Individual researchers
      • Why Global information resource
          • Artefact of cultural and technology change
          • Representative sample of the web: historical and sociological data that may not be found elsewhere
          • Part of national digital heritage - legal requirements
    • 4. UK Web Archive:
    • 5. Web archive as historical documents
    • 6. Multimedia based content
    • 7. 3D visualisation wall
    • 8. Full text search
    • 9. N-gram visualisation
    • 10. N-gram visualisation
    • 11. Media based results
    • 12. Semantic analysis
    • 13. Scale: needle and haystack
      • Google: “seen 1 trillion unique URLs”
      • more than a billion new pages are added to the web every day
      • The UK web domain
        • 9 million .uk domain names registered in December 2010
        • ~ 1 million using other domain names
        • Growing at 11% - 14% per year
        • 40% estimated to be in scope for Legal Deposit
        • Estimated ~110TB each UK domain crawl
      • Subject hierarchy visualisation UK Web Archive
      • ~ 10,000 websites collected since 2004
      • ~ 40,000 instances
    • 14. The value of the haystacks – content visualisation
    • 15. Big Data analytics
      • Java Map/Reduce to use Tika to extract text and generate XML files for Solr ingest
      • Hive & Pig for ad hoc query analysis
    • 16. Search indexing process SOLR Dedicated Indexer SOLR Dedicated Search Hadoop Node 1 Node 50 (w)arcs Document Meta Service Meta Database XML Document store Web Access Replication WCT Crawlers Generate (w)arcs Insert meta information Retrieve (w)arcs and meta information Generate xml files DIH Indexes new xml SOLR Dedicated Indexer XML Image store DIH Indexes new xml SOLR Dedicated Indexer XML Media store DIH Indexes new xml SOLR Dedicated Search SOLR Dedicated Search Replication Replication
    • 17. Tag cloud analysis – General Election 2005
      • Special Collection 2005 general election
          • 147 websites archived during and immediately after the UK general election campaign of 2005.
        • Tag clouds (or weighted lists) generated for websites belonging to key political parties
        • Shows the most frequently used words in the websites during the 2005 election campaign
      • Special collection 2010 general election now available
    • 18. The value of the haystacks – postcode-based access
    • 19. 1: Blue 2-5: Green 5+ Purple 50+ Yellow 100+ Red
    • 20. Questions?
      • Thank you.
      • http://www.webarchive.org.uk
      • [email_address]
      • @relephantdata

    ×