The document summarizes the background, purpose, and methods of the UK Web Archive. It discusses how the archive collects, stores, and provides access to snapshots of UK websites over time to preserve digital cultural heritage. It also describes challenges of scale due to the immense size of web content and techniques like full-text search and data analytics that are used to facilitate discovery of information within the archive.

1. Access and Analytics to the UK Web Archive Lewis Crawford, Web Archive Technical Lead The British Library

2. Introduction This talk will cover: Background of the UK Web Archive Traditional access methods to Web Archives Full text search for resource discovery Problems of scale – needles and haystacks

3. Web Archiving: the basics What Selecting, capturing, storing, preserving and managing access to snapshots of websites over time How Use crawler software to download websites automatically Selective or domain archiving Provide access in a Web Archive When Since mid 1990s Who Heritage and memory organisations, eg (IIPC) University libraries Not-for-profit and commercial organisations, eg Internet Archive Individual researchers Why Global information resource Artefact of cultural and technology change Representative sample of the web: historical and sociological data that may not be found elsewhere Part of national digital heritage - legal requirements

4. UK Web Archive:

5. Web archive as historical documents

6. Multimedia based content

7. 3D visualisation wall

8. Full text search

9. N-gram visualisation

10. N-gram visualisation

11. Media based results

12. Semantic analysis

13. Scale: needle and haystack Google: “seen 1 trillion unique URLs” more than a billion new pages are added to the web every day The UK web domain 9 million .uk domain names registered in December 2010 ~ 1 million using other domain names Growing at 11% - 14% per year 40% estimated to be in scope for Legal Deposit Estimated ~110TB each UK domain crawl Subject hierarchy visualisation UK Web Archive ~ 10,000 websites collected since 2004 ~ 40,000 instances

14. The value of the haystacks – content visualisation

15. Big Data analytics Java Map/Reduce to use Tika to extract text and generate XML files for Solr ingest Hive & Pig for ad hoc query analysis

16. Search indexing process SOLR Dedicated Indexer SOLR Dedicated Search Hadoop Node 1 Node 50 (w)arcs Document Meta Service Meta Database XML Document store Web Access Replication WCT Crawlers Generate (w)arcs Insert meta information Retrieve (w)arcs and meta information Generate xml files DIH Indexes new xml SOLR Dedicated Indexer XML Image store DIH Indexes new xml SOLR Dedicated Indexer XML Media store DIH Indexes new xml SOLR Dedicated Search SOLR Dedicated Search Replication Replication

17. Tag cloud analysis – General Election 2005 Special Collection 2005 general election 147 websites archived during and immediately after the UK general election campaign of 2005. Tag clouds (or weighted lists) generated for websites belonging to key political parties Shows the most frequently used words in the websites during the 2005 election campaign Special collection 2010 general election now available

18. The value of the haystacks – postcode-based access

19. 1: Blue 2-5: Green 5+ Purple 50+ Yellow 100+ Red

20. Questions? Thank you. http://www.webarchive.org.uk [email_address] @relephantdata