Jason Webber (Web Archiving Engagement & Liaison Manager): ‘Using the UK Web Archive for research’

1. Using the UK Web Archive
Collections
Jason Webber
Web Archive Engagement Manager

2. www.bl.uk 2
Goals
•Capture and Preserve
the UK Web Space
•Support access to the
collection
•Promote and enable
research

3. www.bl.uk 3
What are we collecting?
All of the UK Public Web Space
• 5-10 million hosts (websites)
•Up to 80-100TB of data each year
•Over 100 curated collections

4. www.bl.uk 4
What don’t we collect?
• Email
• Intranets
• Anything behind a user login
• Flash
• Most (but not all) video and audio content
•Very little Twitter or Facebook

5. www.bl.uk 5
Why are we collecting websites?

6. www.bl.uk 6
Culture disappears

7. www.bl.uk 7
1996
2001
2006 2011
2016
2017

8. www.bl.uk 8
What We’ve Saved (2004-2015)

9. www.bl.uk 9
How often?
‘Everything’ once a year (takes about 3
months)
• Selected sites more frequently (daily,
weekly, monthly, quarterly, six-monthly)
• News and some other sites daily

10. www.bl.uk 10
Do we collect ‘everything’?
‘Everything’ is not everything
• Most sites capped at 500mb (not BBC)
• Database driven websites very hard to collect
• Don’t always look how they should
• Wordpress is really hard

11. www.bl.uk 11

12. www.bl.uk 12

13. www.bl.uk 13
Access
•Licence required to display website publicly
(approx 15,000 websites)
•Otherwise only in a reading room of a Legal
Deposit Library

14. www.bl.uk 14
Discovery
•How do you find what you want when
there are billions of potential results?
•Search can’t work like google (google
know a LOT about you)

15. www.bl.uk 15

16. www.bl.uk 16

17. www.bl.uk 17
Websites have no borders

18. www.bl.uk 18
Secondary Datasets
• JISC UK Web Domain Dataset (1996-2013):
– Format Profile
– Geo-Index
– Host-Level Links
– Crawled URL Index
– WATs (rich resource-level metadata, not released yet)
• UK Open (Selective) Web Archive:
– Website Classification Dataset
• Available as CC0 downloads:
– http://data.webarchive.org.uk/opendata/

19. www.bl.uk 19
beta.webarchive.org.uk/
www.webarchive.org.uk/shine
data.webarchive.org.uk/

20. www.bl.uk 20
Cats – Dogs – Birds

21. www.bl.uk 21
The Force

22. www.bl.uk 22
War on Terror

23. www.bl.uk 23
webarchive.org.uk
webarchive.org.uk/blog
webarchive.org.uk/videos
webarchive.org.uk/shine
data.webarchive.org.uk/opendata
Useful Links ….