WebART - "Data Digging" - eHumanities Group 2013

1. WebART project Web Archive RetrievalTools Jaap Kamps, Richard Rogers, Arjen deVries Paul Doorenbosch, RenéVoorburg,Victor-JanVos Anat Ben-David, Hugo Huurdeman,Thaer Sammar Flickr: LucViatour eHumanities Group,“NewTrends in eHumanities”, Sept. 19 2013, Meertens Institute

2. WebART project Web Archive RetrievalTools Jaap Kamps, Richard Rogers, Arjen deVries Paul Doorenbosch, RenéVoorburg,Victor-JanVos Anat Ben-David, Hugo Huurdeman,Thaer Sammar Flickr: LucViatour Data Diggin’ @ KB eHumanities Group,“NewTrends in eHumanities”, Sept. 19 2013, Meertens Institute

3. Contents •The WebART project & KB Web archive •Data Diggin’ @ KB •Analysis •DiggingTowards the Future

4. 2012-2016

5. Thaer Samar PhD/programmer Hugo Huurdeman PhD researcher Anat Ben-David Postdoc Arjen deVries Jaap Kamps Richard Rogers Paul Doorenbosch Hildelies Balk Victor-JanVos RenéVoorburg

6. WebART Goals •Evaluating current curation and selection procedures of Web archives •Getting insights into current use of Web archives •Developing new methods and tools for research using Web archives

7. What are Web archives for?

8. Flickr: koninklijkebibliotheek KB:Web archive since 2007 Statistics: •4,000+ websites •17,000+ harvests •7+TerabyteSelective approach

9. KB:Web archive since 2007 Statistics: •4,000+ websites •17,000+ harvests •7+TerabyteSelective approach Original image:A N P

10. ”Wayback Machine” interface

11. Data Diggin’ @ KB •DMI Summer School (2012) • analysis of selection lists KB •DMI Winter School (2013) • use of nu.nl daily harvests KB dataset •Workshop: Sept ‘11 Day (2013) • use of full Web archive KB dataset

12. DMI Summer School (2012)Data digging, part 1 Selection lists KBData: Toolset: Web-based tools Flickr: Silvertje

13. DMI Summer School (2012)

14. • Digital Methods Winter School (Jan. ’13) • Co-design workshop (“Living Lab”) • New Media researchers & developers • ﬁrst use WebARTist Data digging, part II nu.nl daily harvestsData: Toolset: Full-text search Web-based tools

15. • Full-text search:WebARTist (pilot - beta 1) • Initial dataset (corpus) • 432 crawls, 16 months (13.64 GB) KB CommonCrawl+ nu.nl (Dutch news aggregator) Full-text searchData digging, part II

16. Full-text search

20. Word frequency analysis 0 100 200 300 400 500 600 700 800 17/05/2011 25/08/2011 03/12/2011 12/03/2012 20/06/2012 28/09/2012 06/01/2013

21. Co-Word Analysis

22. 1 abcnews.go.com1 brucespringsteen.net 1 theverge.com 1 sportamerika.nl 1 reuters.com 1 ebird.org 1 googleblog.blogspot.co.uk 1 presscentre.sony.eu 1 project.wnyc.org 1 bbc.com 1 poynter.org 1 abclocal.go.com 1 en.wikipedia.org 1 nhc.noaa.gov 1 nypost.com 2 earthcam.com 2 maps.google.com 3 hp.com 4 google.org 4 edition.cnn.com Syria Sandy 7 wired.com 7 allthingsd.com 7 abcnews.go.com 7 thesun.co.uk 7 allesoversterrenkunde.nl 8 volkskrant.nl 9 fd.nl 9 nos.nl 9 mobiel.nuvideo.nl 9 guardian.co.uk 10 bit.ly 10 billboard.biz 10 cbsnews.com 11 usmagazine.com 11 variety.com 12 theverge.com 12 people.com 13 Rutte enVerhagen leggen schuld bij PVV 13 telegraaf.nl 14 washingtonpost.com 18 edition.cnn.com 19 bbc.co.uk 20 youtube.com 20 nytimes.com 21 styletoday.nl 21 bloomberg.com 24 thesistools.com 26 hollywoodreporter.com 30 online.wsj.com 30 deadline.com 33 poll.nupubliek.nl 34 spaarrente.nl 39 gamer.nl 48 reuters.com 52 tmz.com 57 open.spotify.com 78 peil.nl 93 gezondheidsnet.nl US Election 4 1 blogs.aljazeera.net 1 youtube.com 1 worldpressphoto.org 1 wikileaks.org 1 washingtonpost.com 1 eubusiness.com 1 vesti.bg 1 trouw.nl 1 #NAME 1 en.wikipedia.org 1 l 1 sana.sy 1 hosted.ap.org 1 shariah4belgium.com 1 nrc.nl 1 guardian.co.uk 1 geopolicity.com 1 nctb.nl 1 rt.com 1 kaspersky.com 2 todayszaman.com 2 volkskrant.nl 2 spaarrente.nl 2 reuters.com 2 peil.nl 2 hrw.org 2 uk.reuters.com 2 cbsnews.com 3 telegraph.co.uk 3 maps.google.nl 4 bbc.co.uk 5 edition.cnn.com 5 aljazeera.com english.alarabiya.net 7 maps.google.com Outlink Analysis

23. Geomapping location Wire service

24. Temporal Image Analyses

25. Timeline

26. DMI “9/11 Day” (2013)Data digging, part III Full KB ArchiveDatasets: Toolset: Web-based tools nu.nl “host+1” Full-text search+ Geo-index

27. Full-text search+

30. •New Media researchers’ interests: • “derive periodizations of the Web” (Web history) • “source hierarchy” (dominant sources in archive) • “keyword uptake” (terms over time) • e.g.‘geenstijl language in archive’ • “accidental”/“incidental” archiving • e.g.‘the guilty pleasures of the Web of innocence’ DMI “9/11 Day” (2013)Data digging, part III

31. 2009 2010 2011 2012

32. 2009 2010 2011 2012

33. 2009 2010 2011 2012

34. 2009 2010 2011 2012

35. Analysis (1) • studying the ‘archive’ vs. the ‘archived content’ • researchers’ (un)familiarity with temporal (archive) search • “conditioned” to Google-style searching • high demand for export functions and aggregation features

36. Analysis (2) •“data is still a crucial factor” • quantity & quality: inherent incompleteness & inconsistencies • not always clear what’s in & what’s out • crawl settings (e.g depth), temporal gaps • “researchers always want what isn’t there”

37. Digging towards the future Full KB ArchiveDatasets: Toolset: “Toolmaker’s tools” ++

38. A step further... •Build customizable systems, or, toolmakers’ tools •Provide building blocks

39. A step further... use “Hadoop” computing power to build custom dataset, perform high-level analysis, etc.

40. New tools: examples •select,“clean”, ﬁlter & process dataset •employ complex queries & search strategies •search, summarize, aggre- gate & share

41. Moving beyond mere “search” Wayback Machine Search engine “Research” engine explicit support for full research task, including analysis and synthesis steps

42. Summary •The WebART project •Data Diggin’ @ KB •Analysis •DiggingTowards the Future Summary

43. webarchiving.nl @webart12

44. WebART project Web Archive RetrievalTools Jaap Kamps, Richard Rogers, Arjen deVries Paul Doorenbosch, RenéVoorburg,Victor-JanVos Anat Ben-David, Hugo Huurdeman,Thaer Sammar Flickr: LucViatour eHumanities Group, NewTrends in eHumanities, Sept. 19 2013, Meertens Institute

WebART - "Data Digging" - eHumanities Group 2013

TimelessFuture

WebART - "Data Digging" - eHumanities Group 2013