Downloaded 23 times
Uploaded byMax Kaiser
Text Mining in Cultural Heritage: Challenges
Max Kaiser discusses the challenges and advancements in text and data mining within cultural heritage contexts, specifically at the Austrian National Library. The document outlines the extensive digitization efforts, including the collaboration with Google, and the potential of OCR and text mining technologies to enhance accessibility to historical texts. It highlights the complexities involved in integrating these technologies into existing library systems and the need for further research and development to effectively utilize text mining in production environments.
More Related Content
Text Mining in Cultural Heritage: Challenges
- 1. @maxkaiser Text Mining inCultural Heritage: Challenges Max Kaiser Head of Research & Development Austrian National Library max.kaiser@onb.ac.at @maxkaiser Text & Data Mining in Europe The Hague, Nov. 11, 2015https://www.flickr.com/photos/shanegorski/2694765397/ No, this is not the Austrian National Library …
- 2.
- 3.
- 4.
- 5.
- 6.
- 7. @maxkaiser digitisation of the entirehistorical book holdings of the Austrian National Library Austrian Books Online www.onb.ac.at/austrianbooksonline/
- 8. @maxkaiser cooperation with Google largest publicprivate partnership in Austria’s cultural sector
- 9.
- 10.
- 11.
- 12. @maxkaiser Austrian Newspapers Online →Started2003 →More than 14 mio pages digitized →More than 10 mio pages OCR read within several projects →Structured by newspaper & publication date
- 13. @maxkaiser Europeana Newspapers →18 partners →12million pages →1.6 mio newspaper pages OCRed @ ONB
- 14. @maxkaiser critical mass of digitallyavailable texts and (meta) data new research questions to textual material?
- 15. @maxkaiser A short digression… Full text data is already useful today (without text mining) Full text data will be even more useful in the future (with text mining)
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27. @maxkaiser close reading distant reading interpretation/ analysis / edition of individual texts analysis of Big Data textmining
- 28.
- 29. @maxkaiser Hadoop-Cluster →Comprehensive analysis withtools on the entire ONB corpus →Enquiries over the whole range of materials in Austrian Books Online →Scalable processing, eg. for text mining →Processing of analyses on site →User-driven analyses
- 30.
- 31. @maxkaiser 325.000+ books 38TBdata: 105 Millionpages 33 Billion running words 5 Billion lines of text 186 identified languages 0.5TBtext index size in 3 Solr Shards Austrian Books Online Status Oct. 2015
- 32.
- 33. 10% 13% 31% 44% 2% 16. Jh. 17. Jh. 18.Jh. 19. Jh. no year centuries…Austrian Books Online
- 34.
- 35. 0,00% 10,00% 20,00% 30,00% 40,00% 50,00% 60,00% 16. Jh. 17.Jh. 18. Jh. 19. Jh. eng fre ger ita lat Austrian Books Online
- 36.
- 37. @maxkaiser →technology/maths are thereand proven → algorithms & tools for NER → algorithms & tools for topic modelling
- 38.
- 39.
- 40. @maxkaiser → Difficult tointegrate in stable (legacy) systems → Base technologies still evolving → Spread of languages / diachronic evolution of languages makes analysis pipelines complex → (Noisy) OCR results as source for analyses → Data not stable but constantly being updated: → Images improve → OCR improves
- 41. @maxkaiser OCR as basisfor text mining →The simple case →The challenging case
- 42. @maxkaiser → Images areOCRed → Processes subject to continuous improvement Text data
- 43. @maxkaiser → Chunks ofimage and text-blocks are recognized → Content is split into contiguous blocks of text → The yellow boxes are recognized as graphic content Text data
- 44. @maxkaiser → If possible, paragraph structuresare used for chunking. Text data
- 45. @maxkaiser → Lines andwords are identified Text data
- 46.
- 47.
- 48.
- 49.
- 50.
- 51. @maxkaiser →IP or otherwiserestricted content has to be processed on site →(e.g. local Hadoop installation) →Interfaces (APIs etc.) not yet available, but will be in the near future
- 52.
- 53. @maxkaiser → Text miningof OCRed historical texts is still a research topic → R&D in this area is understood, however still requires commitment of using results in production → Requires interfacing between: →R&D →IT services →Library systems → And: what do the users want?
- 54. @maxkaiser not necessarily those atthe Austrian National Library …
- 55. @maxkaiser → “Integration withcurrent infrastructure is complex” → “Legal implications are risky” → “There is still too much R&D involved to be integrated in production environment” → “There are no clear use cases” → “A shift from (human produced) meta-data to machine-understanding primary data is too fundamental” → “Text mining results are opaque (and statistics are hard to grasp)”
- 56.
- 57. @maxkaiser →Topic modelling andNER should be traversing institutional boundaries →Text mining techniques cover a wide spectrum of possible applications →User driven approach applied to material from multiple institutions Q: Who are the main drivers (Europeana?, institutional consortia?, the users?, researchers?)
- 58.
- 59. @maxkaiser → Lack ofintegration in library discovery systems → Limitations in available vendor systems → Difficulty to explain possible instabilities in results → results are expected to improve as corpora evolve → methodology, corpus and implementation are moving targets → New and different resources required → IT infrastructures, tech- and research staff, text mining competency → Progress is fast! → The expectation of even more potential in the future delays investment in this future, now.
- 60.
- 61. @maxkaiser →Content discovery →Recommender systems(for librarians, for patrons) →Focused user-driven aggregation of content →Cross referencing of NERs in materials →Crowd-sourcing and user-driven development
- 62.
- 63.