Downloaded 14 times
More Related Content
Similar to Content Mining for Machines and Humans
More from petermurrayrust
![Polymer [ बहुलक ] Chemistry Notes PDF - Irfanullah Mehar - JJ Sir Chemistry.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/polymerchemistrynotespdf-irfanullahmehar-jjsirchemistry-260210172118-3f9b37f7-thumbnail.jpg?width=640&height=640&fit=bounds)
Content Mining for Machines and Humans
- 1. Content-Mining for Machinesand Humans Peter Murray-Rust contentmine.org WellcomeTrust, London, 2015-03-06 • Extract 100 million facts (CC0) from the scientific literature per year • Grow communities and give everyone the tools and know-how to mine science
- 2. What is “Content”? http://www.plosone.org/article/fetchObject.action?uri=info:doi/10.1371/journal.pone.01113 03&representation=PDFCC-BY SECTIONS MAPS TABLES CHEMISTRY TEXT MATH contentmine.org tackles these
- 3. Machine-Human symbioses • Wikipedia •Open StreetMap • Google We aim to make it trivial for a human+machine to mine the scientific literature. By building Communities
- 4. Workshops and Hackdays OpenScience Brazil, 2014-08 Easily distributed software Get started in 30 mins Build application in a morning Start simple: bagOfWords, Stemming, Regex, templates
- 5. Oxford 2013 Berlin 2014 Delhi2014 Jenny Molloy with mascot AMI
- 6. • CRAWL theweb for scientific documents (articles, grey literature, repositories) • quickSCRAPE pages (text, graphics, images, data) • NORMA-lize page to semantic form …Open semantic science … • MINE pages with your methods and tools (AMI) • CAT-alogue results in searchable index • Automate daily process (CANARY) contentmine.org Infrastructure
- 7. quickscrape Crawl Feed Norma Index & Transform PDF XML URL DOI Scientific literature RepositoriesDOC CSV sHTML Plugins Regex SequencesSpecies Bespoke Scrapers XPathPer-Journal Taggers Per- Journal MetadataChemistry Phylogenetics Farming AMI BadHTML OCR Diagrams Open NORMA-lized Scientific Literature + Facts CANARY pipeline CAT-alogue index
- 8. https://commons.wikimedia.org/wiki/File:Flickr_-_DVIDSHUB_-_RSP_Warrior_Challenge_Prepares_Soldiers_Mentally,_Physically_%281%29.jpg CRAWLing the Literature NOCentral Table of Contents Massive technical, political, legal opposition Little interest from Academia Tedious Few general tools
- 9. The Right toRead is The Right To Mine PMR in 2012: http://blog.okfn.org/2012/06/01/the-right-to-read-is-the-right-to-mine/
- 10. SCRAPE https://en.wikipedia.org/wiki/Gleaning#mediaviewer/File:Millet_Gleaners.jpg PublicDomain PDF HTML XML quickscrape* *Scraperscreated by Richard Smith-Unna + Community HTML PDF XML PNG SVG CSV DOC LaTeX CIF … Non-standard per-publisher site
- 11. https://en.wikipedia.org/wiki/W._Heath_Robinson#mediaviewer/File:Robinson%28WH%29-%28%27Uncle_Lubin%27%29.jpg PublicDomain NORMA-lization ofScientific Literature PDFs, Broken HTML PNGs for Math, etc. NORMA Unicode Diacritics Well-formed Sectioned Tagged SVG diagrams
- 12. AMI-plugins • BagOfWords, Stemmingand Regular Expressions • Species • Biological Sequences • Chemical compounds & reactions • Farming * (Rory Aaronson) • Crystallography * (Saulius Grazulis, COD) • Clinical Trials * (Amy Price) • Phylogenetics * (Ross Mounce) • Phytochemistry * (Chris Steinbeck, PMR) * subcommunities
- 13.
- 14. Open Content Miningof FACTs Machines can interpret chemical reactions We have done 500,000 patents. There are > 3,000,000 reactions/year. Added value > 1B Eur.
- 15.
- 16. Dumb PDF CSV Semantic Spectrum 2nd Derivative Smoothing GaussianFilter Automatic extraction
- 17. https://blogs.ch.cam.ac.uk/pmr/2014/06/25/content-mining-we-can-now- mine-images-of-phylogenetic-trees-and-more/ for storyof extraction Thinning Topology Serialization Newick
- 18. https://en.wikipedia.org/wiki/Irrigation#mediaviewer/File:Pump- enabled_Riverside_Irrigation_in_Comilla,_Bangladesh,_25_April_2014.jpg CC BY-SA3.0 Daily Stream of 100,000 Open Facts Twitter?Indexed by CAT
- 19. Phytochemistry extraction O. dayi “volatilecomposition of “ A.sibeiri A. judaica Displayed by CAT (CottageLabs)
- 20. Workshops (1-hour -> fullday or more) 2014-May->Nov • Budapest/Shuttleworth • Leicester Univ • Electronic Theses and Dissertations • Austrian Science Fund AT • OKFest DE • Eur. Bioinformatics Institute • Open Science Rio de Janeiro BR • Sci DataCon , Delhi IN • Univ of Chicago US • OpenCon 2014, Wash DC. US • JISC , London Upcoming • LIBER • Cochrane • BL • Wellcome Trust (April) • WHO Collaborators • Wikimedia/Wikidata • Mozilla • Open Knowledge • LIBER (European Research Libraries) • British Library • Wellcome Trust • EBI (Eur. Bioinf. Inst.) • JISC • Open Access Button • SPARC • Creative Commons • CORE
- 21. contentmine.org proposed Services •Workshops • Repository indexing • Funder Compliance • Publication enhancement • Extraction of scientific data
- 22.
- 24. Bacterial WP_phylogenetic tree Ourmachines have read and interpreted 4300 in an hour with > 95% accuracy Trees From http://ijs.sgmjournals.org/ used under new UK legislation (Hargreaves) WP: Clostridium_butyricum Genbank ID American Type Culture Collection
- 25.
- 26. RSU: Richard Smith-Unna PMR:Peter Murray-Rust CL: CottageLabs Queues Repos Scientific literature Science Plugins Science Volunteers Collaboration with Open Access Button
- 27. Daily stream of300,000 facts https://commons.wikimedia.org/wiki/File:Rapid_stream.jpg Public Domain
- 28.
- 29. AMI Demo http://www.mdpi.com/2218-1989/2/1/39/pdf https://bitbucket.org/AndyHowlett/ami2-poc ami2-poc -iexample -v org.xmlcml.xhtml2stm.visitor.chem.ChemVisitor May take time to start if not connected to web Output:./target/output/reactionsexample/ SVG: ./page1annotated.svg CML: image.g.1.4.svg.reaction0.cml Avogadro Viewer: