Downloaded 11 times
![ContentMine: We use machines to
liberate[1] 100 million facts /yr from
the scientific scholarly literature and
make them free for everyone
(WikiData)
WikiData and ContentMines are social
machines
There are no longer any technical
obstacles, only people.
[1] Friday workshop: build your own social machine: scraping XML,](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-2-320.jpg)
![http://www.budapestopenaccessinitiative.org/read
… an unprecedented public good. …
… completely free and unrestricted access to [peer-reviewed
literature] by all scientists, scholars, teachers,
students, and other curious minds. …
…Removing access barriers to this literature will
accelerate research, enrich education, share the
learning of the rich with the poor and the poor with
the rich, make this literature as useful as it can be, and
lay the foundation for uniting humanity in a common
intellectual conversation and quest for knowledge.
(Budapest Open Access Initiative, 2003)](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-5-320.jpg)
![Scientific and Medical publication (STM)[+]
• World Citizens pay $400,000,000,000…
• … for research in 1,500,000 articles …
• … cost $300,000 each to create …
• … $7000 each to “publish” [*]…
• … $10,000,000,000 from academic libraries …
• … to “publishers” who forbid access to 99.9% of
citizens of the world …
[+] Figures probably +- 50 %
[*] arXiV preprint server costs $7 USD per paper](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-6-320.jpg)
![petermr: I believe in Wikipedia
• 2006 http://en.wikipedia.org/wiki/User:Petermr
• 2006 started Open Data (term unknown then!)
• 2009: “the bit of Wikipedia that I wrote is correct” [challenging the
idea of “WP is junk”]
• 2009: “Wikipedia is the digital library of this century”
• 2012: I alert WP that Springer has copyrighted > 1000 of our
images [Springergate]
• 2014: “For facts in maths, physical and biological sciences I trust
Wikipedia.” (Wikimania2014)](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-7-320.jpg)
![Bad publication wastes science
…three problems—flawed design, non-publication,
and poor reporting—together
meant >85% of research funds were wasted, a
global total loss >100 billion USD per year.
[Lancet 2009]
[Even more] waste clearly occurs after
publication: from poor access, poor
dissemination, and poor uptake of the findings
of research. [PLOS Medicine 2014-05-27]](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-12-320.jpg)
![Elsevier wants to control Open Data
[asked by Michelle Brook]](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-14-320.jpg)
![Licences destroy Content Mining
WE WALKED OUT
• Brit Library
• JISC
• RLUK
• OKFN
• …
• Ross Mounce
• PM-R
STM Publishers Licence
2012_03_15_Sample_Licence_Text_Data_Mining.pdf
(Summary: PMR has NO rights)
• [cannot publish to: ] “libraries, repositories, or archives”
• [cannot] “Make the results of any TDM Output available on an externally facing server or
website”
• “Subscriber shall pay a […] fee”
Heather Piwowar: “negotiating with publishers [made me physically ill]”](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-15-320.jpg)
![Open notebook science is the practice of
making the entire primary record of a research
project publicly available online as it is
recorded. (WP)
Jean-Claude Bradley was a chemist who
actively promoted Open Science in
chemistry,… He coined the term Open
Notebook Science. … A memorial
symposium was held July 14, 2014 at
Cambridge University, UK.[9]](https://image.slidesharecdn.com/chicago20141112-141113140242-conversion-gate01/85/ContentMine-Open-Data-and-Social-Machines-53-320.jpg)
More Related Content
Similar to ContentMine: Open Data and Social Machines
More from petermurrayrust
ContentMine: Open Data and Social Machines
- 1. ContentMine: Open data And Social Machines Peter Murray-Rust , Computation Lab, Univ of Chicago, 2014-11-12
- 2. ContentMine: We usemachines to liberate[1] 100 million facts /yr from the scientific scholarly literature and make them free for everyone (WikiData) WikiData and ContentMines are social machines There are no longer any technical obstacles, only people. [1] Friday workshop: build your own social machine: scraping XML,
- 3.
- 4. http://en.wikipedia.org/wiki/Tim_Berners-Lee Everything inthis presentation is ODOSOS (Open Data, Open Standards, Open Source) CC0, CC-BY, W3C etc., Apache2, etc. Open = “Free to use, re-use and redistribute http://contentmine.org http://bitbucket.org/petermr http://wwmm.ch.cam.ac.uk A promise: I (Petermr) will never sell out to non-transparent organizations.
- 5. http://www.budapestopenaccessinitiative.org/read … anunprecedented public good. … … completely free and unrestricted access to [peer-reviewed literature] by all scientists, scholars, teachers, students, and other curious minds. … …Removing access barriers to this literature will accelerate research, enrich education, share the learning of the rich with the poor and the poor with the rich, make this literature as useful as it can be, and lay the foundation for uniting humanity in a common intellectual conversation and quest for knowledge. (Budapest Open Access Initiative, 2003)
- 6. Scientific and Medicalpublication (STM)[+] • World Citizens pay $400,000,000,000… • … for research in 1,500,000 articles … • … cost $300,000 each to create … • … $7000 each to “publish” [*]… • … $10,000,000,000 from academic libraries … • … to “publishers” who forbid access to 99.9% of citizens of the world … [+] Figures probably +- 50 % [*] arXiV preprint server costs $7 USD per paper
- 7. petermr: I believein Wikipedia • 2006 http://en.wikipedia.org/wiki/User:Petermr • 2006 started Open Data (term unknown then!) • 2009: “the bit of Wikipedia that I wrote is correct” [challenging the idea of “WP is junk”] • 2009: “Wikipedia is the digital library of this century” • 2012: I alert WP that Springer has copyrighted > 1000 of our images [Springergate] • 2014: “For facts in maths, physical and biological sciences I trust Wikipedia.” (Wikimania2014)
- 8. A meritocratic critical volunteer community
- 9. Volunteer community inchemistry: Open Data/Source/Standards
- 10. 4 Billion USDon human genome yielded 800 Billion USD and 4 M job-years
- 11.
- 12. Bad publication wastesscience …three problems—flawed design, non-publication, and poor reporting—together meant >85% of research funds were wasted, a global total loss >100 billion USD per year. [Lancet 2009] [Even more] waste clearly occurs after publication: from poor access, poor dissemination, and poor uptake of the findings of research. [PLOS Medicine 2014-05-27]
- 13. Publishers’ PDFs destroyscience PDFs do not contain words or subscripts! PDFs do not contain tables and do not have columns SVG is turned into JPEG because it’s easier to process
- 14. Elsevier wants tocontrol Open Data [asked by Michelle Brook]
- 15. Licences destroy ContentMining WE WALKED OUT • Brit Library • JISC • RLUK • OKFN • … • Ross Mounce • PM-R STM Publishers Licence 2012_03_15_Sample_Licence_Text_Data_Mining.pdf (Summary: PMR has NO rights) • [cannot publish to: ] “libraries, repositories, or archives” • [cannot] “Make the results of any TDM Output available on an externally facing server or website” • “Subscriber shall pay a […] fee” Heather Piwowar: “negotiating with publishers [made me physically ill]”
- 16. CLOSED ACCESS MEANSPEOPLE DIE CLOSED DATA MEANS PEOPLE DIE
- 17.
- 18. The scientist’s amanuensis • "The bane of my life is doing things I know computers could do for me" (Dan Connolly, W3C) Example: A semantic amanuensis could • Give me a daily digest of mineralogy papers • Extract all the crystal structures from them • Compute physical properties with GULP and NWChem • Compare the results statistically • Preserve and distribute the complete operation • Prepare the results for publication The semantic web is having a personal amanuensis
- 19. Artificial Intelligence inscience In 1970 chess and chemistry were the sandboxes for AI. Some approaches: • Lookup (Knowledge) • Natural Language Processing (NLP) • Brute force calculation (inc. physical methods) • Tree-pruning and heuristics • Logic (cf. OWL-DL) • Human-machine integration (crowdsourcing) • Computer Vision Domain-specific Turing test: Can a machine pass a first-year chemistry exam?
- 20. The Semantic Web "The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation." Tim Berners-Lee, James Hendler, Ora Lassila, The Semantic Web, Scientific American, May 2001 CC-BY-SA Images from Wikipedia
- 21. Linked Open datafrom Wikipedia “Which Rivers flow into the Rhine and are longer than 50 kilometers?” or “Which Skyscrapers in China have more than 50 floors and have been constructed before the year 2000?” Open Crystallography? “Which countries where tropical diseases are endemic have published structures of chiral natural products?” CC-BY-SA from Wikipedia
- 22. The Right toRead is the Right to Mine http://contentmine.org
- 23. • Science canbe read and understood by human-machine Amanuensis-symbionts. • Amanuenses are based on Wikipedia, databases and software (e.g. ContentMine’s AMI) • The results are fed back into WP and WikiData http://en.wikipedia.org/wiki/Eric_Fenby http://en.wikipedia.org/wiki/Symbiosis
- 24. Machine Extraction ofscientific facts • Crawl scientific literature (Open Bibliography) • Scrape each scientific article (ContentMine-quickscrape) • Extract the facts (ContentMine-AMI) • Index (Wikipedia) • Republish (WikiData)
- 25. RSU: Richard Smith-Unna PMR: Peter Murray-Rust CL: CottageLabs Queues Repos Scientific literature Science Plugins Science Volunteers
- 26. Linked Open Data– the world’s knowledge GOV.uk very little physical science DBPedia BIO http://upload.wikimedia.org/wikipedia/commons/3/34/LOD_Cloud_Diagram_as_of_September_2011.png Lib Comp PDB Ontologies GOV Music, Art Literature Social Knowledge bases RDF triples
- 27. Part of aCOD RDF entry The Semantic Web understands this
- 28. Mathematics Markup Language Energy of c.c.p lattice of argon Human-friendly 4 pages clipped Machine-friendly Many editors and tools exist We used MathWeaver Automatic! MathML
- 29. CML (Chemical MarkupLanguage) Automatic! Human-friendly Machine-friendly
- 30. Innovation with Componentisation Individual, manual, unreusable, flaky Commodity, standard, reliable, re-usable
- 31. Current scientific informationflow … is broken for data-rich science Non-semantic data Human input Data extraction difficult and incomplete Human readers PDF Lineprinter output Text files
- 32. Semantic network closesthe loop Data mined from document Computation Measurement Semantic Authoring Community Data available for e-science and re-use Analysis
- 33. The network growsautonomously Machine-machine Human-machine Machine-human Human-human
- 34. Humans and machinesuse different languages
- 35. How a machinereads a chemical thesis nodes are compounds; arrows are reactions
- 36. Human-machine symbionts canread science! WP_Lion WP_Aspergillus_oryzae WP_Soybean
- 37. With Wikipedia everyonecan be a scientist Facts Marked by “non-scientists” in ContentMine workshops
- 38. “nuggets” in ascientific paper project places quantity units Value ranges chemical Humans aren’t designed to mine this …
- 39. Parsing chemical sentences A FACT, uncopyrightable, and representable by triples
- 40. http://wwmm.ch.cam.ac.uk/chemicaltagger • Typical Typical chemical synthesis
- 41. Open Content Miningof FACTs Machines can interpret chemical reactions We have done 500,000 patents. There are > 3,000,000 reactions/year. Added value > 1B Eur.
- 42. We can’t turna hamburger into a cow But we can now turn PDFs into Science
- 43. UNITS TICKS QUANTITY SCALE TITLES DATA!! 2000+ points
- 44. Dumb PDF CSV Semantic Spectrum Automatic extraction Takes < 1 second Gaussian Filter 2nd Derivative
- 45. Chemical Computer Vision Raw Mobile photo; problems: Shadows, contrast, noise, skew, clipping
- 46. Binarization (pixels =0,1) Irregular edges
- 47. Thinning: thick linesto 1-pixel
- 48. Chemical Optical CharacterRecognition Small alphabet, clean typefaces, clear boundaries make this relatively tractable. Problems are “I” “O” etc.
- 49. AMI https://bitbucket.org/petermr/xhtml2stm/wiki/Home AMIreads the complete diagram, recognizes the paths and generates the molecules. Then she creates a stop-fram animation showing how the 12 reactions lead into each other CLICK HERE FOR ANIMATION (may be browser dependent) Example reaction scheme, taken from MDPI Metabolites 2012, 2, 100-133; page 8, CC-BY:
- 50. AMI Demo http://www.mdpi.com/2218-1989/2/1/39/pdf https://bitbucket.org/AndyHowlett/ami2-poc ami2-poc -i example -v org.xmlcml.xhtml2stm.visitor.chem.ChemVisitor May take time to start if not connected to web Output:./target/output/reactionsexample/ SVG: ./page1annotated.svg CML: image.g.1.4.svg.reaction0.cml Avogadro Viewer:
- 51. Bacterial WP_phylogenetic tree Genbank ID American Type Culture Collection WP: Clostridium_butyricum Our machines have read and interpreted 4300 in an hour with > 95% accuracy Trees From http://ijs.sgmjournals.org/ used under new UK legislation (Hargreaves)
- 52. http://en.wikipedia.org/wiki/Digital_image_processing http://en.wikipedia.org/wiki/Newick_format http://en.wikipedia.org/wiki/Phylogenetics ((n122,((n121,n205),((n39,(n84,((((n35,n98),n191),n22),n17))),((n10,n182),( (((n232,n76),n68),(n109,n30)),(n73,(n106,n58))))))),((((((n103,n86),(n218,(n 215,n157))),((n164,n143),((n190,((n108,n177),(n192,n220))),((n233,n187), n41)))),((((n59,n184),((n134,n200),(n137,(n212,((n92,n209),n29))))),(n88,(n 102,n161))),((((n70,n140),(n18,n188)),(n49,((n123,n132),(n219,n198)))),((( n37,(n65,n46)),(n135,(n11,(n113,n142)))),(n210,((n69,(n216,n36)),(n231,n1 60))))))),(((n107,n43),((n149,n199),n74)),(((n101,(n19,n54)),n96),(n7,((n139 ,n5),((n170,(n25,n75)),(n146,(n154,(n194,(((n14,n116),n112),(n126,n222))) )))))))),(((((n165,(n168,n128)),n129),((n114,n181),(n48,n118))),((n158,(n91,( n33,n213))),(n87,n235))),((n197,(n175,n117)),(n196,((n171,(n163,n227)),(( n53,n131),n159))))))); (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0036933 – “Adaptive Evolution of HIV at HLA Epitopes Is Associated with Ethnicity in Canada” .
- 53. Open notebook scienceis the practice of making the entire primary record of a research project publicly available online as it is recorded. (WP) Jean-Claude Bradley was a chemist who actively promoted Open Science in chemistry,… He coined the term Open Notebook Science. … A memorial symposium was held July 14, 2014 at Cambridge University, UK.[9]
- 54. RSU: Richard Smith-Unna PMR: Peter Murray-Rust CL: CottageLabs Queues Repos Scientific literature Science Plugins Science Volunteers
- 55. Thanks • ShuttleworthFoundation and Fellowship • Contentmine.org: Michelle Brook, Jenny Molloy, Ross Mounce, Richard Smith-Unna, CottageLabs, Charles Oppenheim • Open Knowledge Foundation Community • Wikimedia Community • Blue Obelisk Community
- 56. My/our Dream •An Open Bibliography of science, updated daily • An interface for ContentMine to feed new facts into WikiData • Domain-specific enthusiasts to create and run fact extraction and validation • Wikipedia to become a C21 publisher of reference science