0
Weaving a New Web for Chemistry  Antony Williams
Structure-Enabled Articles
Searching from the Structure Balloon
Imagine a time when …. <ul><li>The internet is searchable by chemical structure and substructure </li></ul><ul><li>Chemist...
ChemSpider - A Search Engine for Chemists <ul><li>Questions a chemist might ask… </li></ul><ul><ul><li>What is the melting...
ChemSpider Data Content <ul><li>Over 21.5 million unique chemical structures from ca. 150 data sources </li></ul><ul><ul><...
Tell me about Aspirin
Tell me about Aspirin
Link outs
Links out to KEGG Kyoto Encyclopedia of Genes and Genomes
Tell me about Aspirin
Tell me About Aspirin
Tell me about Aspirin
Tell me about Aspirin
Tell me about Aspirin
Text- Indexing  and ChemSpider? <ul><li>ChemSpider text-indexes almost 500,000 Open Access and Free Access articles </li><...
Open Access Literature Search
Search PubMed – ChemSpider
Other Searches <ul><li>What compounds have a mass of 300+/-0.001? </li></ul><ul><li>or search a combination of intrinsic/p...
Other Searches
Complex Search
The Quality of Data Online… <ul><li>Aggregating data opens up quality issues </li></ul><ul><li>Structure-identifier associ...
Who holds THE Quality Authority? <ul><li>Chemical Abstracts Service is the structural authority today. 1400 (?) employees,...
Quality is a Major Issue- Search Butanol
Wikipedia – Crowdsourcing Chemistry
Wikipedia Chemistry Curation project <ul><li>Only ca. 5000 organic structures, 7000 total structures </li></ul><ul><li>MON...
Thymol Blue on ChemSpider <ul><li>Data online includes: </li></ul><ul><ul><li>UV-vis spectrum </li></ul></ul><ul><ul><li>M...
Differences between ChemSpider/Wikipedia No Analytical Data Active editors – about 50 (?) Active depositors/curators – 30 ...
Differences between Wikipedia/ChemSpider Growing reputation as focused on quality Worldwide reputation as quality source C...
Crowd-sourcing Curation <ul><li>How to curate data for millions of structures?  </li></ul><ul><li>Robot processes can clea...
Multi-level Curation and Approval
Post Comments <ul><li>Anyone can “Post Comments” associated with a structure. To curate data we require login to track </l...
Crowd-sourcing Chemistry <ul><li>Crowd-sourced curation: identify and tag errors, edit names, synonyms, identify records f...
But, when registered and logged in… <ul><li>Ability to curate and add to the database </li></ul><ul><ul><li>Add structures...
Adding to the Database - Structure
Adding New Text Data Add Publication Add Identifier Add URL
Adding Supplementary Info to a Structure
ChemSpider TouchGraph
Structure-Centric  <ul><li>We want to search Open-Access articles by structure, substructure, similarity of structure </li...
“Entity Extraction” <ul><li>Rule-based recognition of systematic names: </li></ul><ul><ul><li>Use a lexeme of name fragmen...
Name Recognition <ul><li>Azo aldehyde  2   was  synthesized according to a reported  method [17]. To  a stirred  solution ...
Name Recognition <ul><li>Azo aldehyde  2   was  synthesized according to a reported  method [17]. To  a stirred  solution ...
How Many Chemical Names? <ul><li>“ She had the drive to derive success in any venture and was well versed in Karate. When ...
How Many Chemical Names? <ul><li>“ She had the  drive  to derive  success  in any venture and was well  versed  in  Karate...
Making Open Access Articles Searchable Proof of Concept <ul><li>Can we HOST Chemistry Open Access articles on ChemSpider a...
Document markup <ul><li>ChemSpider now hosting Open Access articles from MDPI, Molecular Diversity Preservation Internatio...
A Standard for Document Markup? <ul><li>NLM-DTD: National Library of Medicine; Document Type Definition </li></ul><ul><li>...
NLM/DTD markup
Chemistry and Biology
Chemistry and Biology <ul><li>Menus can be extended as necessary  </li></ul>
Document markup
Searching from the Structure Balloon
A Platform for Markup <ul><li>Can we provide a platform for document markup for chemists? </li></ul><ul><li>Workflow: </li...
Online Markup
Automated markup
Name to Structure Conversion
Conversion of Structure Images <ul><li>Not all compounds have a “name” </li></ul><ul><li>Structure images can be converted...
Cryptomisrine
Structure Conversion from Images-CLiDE <ul><li>Conversion dependent on zoom-factor can give perfect conversion! </li></ul>
Supports Word .DOC, HTML, RTF
Extensible Markup Process  <ul><li>Markup process is easily extendable </li></ul><ul><li>Configurable from one XML file </...
Tipping Point <ul><li>Tipping point  - the  point  at which a slow gradual change becomes irreversible and then proceeds w...
Our Challenges <ul><li>There are “no employees” </li></ul><ul><li>ChemSpider is non-funded </li></ul><ul><li>System is hyp...
What’s Coming? <ul><li>Agreement with Royal Society of Chemistry that we can add their structure-based RSS feeds to ChemSp...
Conclusions <ul><li>The quality of structure-based data online should always be questioned – that includes ChemSpider </li...
Further reading <ul><li>www.chemspider.com/blog </li></ul><ul><li>Internet-based tools for communication and collaboration...
ChemSpider Forums/Blogs <ul><li>Forum.chemspider.com </li></ul><ul><li>www.chemspider.com/blog </li></ul>
Acknowledgments  <ul><li>The ChemSpider team of volunteer developers </li></ul><ul><li>ChemSpider Advisory Group </li></ul...
Upcoming SlideShare
Loading in...5
×

Web Crawling Chemistry

1,098

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,098
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Web Crawling Chemistry"

  1. 1. Weaving a New Web for Chemistry Antony Williams
  2. 2. Structure-Enabled Articles
  3. 3. Searching from the Structure Balloon
  4. 4. Imagine a time when …. <ul><li>The internet is searchable by chemical structure and substructure </li></ul><ul><li>Chemistry articles are indexed and searchable by a free online service </li></ul><ul><li>Publicly funded research data can be shared and discussed in the Open </li></ul><ul><li>Cheminformatics has as much of a public face as bioinformatics </li></ul>
  5. 5. ChemSpider - A Search Engine for Chemists <ul><li>Questions a chemist might ask… </li></ul><ul><ul><li>What is the melting point of n-butanol? </li></ul></ul><ul><ul><li>What is the chemical structure of Xanax? </li></ul></ul><ul><ul><li>Chemically, what is phenolphthalein? </li></ul></ul><ul><ul><li>What are the stereocenters of cholesterol? </li></ul></ul><ul><ul><li>Where can I find publications about xylene? </li></ul></ul><ul><ul><li>What are the different trade names for Ketoconazole? </li></ul></ul><ul><ul><li>What is the NMR spectrum of Aspirin? </li></ul></ul><ul><ul><li>What are the safety handling issues for Thymol Blue? </li></ul></ul><ul><ul><li>ChemSpider can answer all of these questions </li></ul></ul>
  6. 6. ChemSpider Data Content <ul><li>Over 21.5 million unique chemical structures from ca. 150 data sources </li></ul><ul><ul><li>Online Databases –PubChem, Drugbank, HMDB, Wikipedia </li></ul></ul><ul><ul><li>Chemical Vendors – over 40 different vendors and growing </li></ul></ul><ul><ul><li>Personal Depositions – individual contributions </li></ul></ul><ul><ul><li>Journal Publishers </li></ul></ul><ul><ul><li>Content database vendors </li></ul></ul><ul><ul><li>Analytical data collections </li></ul></ul><ul><ul><li>Patents (9 MILLION Structures being deposited now ) </li></ul></ul><ul><ul><li>Web scraping </li></ul></ul><ul><ul><li>Content is generally linked back to the original data sources </li></ul></ul>
  7. 7. Tell me about Aspirin
  8. 8. Tell me about Aspirin
  9. 9. Link outs
  10. 10. Links out to KEGG Kyoto Encyclopedia of Genes and Genomes
  11. 11. Tell me about Aspirin
  12. 12. Tell me About Aspirin
  13. 13. Tell me about Aspirin
  14. 14. Tell me about Aspirin
  15. 15. Tell me about Aspirin
  16. 16. Text- Indexing and ChemSpider? <ul><li>ChemSpider text-indexes almost 500,000 Open Access and Free Access articles </li></ul><ul><li>Collection is growing weekly and more publishers have already agreed </li></ul>
  17. 17. Open Access Literature Search
  18. 18. Search PubMed – ChemSpider
  19. 19. Other Searches <ul><li>What compounds have a mass of 300+/-0.001? </li></ul><ul><li>or search a combination of intrinsic/predicted properties </li></ul>
  20. 20. Other Searches
  21. 21. Complex Search
  22. 22. The Quality of Data Online… <ul><li>Aggregating data opens up quality issues </li></ul><ul><li>Structure-identifier associations are “dirty” </li></ul><ul><li>Structures are COMMONLY incorrect – stereochem issues </li></ul><ul><li>Manual curation of small databases is enough work – what about millions of structures? </li></ul><ul><li>Structures are far from perfect. What is a “correct structure”? </li></ul><ul><ul><li>Full stereochemistry? </li></ul></ul><ul><ul><li>Historical timeline of structure? </li></ul></ul><ul><ul><li>Who is the authority? </li></ul></ul>
  23. 23. Who holds THE Quality Authority? <ul><li>Chemical Abstracts Service is the structural authority today. 1400 (?) employees, world standard in chemistry information </li></ul><ul><li>101 years of knowledge, process and expertise. MANUAL curation is key. Robotic curation is enabling </li></ul><ul><li>How can an online, free access system peacefully co-exist with the authority? </li></ul>
  24. 24. Quality is a Major Issue- Search Butanol
  25. 25. Wikipedia – Crowdsourcing Chemistry
  26. 26. Wikipedia Chemistry Curation project <ul><li>Only ca. 5000 organic structures, 7000 total structures </li></ul><ul><li>MONTHS of work so far for a team of 6 people </li></ul><ul><li>Many errors removed in the process. Curation process is a daily event for users/depositors </li></ul><ul><li>Slow and torturous process for stereo molecules. </li></ul>
  27. 27. Thymol Blue on ChemSpider <ul><li>Data online includes: </li></ul><ul><ul><li>UV-vis spectrum </li></ul></ul><ul><ul><li>Measured experimental properties </li></ul></ul><ul><ul><li>Link to Wikipedia article </li></ul></ul><ul><ul><li>Links to chromatography details </li></ul></ul><ul><ul><li>Multiple identifiers/trade names etc. </li></ul></ul><ul><ul><li>Links to vendors/suppliers/other databases </li></ul></ul><ul><ul><li>Safety information </li></ul></ul>
  28. 28. Differences between ChemSpider/Wikipedia No Analytical Data Active editors – about 50 (?) Active depositors/curators – 30 No Prediction of properties ???? 5000 people/day; 1100 registered Detailed compound monographs Compound monographs linked Text Complex queries – Properties, Text, structure/substructure, OA publishers, Data Sources, … ~5000 organics, 2000 others >21 million unique structures Wikipedia ChemSpider
  29. 29. Differences between Wikipedia/ChemSpider Growing reputation as focused on quality Worldwide reputation as quality source Chemistry is the focus of ‘Spider Chemistry is a subset of the ‘Pedia Mixed “licensing” GFL licensing for everything Growing team of WP:Chem advocates, curators and admins Strong team of WP:Chem advocates, curators and admins “ Out of a basement” on three servers and 5 volunteers Established infrastructure and Wikipedia Foundation Team Primarily Microsoft .NET technologies with OS components Supported by tried and tested Media-Wiki platform. ChemSpider Wikipedia
  30. 30. Crowd-sourcing Curation <ul><li>How to curate data for millions of structures? </li></ul><ul><li>Robot processes can clean up depositions </li></ul><ul><ul><li>Search for Chloride and check molecular formula for Cl </li></ul></ul><ul><ul><li>Check for stereochemistry and remove names with stereo </li></ul></ul><ul><li>Provide a simple-to-use platform to curate, annotate and tag data </li></ul><ul><li>Provide curator administration to prevent vandalism (Veropedia) </li></ul>
  31. 31. Multi-level Curation and Approval
  32. 32. Post Comments <ul><li>Anyone can “Post Comments” associated with a structure. To curate data we require login to track </li></ul>
  33. 33. Crowd-sourcing Chemistry <ul><li>Crowd-sourced curation: identify and tag errors, edit names, synonyms, identify records for deprecation </li></ul><ul><li>ALSO </li></ul><ul><li>Crowd-sourced deposition: anyone can deposit data (structures, text, images, analytical data) </li></ul>
  34. 34. But, when registered and logged in… <ul><li>Ability to curate and add to the database </li></ul><ul><ul><li>Add structures </li></ul></ul><ul><ul><li>“ Clean” structures </li></ul></ul><ul><ul><li>Add data (spectra, CIFs, images) </li></ul></ul><ul><ul><li>Add links to other pages (URLs) </li></ul></ul><ul><ul><li>Add publication details </li></ul></ul>
  35. 35. Adding to the Database - Structure
  36. 36. Adding New Text Data Add Publication Add Identifier Add URL
  37. 37. Adding Supplementary Info to a Structure
  38. 38. ChemSpider TouchGraph
  39. 39. Structure-Centric <ul><li>We want to search Open-Access articles by structure, substructure, similarity of structure </li></ul><ul><li>Standard approaches would be: </li></ul><ul><ul><li>Identify chemical names “entity extraction” </li></ul></ul><ul><ul><li>Convert chemical names to structures and index </li></ul></ul><ul><li>ChemSpider has a validated dictionary of structure-name pairs </li></ul><ul><li>Use name extraction, name-conversion and dictionary look-up. THEN curate. </li></ul>
  40. 40. “Entity Extraction” <ul><li>Rule-based recognition of systematic names: </li></ul><ul><ul><li>Use a lexeme of name fragments </li></ul></ul><ul><ul><li>Rules for identifying bounds of a name </li></ul></ul><ul><li>Look-up dictionary: </li></ul><ul><ul><li>Drug Names </li></ul></ul><ul><ul><li>Trivial Names </li></ul></ul><ul><ul><li>Numbers : Registry IDs, EINECS/ELINCS/Beilstein IDs </li></ul></ul>
  41. 41. Name Recognition <ul><li>Azo aldehyde 2   was  synthesized according to a reported  method [17]. To  a stirred  solution  of azo aldehyde 2   (1.08 g, 3.76 mmol )  in  dry CH2Cl2  (30.00 mL) at  0 oC  were  successively  added (3,4-diaminophenyl)phenyl methanone 1 (0.40 g, 1.88 mmol) and a excces of anhydrous MgSO4 (2.00 g,16.67 mmol) . The resulting  mixture  was  stirred  for  6 hours  at room temperature [18]. The mixture was  filtered and washed with dichloromethane . Then the solvent was  evaporated under reduced pressure to  give azo Schiff base 3   as a red solid which was recrystalized from ethanol 95%    (1.28 g, 91 %) </li></ul>
  42. 42. Name Recognition <ul><li>Azo aldehyde 2   was  synthesized according to a reported  method [17]. To  a stirred  solution  of azo aldehyde 2   (1.08 g, 3.76 mmol )  in  dry CH2Cl2   (30.00 mL) at  0 oC  were  successively  added  (3,4-diaminophenyl)phenyl methanone 1 (0.40 g, 1.88 mmol) and a excess of anhydrous MgSO 4 (2.00 g,16.67 mmol) . </li></ul><ul><li>The resulting  mixture  was  stirred  for  6 hours  at room temperature [18]. The mixture was  filtered and washed with dichloromethane . Then the solvent was  evaporated under reduced pressure to  give azo Schiff base 3   as a red solid which was recrystalized from ethanol 95%    (1.28 g, 91 %) </li></ul>
  43. 43. How Many Chemical Names? <ul><li>“ She had the drive to derive success in any venture and was well versed in Karate. When the man in the tartan shirt approached her with a dagger in his hand she spat in his face, took the stance of a commando and took advantage of his shock to release the dagger from his grip and causing him to recoil. He went home and took an aspirin after the beating.” </li></ul>
  44. 44. How Many Chemical Names? <ul><li>“ She had the drive to derive success in any venture and was well versed in Karate . When the man in the tartan shirt approached her with a dagger in his hand she spat in his face, took the stance of a commando and took advantage of his shock to release the dagger from his grip and causing him to recoil . He went home and took an aspirin after the beating.” </li></ul>
  45. 45. Making Open Access Articles Searchable Proof of Concept <ul><li>Can we HOST Chemistry Open Access articles on ChemSpider and add-value </li></ul><ul><li>Can we identify chemical names in Open Access articles in a user-friendly manner </li></ul><ul><li>Can we convert names to structures in Open-Access articles and expand ChemSpider and provide structure searching of Open Access chemistry articles? </li></ul><ul><li>Can we provide an environment for chemists to mark-up their own articles and crowd-source markup of an archive? </li></ul>
  46. 46. Document markup <ul><li>ChemSpider now hosting Open Access articles from MDPI, Molecular Diversity Preservation International </li></ul><ul><li>Hosting the Molbank collection at present </li></ul>
  47. 47. A Standard for Document Markup? <ul><li>NLM-DTD: National Library of Medicine; Document Type Definition </li></ul><ul><li>Approved markup definitions to apply to journal articles – extended as necessary for our purposes </li></ul>
  48. 48. NLM/DTD markup
  49. 49. Chemistry and Biology
  50. 50. Chemistry and Biology <ul><li>Menus can be extended as necessary </li></ul>
  51. 51. Document markup
  52. 52. Searching from the Structure Balloon
  53. 53. A Platform for Markup <ul><li>Can we provide a platform for document markup for chemists? </li></ul><ul><li>Workflow: </li></ul><ul><ul><li>Upload word docs, RTF files or point to HTML and load </li></ul></ul><ul><ul><li>Apply entity extraction, convert names to structures, mark-up automatically and ask for user participation </li></ul></ul><ul><ul><li>Publish final version with NLM-DTD markup </li></ul></ul><ul><ul><li>Deposit all structures on ChemSpider under embargo and wait for article DOI to release </li></ul></ul>
  54. 54. Online Markup
  55. 55. Automated markup
  56. 56. Name to Structure Conversion
  57. 57. Conversion of Structure Images <ul><li>Not all compounds have a “name” </li></ul><ul><li>Structure images can be converted to connection tables </li></ul>
  58. 58. Cryptomisrine
  59. 59. Structure Conversion from Images-CLiDE <ul><li>Conversion dependent on zoom-factor can give perfect conversion! </li></ul>
  60. 60. Supports Word .DOC, HTML, RTF
  61. 61. Extensible Markup Process <ul><li>Markup process is easily extendable </li></ul><ul><li>Configurable from one XML file </li></ul><ul><li>NLM/DTD is incorporated but is easy to extend </li></ul>
  62. 62. Tipping Point <ul><li>Tipping point - the point at which a slow gradual change becomes irreversible and then proceeds with gathering pace </li></ul>
  63. 63. Our Challenges <ul><li>There are “no employees” </li></ul><ul><li>ChemSpider is non-funded </li></ul><ul><li>System is hyper-dependent on ISP, power and limited compute power </li></ul><ul><li>We are upsetting some people – specifically “closed” data content providers </li></ul>
  64. 64. What’s Coming? <ul><li>Agreement with Royal Society of Chemistry that we can add their structure-based RSS feeds to ChemSpider </li></ul><ul><li>Agreement with Nature Publishing Group to add their Nature Chemical Biology structure collections to ChemSpider as they issue </li></ul><ul><li>Presently indexing Acta Chemica Scandanavica, 1947-1999 PDF backfile – our first foray into OCR </li></ul><ul><li>Presently indexing PLoS journals directly </li></ul><ul><li>More publishers have agreed… </li></ul>
  65. 65. Conclusions <ul><li>The quality of structure-based data online should always be questioned – that includes ChemSpider </li></ul><ul><li>Robots and software algorithms can help but eyeballs are necessary </li></ul><ul><li>Data on ChemSpider are being added and curated on a daily basis but we need more eyeballs helping always </li></ul><ul><li>ChemSpider now has a large validated structure-name dictionary </li></ul>
  66. 66. Further reading <ul><li>www.chemspider.com/blog </li></ul><ul><li>Internet-based tools for communication and collaboration in chemistry, Drug Discovery Today, Volume 13, Numbers 11/12, June 2008 502-506, doi:10.1016/j.drudis.2008.03.015 </li></ul><ul><li>A perspective of publicly accessible/open-access chemistry databases, Drug Discovery Today, Volume 13, Numbers 11/12, June 2008, 495-501, doi:10.1016/j.drudis.2008.03.017 </li></ul>
  67. 67. ChemSpider Forums/Blogs <ul><li>Forum.chemspider.com </li></ul><ul><li>www.chemspider.com/blog </li></ul>
  68. 68. Acknowledgments <ul><li>The ChemSpider team of volunteer developers </li></ul><ul><li>ChemSpider Advisory Group </li></ul><ul><li>Our curators, depositors and users </li></ul><ul><li>Suppliers of commercial software – Microsoft, ACD/Labs, OpenEye, ChemAxon, SimBioSys </li></ul><ul><li>SureChem – Structure Based Online Patent Searching </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×