A 'skunkworks' project to re-purpose data from the published NIST-IUPAC Solubility Data volumes via a REST interface using CakePHP and MySQL

1. A REST API for
The IUPAC Solubility Data Series:
A ‘Skunkworks’ Project
Stuart J. Chalk
Department of Chemistry
University of North Florida
schalk@unf.edu
2014 Fall ACS Meeting

2.  Motivation
 What is Website ‘Scraping’?
 What are REST and API?
 Project Process
 NIST Website Analysis
 Database Definition
 Data Ingestion
 Project Website Design
 Using the Website
 Future Plans
 Conclusion
Outline

3.  Linked Open Data (LOD) is important for science
 Defining a process for grabbing high quality science
data and making it semantically available is useful
 Providing a REST API makes information easy to find
 Providing unique REST URLs for data allows linking
 A semantic description of data makes it more useful
 Increase value added -> link data to other available data
 SDS data is fundamentally important to chemistry
Motivation
(1) http://en.wikipedia.org/wiki/Linked_data

4.  Data in web pages is available for users to copy/paste
 When the available data is large, automation of the
scripts is necessary
 ‘Scraping’ is the processing of web page data using a
scripting language
 Data can be captured and stored in any format
 Most useful to capture data in a relational database
so that it can be repurposed at another website
 This is usually done without the permission of the
authors of the ‘scraped’ web page(s) 
What is Website Scraping?

5.  Representational State Transfer (REST) is…
“is a software architectural style consisting of a coordinated
set of architectural constraints applied to components,
connectors, and data elements, within a distributed
hypermedia system”2
 REST is applied to websites as a style for providing URL
access to information in a structured human readable way
 Application Programming Interface (API) is…
A standardized way for one computer/software system to
talk to another. For REST this a set of remote (http) based
calls to pre-defined URL’s
What are REST and API?
(2) http://en.wikipedia.org/wiki/Representational_state_transfer
(3) http://en.wikipedia.org/wiki/API

6.  Analysis of current NIST Solubility Database website
 Definition of database tables needed
 Code generation to automate data scraping
 Data cleanup
 REST API definition and description
 REST API development
 Output file format generation
 Addition of bells and whistles (if there’s time )
Project Process

7.  http://srdata.nist.gov/solubility/dataSeries.aspx
contains links to all the volumes that are available => volID
 http://srdata.nist.gov/solubility/sys_category.aspx
contains all the system types as part of a select list => typeID
 http://srdata.nist.gov/solubility/sol_sys_lst.aspx?sysID=<typeID>&FROM=
SSN contains the different datasets for a specific system type => sysID
 http://srdata.nist.gov/solubility/sol_detail.aspx?sysID=<sysID>
contains details of system: citation, data tables, refs, preparer etc.
 http://srdata.nist.gov/solubility/sol_2casno.aspx?
STR1=<CASRN1>&STR2=<CASRN2>&OPTION=CASNO allows searching by
chemical CASRN (also name (OPTION=CHEM) or formula (OPTION=MOL)
 http://srdata.nist.gov/solubility/citation_detail.aspx?REF_NO=<?REFNO?>
allows searching system date by paper
NIST Website Analysis

8.  What types of data are available and how should it be
organized?
 By Volume => volID
 By System Type => typeID
 By System => sysID
 By Chemical => CASRN, name, formula
 By Citation => refNO
 By Author (new)
 Also added Tables and Variables during development
 Note: the actual site uses sysID for the system and
type and particular set of data about a system type
Database Definition

9.  Data was imported into MySQL either from a tab
delimited text file or insertion via PHP scripts
 Scraped the volume id’s from
http://srdata.nist.gov/solubility/dataSeries.aspx html
cleaned up to generate a tab delimited text file
18 rows
 Similarly the system types were scraped from
http://srdata.nist.gov/solubility/sys_category.aspx
into a tab delimited text file => 2564 rows
Data Ingestion

10.  Individual systems with data were scraped using a PHP
script which involved
 Lookup of system type and retrieval of typeID
 Construction of system type page URL
http://srdata.nist.gov/solubility/sol_sys_lst.aspx?sysID=<typeID>&FROM=SSN
 Retrieval of the page content (HTML) into a PHP variable
 PCRE Regex expression match for the sysID of each system
 Creation of a new entry in the system database table
 4817 rows
Data Ingestion

11.  System details were scraped using a PHP script by
 Lookup of system and retrieval of sysID
 Construction of system detail page URL
http://srdata.nist.gov/solubility/sol_detail.aspx?sysID=<sysID>
 Retrieval of the page content (HTML) into a PHP variable
 Processing of HTML to retrieve
citation, variables, data analysis and tables, method, source,
errors, references
 Saving of details to systems table and related tables
Data Ingestion

12.  In addition to data extraction
 Chemical InChI strings were retrieved from NIH CIR1
 Citation DOI’s were retrieved from CrossRef2 and saved
(article titles and full author names were also added)
 Data tables were converted to JSON format for storage
and reproduction
 Table notes, sources, and additional refs were converted to
JSON for storage
Data Ingestion
(1) http://cactus.nci.nih.gov/chemical/structure
(2) http://www.crossref.org

13. Database

14. Database

15.  Constructed using the CakePHP framework (PHP)
 Index (listing) and view pages for each of
 Authors
 Chemicals
 Citations
 Systems
 System Types
 Volumes
 Search functionality provided via the homepage
 Example URL
http://chalk.coas.unf.edu/solubility/systems/view/20_135
Project Website Design

16. Project
Website
Design

17. Project
Website
Design

18.  Get this project funded
 Clean up references and link to DOI’s
 Clean up authors and link to ORCIDs
 Add procedural references
 Convert table data into searchable/linked format
 Add measurement type, unit, error, and variables
 Provide searching and plotting of data
 Automated calculation of additional parameters
e.g. solubility in different units, mole ratio
 Create solubility ontology => add RDF + searching
 Add microdata1 to each web page
 Next phase ? => Add the other volumes
Future Plans
(1) http://www.w3.org/TR/microdata/

19.  A RESTful version of the IUPAC-NIST Solubility Series
Database was successfully created and made available
 Metrics
 20 Volumes
 2564 System Types
 4817 Systems
 1484 Chemicals
 1247 References
 1968 Authors
 11 MB size of database
 One week worth of work
Conclusion

20.  schalk@unf.edu
 Phone: 904-620-5311
 Skype: stuartchalk
 LinkedIn/Slidehare: https://www.linkedin.com/in/stuchalk
 ORCID: http://orcid.org/0000-0002-0703-7776
 ResearcherID: http://www.researcherid.com/rid/D-8577-2013
Questions?