Open PHACTS Architecture and Docker install Big Data Europe (BDE) webinar 2016-07-06 https://slides.com/soilandreyes/2016-07-06-openphacts

1. Open PHACTS
Architecture and
Docker install
Stian Soiland-Reyes, University of Manchester
http://orcid.org/0000-0001-9842-9718
@soilandreyes
This work is licensed under a
.Creative Commons Attribution 4.0 International License
Big Data Europe Webinar, 2016-07-06
This work has been done as part of the BioExcel CoE ( ),
a project funded by the EC H2020 program, contract number
www.bioexcel.eu
EINFRA-5-2015 675728
https://slides.com/soilandreyes/2016-07-06-openphacts
1

2. http://www.openphacts.org/
Bringing together pharmacological data resources
in an integrated, interoperable infrastructure
Data sources integrated and linked together
so that you can easily see the relationships
between compounds, targets, pathways,
diseases and tissues.
, , , ,
, , , ,
, ,
ChEBI ChEMBL ChemSpider ConceptWiki
DisGeNET DrugBank FAERS Gene Ontology
neXtProt SureChEMBL, UniProt WikiPathways
2 . 1

3. Data integration
https://www.openphacts.org/2/sci/data.html
2 . 2

4. https://dev.openphacts.org/docs/2.1
Re-exposed as
public API
2 . 3

5. {
"format": "linked-data-api",
"version": "1.5",
"result": {
"_about": "https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.conceptwiki.org%2Fconce
"definition": "https://beta.openphacts.org/api-config",
"extendedMetadataVersion": "https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.concep
"linkPredicate": "http://www.w3.org/2004/02/skos/core#exactMatch",
"activeLens": "Default",
"primaryTopic": {
"_about": "http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7bdf9d5",
"inDataset": "http://www.conceptwiki.org",
"exactMatch": [
{
"_about": "http://bio2rdf.org/drugbank:DB00398",
"description_en": "Sorafenib (rINN), marketed as Nexavar by Bayer, is a drug approved for th
"description": "Sorafenib (rINN), marketed as Nexavar by Bayer, is a drug approved for the t
"drugType_en": [
"investigational",
"approved"
],
"drugType": [
"investigational",
"approved"
],
"genericName_en": "Sorafenib",
"genericName": "Sorafenib",
"metabolism_en": "Sorafenib is metabolized primarily in the liver, undergoing oxidative meta
"metabolism": "Sorafenib is metabolized primarily in the liver, undergoing oxidative metabol
"proteinBinding_en": "99.5% bound to plasma proteins.",
"proteinBinding": "99.5% bound to plasma proteins.",
"toxicity_en": "The highest dose of sorafenib studied clinically is 800 mg twice daily. The
"toxicity": "The highest dose of sorafenib studied clinically is 800 mg twice daily. The adv
"inDataset": "http://www.openphacts.org/bio2rdf/drugbank", 2 . 4

6. <?xml version="1.0" encoding="utf-8"?>
<result format="linked-data-api" version="1.5" href="https://beta.openphacts.org/1.5/compound?uri=
<primaryTopic href="http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7bdf9d5">
<prefLabel xml:lang="en">Sorafenib</prefLabel>
<exactMatch>
<item href="http://rdf.ebi.ac.uk/resource/chembl/molecule/CHEMBL1336">
<type href="http://rdf.ebi.ac.uk/terms/chembl#SmallMolecule"/>
<inDataset href="http://www.ebi.ac.uk/chembl"/>
<mw_freebase datatype="double">464.82</mw_freebase>
</item>
<item href="http://ops.rsc.org/OPS379634">
<smiles>CNC(=O)C1=NC=CC(=C1)OC2=CC=C(C=C2)NC(=O)NC3=CC(=C(C=C3)Cl)C(F)(F)F</smiles>
<rtb datatype="double">5.0</rtb>
<ro5_violations datatype="double">1.0</ro5_violations>
<psa datatype="double">92.35</psa>
<molweight datatype="double">464.825</molweight>
<molformula>C21H16ClF3N4O3</molformula>
<logp datatype="double">5.158</logp>
<inchikey>MLDQJTXFUGDVEO-UHFFFAOYSA-N</inchikey>
<inchi>InChI=1S/C21H16ClF3N4O3/c1-26-19(30)18-11-15(8-9-27-18)32-14-5-2-12(3-6-14)28-20(31
<hbd datatype="double">3.0</hbd>
<hba datatype="double">7.0</hba>
<inDataset href="http://ops.rsc.org"/>
</item>
<item href="http://aers.data2semantics.org/resource/drug/NEXAVAR">
<prefLabel>NEXAVAR</prefLabel>
<reportedAdverseEvent>
<item href="http://aers.data2semantics.org/resource/diagnosis/HEAD_INJURY">
<prefLabel>HEAD INJURY</prefLabel>
<inDataset href="http://aers.data2semantics.org/"/>
</item>
<item href="http://aers.data2semantics.org/resource/diagnosis/SUPRAVENTRICULAR_TACHYCARD
<prefLabel>SUPRAVENTRICULAR TACHYCARDIA</prefLabel>
<inDataset href="http://aers.data2semantics.org/"/> 2 . 5

7. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix void: <http://rdfs.org/ns/void#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix ns0: <http://www.openphacts.org/api#> .
@prefix ns1: <http://bio2rdf.org/> .
@prefix ns2: <http://rdf.ebi.ac.uk/terms/chembl#> .
@prefix chembl1336: <http://rdf.ebi.ac.uk/resource/chembl/molecule/CHEMBL1336#> .
@prefix linked-data: <http://purl.org/linked-data/api/vocab#> .
@prefix msg0: <http://www.openphacts.org/api/> .
<http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7bdf9d5>
skos:exactMatch <http://aers.data2semantics.org/resource/drug/NEXAVAR> ;
skos:exactMatch <http://aers.data2semantics.org/resource/drug/SORAFENIB> ;
skos:exactMatch <http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7b
skos:exactMatch <http://bio2rdf.org/drugbank:DB00398> ;
skos:exactMatch <http://rdf.ebi.ac.uk/resource/chembl/molecule/CHEMBL1336> ;
skos:exactMatch <http://ops.rsc.org/OPS379634> ;
skos:prefLabel "Sorafenib"@en ;
void:inDataset <http://www.conceptwiki.org> ;
foaf:isPrimaryTopicOf <https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.conc
<https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.conceptwiki.org%2Fconcept%2F3893255
foaf:primaryTopic <http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7
linked-data:definition <https://beta.openphacts.org/api-config> ;
msg0:activeLens "Default" ;
void:linkPredicate skos:exactMatch ;
linked-data:extendedMetadataVersion <https://beta.openphacts.org/1.5/compound?uri=http%3A
<http://ops.rsc.org/OPS379634>
void:inDataset <http://ops.rsc.org> ;
ns0:smiles "CNC(=O)C1=NC=CC(=C1)OC2=CC=C(C=C2)NC(=O)NC3=CC(=C(C=C3)Cl)C(F)(F)F" ;
ns0:inchi "InChI=1S/C21H16ClF3N4O3/c1-26-19(30)18-11-15(8-9-27-18)32-14-5-2-12(3
ns0:inchikey "MLDQJTXFUGDVEO-UHFFFAOYSA-N" ; 2 . 6

8. explorer.openphacts.org
3

9. Architecture
4 . 1

10. API architecture
4 . 2

11. Chemical Structure Search
RDF/SPARQL
(Virtuoso)
Identity Mapping Service
Identity Resolution Service
(ConceptWiki)
Chembl, Uniprot, ...
Data loading
4 . 3

12. Linux Container technology
..light-weight "virtual" virtual machine
A container is started from a image
Images downloaded from Docker Hub
Dockerfile: Layer-based recipe
Philosophy: One service, one image → microservices
Cloud's best friend: scalable, reproducible, customizable
https://www.docker.com/
5 . 1

13. https://hub.docker.com/r/openphacts/
5 . 2

14. ops-ims
ops-mysql
ops-virtuoso
ops-apiops-memcached
ops-virtuosodata
ops-mysqldata
ops-virtuosostaging
ops-mysqlstaging
https://data.openphacts.org/
ops-explorer
:3001
:3002
:3004:3003
https://hub.docker.com/
ops-docker
https://github.com/openphacts/ops-docker/
5 . 3

15. Docker Compose
https://www.docker.com/products/docker-compose
Which images to download
Which data volumes to use
Which network ports are exposed
How are containers linked
How to start/stop the containers
$ docker-compose up -d
5 . 4

16. docker-compose.yml
# Open PHACTS platform
# Docker Compose configuration
explorer:
image: openphacts/explorer2
ports:
- "3001:3000"
links:
- api
environment:
- API_URL=http://localhost:3002
#restart: always
api:
image: openphacts/ops-linkeddataapi
ports:
- "3002:80"
links:
- ims
- memcached
- virtuoso:sparql
# SPARQL server
virtuoso:
build: virtuoso-ops
ports:
- "3003:8890"
volumes_from:
- virtuosodata
virtuosodata:
image: busybox
volumes:
- /virtuoso 5 . 5

17. Data staging
6 . 1

18. Docker and data?
Docker Hub maximum image size: 10 GB
Open PHACTS data (compressed): ~30 GB
Open PHACTS data (installed): ~200 GB
Solution: Added staging Docker containers
Download from
Verify consistency
Import into Virtuso and mySQL
https://data.openphacts.org/
6 . 2

19. https://data.openphacts.org/
6 . 3

20. https://data.openphacts.org/
data.openphacts.org
RDF datasets
RDF linksets
VoID metadata/provenance
mySQL-imported linksets
Virtuoso-imported datasets
→ Maven repository
release data as software
→Research Objects
propagate metadata
6 . 4

21. Try it!
7 . 1

22. https://github.com/openphacts/ops-docker
Hardware requirements:
150 GB of disk space (ideal: 250 GB)
16 GB of RAM (ideal: 128 GB)
4 CPU core (ideal: 8 cores)
Prerequisites:
Recent x64 Linux (Ubuntu 14.04 LTS, Centos 7)
Fast Internet connection
Docker
Docker Compose
What do I need?
7 . 2

23. https://github.com/openphacts/ops-docker
Follow the GitHub tutorial exactly, customize later
Install latest Docker and Docker Compose
Just testing on Windows or OS X?
.. modify Docker's Linux VM to have enough disk and memory
Firewall? Different settings depend on your firewall details.
Don't worry - Docker is containerized!
..you won't break your machine
Don't jump ahead..
7 . 3

24. https://github.com/openphacts/ops-docker
Get the software
curl -L https://github.com/openphacts/ops-docker/archive/master.tar.gz | tar xzv
cd ops-docker-master
sudo docker-compose pull
7 . 4

25. https://github.com/openphacts/ops-docker
Get the data
$ sudo docker-compose up --no-recreate -d mysqlstaging virtuosostaging
$ sudo docker-compose logs mysqlstaging virtuosostaging
ops-mysqlstaging | mySQL staging finished
ops-mysqlstaging exited with code 0
ops-virtuosostaging | 09:13:35 --> Backup file # 675 [0x3F02-0x74-0x8A]
ops-virtuosostaging | 09:13:36 --> Backup file # 676 [0x3F02-0x74-0x8A]
ops-virtuosostaging | 09:13:37 End of restoring from backup, 6751701 pages
ops-virtuosostaging | 09:13:37 Server exiting
ops-virtuosostaging | Loading completed
ops-virtuosostaging exited with code 0
7 . 5

26. https://github.com/openphacts/ops-docker
Start the services
$ sudo docker-compose up --no-recreate -d
$ sudo docker-compose logs --tail=5
7 . 6

27. Using the services
8 . 1

28. http://localhost:3001/ Explorer
8 . 2

29. http://localhost:3002/ API
8 . 3

30. http://localhost:3003/ SPARQL
8 . 4

31. http://localhost:3004/QueryExpander Identity Mapping
8 . 5

32. What's next?
9 . 1

33. Custom data staging
Different Open PHACTS 2.1 licensing options:
Non-Commercial users: Everything
Commercial users: No DrugBank, partial SureChembl
Open PHACTS members: Full SureChembl
9 . 2

34. Microservices pr dataset
Most queries have separate fragments per dataset
..which could be executed on separate microservices
Better cloud scalability
Easier to test upgrades of individual datasets
But still need "API" layer to do Identity Mapping
and selecting datasets to query
9 . 3

35. BioExcel Workflow blocks
BioExcel approach: Spin up virtual machine when an
Open PHACTS workflow is started
Workflow bound dynamically to VM instance(s)
Scalability (exclusive access)
Reproducibility (independent/fixed OPS install)
Tool descriptions - exposed in bio.tools
9 . 4

36. Customization
Make it easier to add third-party data:
datasets, linksets, queries, API calls
..so pharma industry can mix in their in-house data
.. so academics can upgrade and expand datasets
More tooling,
more documentation,
or more training?
9 . 5

37. Feedback
https://github.com/openphacts/ops-docker/issues
http://support.openphacts.org/
http://ask.bioexcel.eu/
https://data.openphacts.org/
10