A developer-perspective overview of Araport

1. araport.org
Extending the Arabidopsis
Information Portal: A Developer’s
Perspective
Matt Vaughn
Director, Life Sciences Computing
Texas Advanced Computing Center

2. araport.org
Web APIs: Problem Statement
• Lack of web services for legacy data
– There are a lot of web SITES
• Existing web services don’t share
information architecture
– Negatively impacts interoperability,
discoverability, & usability
• Browser security models are punitively
complex
– Hard to build apps integrating multiple
sources

3. araport.org
Gold standard Data APIs
• Implement REST-like interfaces
• Served over HTTPS (with valid SSL certificate)
• Allow Cross Origin Scripting Support (CORS)
• Require authentication
– Understand and respond to client demographics
– Meter access to services
• Simple controlled vocabulary + metadata for query
parameters
• Responses conform to accepted JSON schemas*
• Support future AIP deep caching & mining efforts**
* Except where it makes sense not to
** Based on tech like ElasticSearch or neo4j

4. araport.org
Araport Service Architecture
RESTful API @ https://api.araport.org/
CLI clients,
Scripts, 3rd party
applications
Physical
resources
Agave Core
apps
meta
files
profile
jobssystems
ADAMA
manage
enroll
a b c d e f
AIP + 3rd party data
providers
API Types
• Query
• Map*
• Generic
• Pass-through
• Single-sign on
• Metering
• Unified logging
• API versioning
• Automatic HTTPS +
CORS
REST*
CGI
SOAP
New
Web
Services
InterMin
e
Chado &
Tripal
Computing
Storage
Database

5. araport.org
Araport Service Architecture
RESTful API @ https://api.araport.org/
CLI clients,
Scripts, 3rd party
applications
Physical
resources
Agave Core
apps
meta
files
profile
jobssystems
ADAMA
manage
enroll
a b c d e f
AIP + 3rd party data
providers
API Types
• Query
• Map*
• generic
• pass-through
• Single-sign on
• Throttling
• Unified logging
• API versioning
• Automatic
HTTPS
REST*
CGI
SOAP
New
Web
Services
InterMin
e
Chado &
Tripal
Computing
Storage
Database

6. araport.org
Data API Types
Type Inputs Outputs Notes
query AIP parameters
mandatory
AIP-aligned JSON Gold standard
data APIs
map AIP parameters
preferred
Transformed JSON Ideal for
implementing
namespace
transformations or
filters
generic AIP parameters
preferred
Specified within
code but can be any
valid Content-type
Implement return
of non-JSON data
passthrough Specified by remote
service
Specified by remote
service
Allows existing
services to be
discoverable from
AIP data store

7. araport.org
Data API Reserved Parameters
Name Description Validator (Case-insensitive)
locus AGI Gene Locus
Identifiers
AT[1-5GM][0-5]{5,5}$
transcript AGI Transcript
Identifiers
AT[1-5GM][0-9]{5,5}.[0-9]{1,3}$
identifier Another string plausibly
expected to identify a
gene or transcript
Valid alphanumeric string. No
whitespace.
chromosom
e
A. thaliana Col-0
chromosome identifiers
CHR[1-5MC]$
start/end Coordinates within Col-0
assembly
Numeric. Should be range-checked.
strand Defines genomic strand [+-.]{1,1}
accession Ecotypes or natural
accessions
Not validated at present
term Generic search term Valid text string. Useful for

8. araport.org
Rationalized Responses via
lightweight JSON schemas
• Facilitate creation of mash-up client
applications
• Enable extraction and mining of the
Arabidopsis deep web
• Facilitate future interoperability with
semantic web technology without forcing
their adoption
Minimal, machine validated rules for what AIP
responses should look like

9. araport.org
curl –skL -XGET -H "Authorization: Bearer
624513772fbc2caf662b9accbf10380"
https://api.araport.org/community/v0.3/aip/resolver_fetch_locus_by_syn
onym_v0.2/search?identifier=URIC_ARATH
{"result":[
{"relationships":[
{"direction":"undirected",
"type":"synonymous_with",
"scores":[
{"confidence":1}]}],
"related_entity":"URIC_ARATH",
"class":"locus_id_mapping",
"locus":"AT2G26230",
"related_entity_kind":"UniProtKB-ID"}],
"metadata":
{"time_in_main":0.020552873611450195},
"status":"success"}
Example Araport JSON (1)

10. araport.org
Interacting with Araport APIs (1)
Araport web services publish live, interactive documentation

11. araport.org
Interacting with Araport APIs (2)
Araport web services are available in every Javascript console
Data API namespace
Individual Data API
> Agave.api.adama.getNamespaces()

12. araport.org
Interacting with Araport APIs (3)
Araport web services power Science Apps!

13. araport.org
Creating an Araport Data API (1)
• Decide on a type of Data API to build
• Initialize a local Git repository
• Author a main function (Python only for now)
• Test that it works in your local Python
interpreter
• Write a metadata.yml file describing the
service
• Push the local repository up to Github*
• Perform an authenticated HTTP POST to the
ADAMA service with a link to your repo
• Verify that the service was created
successfully
• Test it out via HTTP request* Or any public git server
Principles
A. All development is done on a
local system
B. Almost no software
dependencies beyond standard
system contents
C. Source code is always public
D. Testing via same routes as
usage
E. Easy to iterate if things go awry
2
3
4
5
1. Write code
2. Publish code
3. Register repository
4. Code deployed
5. Use web service
1

14. araport.org
Science Apps: Problem Statement
• Technical hurdles for developing web
applications
– Technology selection
– Development and testing environment setup
• The small number of applications that get
built are often not reusable

15. araport.org
Apps Infrastructure

16. araport.org
Apps Development
• Industry-standard, open-source tooling
– Node.js
– Yeoman
– Grunt
– Bower
• Application generator for quickly
bootstrapping application development
$ yo aip-science-app
$ grunt

17. araport.org

18. araport.org

19. araport.org
App Security
• Apps deployed to AIP are sandboxed
– Only the user creating the app can
access/use
– Publication workflow for AIP staff to code
review and functionality review before making
public
• App code is partitioned
– Kept separate from the rest of AIP Portal code
– Only executes in user’s browser, not on
server
• App artifact hosting is limited

20. araport.org
Apps Workspace
• Drupal module
• Apps upload/ingest from public git
repositories
• User-created “workspaces”
• Private, shared*, public apps

21. araport.org
Apps Workspace (2)

22. araport.org

23. araport.org

24. araport.org
Apps Examples
• Query app (ATTED-II)
• Visualization app (EBI Interaction Viewer)
• Computational app (BLAST)
• Other types (Notebook)

25. araport.org

26. araport.org

27. araport.org

28. araport.org

29. araport.org

30. araport.org
Developer Support
Online Tutorial Topic Link
Getting started http://bit.ly/aip-get-started
Technical overview http://bit.ly/aip-overview
Your first AIP app http://bit.ly/aip-first-app
Araport APIs and authentication http://bit.ly/aip-agave-auth
Creating a data-driven
application
http://bit.ly/aip-build-app
Deploying your app to Araport http://bit.ly/aip-deploy
Creating web services for Araport http://bit.ly/aip-websvcs
Linking to Araport content http://bit.ly/aip-link
• Bookmark araport.org/devzone
• Follow @araport on Twitter
• Join araport-developers Google Group
• Follow Arabidopsis-Information-Portal GitHub

31. araport.org
Chris Town, PI
Lisa McDonald
Education and
Outreach
Coordinator
Chris Nelson
Project
Manager
Jason Miller, Co-PI
JCVI Technical Lead
Erik Ferlanti
Software Engineer
Vivek Krishnakumar
Bioinf. Engineer
Svetlana Karamycheva
Bioinf Engineer
Eva Huala
Project lead, TAIR
Bob Muller
Technical lead, TAIR
Gos Micklem, co-PI Sergio Contrino
Software Engineer
Matt Vaughn
co-PI
Steve Mock
Portal Engineer
Rion Dooley,
API Engineer
Matt Hanlon,
Portal Engineer
Maria Kim
Bioinf Engineer
Ben Rosen
Bioinf
Analyst
Joe Stubbs,
API
Engineer
Walter Moreira,
API Engineer

32. araport.org

33. araport.org
Araport Service Architecture
RESTful API @ https://api.araport.org/
CLI clients,
Scripts, 3rd party
applications
Physical
resources
Agave Core
apps
meta
files
profile
jobssystems
ADAMA
manage
enroll
a b c d e f
AIP + 3rd party data
providers
API Types
• Query
• Map*
• generic
• pass-through
• Single-sign on
• Throttling
• Unified logging
• API versioning
• Automatic
HTTPS
REST*
CGI
SOAP
New
Web
Services
InterMin
e
Chado &
Tripal
Computing
Storage
Database

34. araport.org
ADAMA Road Map
• Automatic live documentation including
params
• Parameter validation at query time
• Response validation via JSON schema
• Automated provenance and attribution
• Language support (Java, Javascript, Perl)
• Full command line interface
• Status monitoring and notification
• Better “Data API Store”
• Per-namespace and-service Access Control
Lists

35. araport.org
Community Engagement
• Existing APIs + source turned over to the community
for additional development
• Community request for comment (RFC)
– Parameter metadata
– JSON Response schemas
– Provenance and attribution features
• Developing documentation, examples and tutorial
material
– Complete the entire API publication and usage lifecycle
without direct AIP intervention or personal support
• Assisting community in their development efforts

36. araport.org
Code Examples
• https://github.com/Arabidopsis-Information-Portal/jcvi-rtpcr-demos
• https://github.com/Arabidopsis-Information-
Portal/aip_thalemine_webservices
• https://github.com/Arabidopsis-Information-Portal/atted_webservices
• https://github.com/Arabidopsis-Information-Portal/bar_webservices_demos
In addition to our tutorial code, these are good, illustrative examples of ADAMA
web services.

37. araport.org
ADAMA: Araport DAta Mediator API
AGAVE
API MANAGER
NoSQL intermediary
Endpoint
https://api.araport.org/community/v0.3/
Live Docs
https://adama-dev.tacc.utexas.edu/api/adama.html

38. araport.org
API Manager + Enterprise Service Bus
Araport architecture (2)
Secure, rationalized REST services
Consumer Applications
Simple
Proxy
ThaleMine, Data
integration, other
services
Cache
XML-to-
JSON
SOAP-to-
REST
CGI-to-
REST
Throttle
Legacy
API A
Legacy
API B
REST
API C
Simple
Proxy
• Single-sign on
• Throttling
• Unified
logging
• API versioning
• Mediation and
translation
• Dev-friendly
interfaces
• Rationalized
REST for
consumer
apps
Mediators

39. araport.org
Science Objectives
• Make more, varied data available to the
Arabidopsis (and other) communities
within a unified user experience
• Enhance the innate value of data by
offering enhanced search, retrieval, and
display capabilities
• Facilitate analysis of user data
• Enable community participation in
functional annotation

40. araport.org
Technical Objectives
• Deploy a responsive, flexible community-
extensible system
• Provide APIs everywhere!
• Promote and facilitate data integration
• Enable language- and region-specific
presentation of scientific content
• Meet mobile computing on its own terms

41. araport.org
Local vs. Data-driven Apps
Resources are local and
inherently offline. Operating
on local data using local
computing.
Resources are cloud-based and
inherently online. Multiple data
streams integrated, queried,
presented in context of broader
objective.
Photoshop Express KAYAK Pro

42. araport.org
Araport Bill of Materials
• Araport is currently built using
– Drupal 7.25
• Developer-oriented content management system
– Bootstrap.js and some other Javascript toolkits
– InterMine (with modifications)
– Bioinformatics infrastructure + misc. other bits
– Agave 2.0 Software as a Service platform
• Developed by iPlant Collaborative project
• Bulk data, metadata, authentication, HPC app and job
management, notifications & events, and more
• OAuth2 out of the box
• Enterprise service bus (ESB) architecture
• http://agaveapi.co/

43. araport.org
Agave wso2 interface
Cache (Technology TBD)
CSV
Araport APIM Architecture (1)
POLYMORPH CGI
Form
Input Key
Map
Output
Key Map
Input
Transform
Output
Transform
Listen Respond
Send Listen
Input Key
Map
Output
Key Map
Input
Transform
Output
Transform
Listen Respond
Send Listen
Araport API
Manager
JSON Query JSON Response
ElasticSearch
Remote Services
SNP by Locus REST Indel by Position REST Enroll Manage

44. araport.org
Araport Architecture: Use Cases (1)
• 1001 Genomes POLYMORPH tools
– Provides variation data via locus or positional
search
– Total of seven variant types available for search
– Search parameterization depends a lot on variant
type
– Example of a plain-text CGI service
– Returns results as CSV with named columns
• Objective: Transform into a RESTful API that
expects and returns rationalized JSON
http://polymorph.weigelworld.org

45. araport.org
Araport Architecture: Use Cases (2)
• ThaleMine
– Has native REST interface for general queries
– Has templates which can form basis of specific
services
• Objective: Offer both Intermine-native and
AIP-conformant interfaces as Data APIs
• Current path
– Enroll native services in our APIM
– Develop template-based AIP-conformant services
http://polymorph.weigelworld.org

46. araport.org
Data APIs: Getting Started
Service Queries Notes
BAR eFP Locus
BAR Expressologs Locus
BAR Interactions Locus
COGe Position Special case – output transform only
NASC $SERVICE Locus
SOAP based but may be offline
permanently
OrthologFinder Locus Based on a Thalemine template
POLYMORPH Locus, Position Actually seven CGI services
SUBA3 Locus
Compiling example queries, parameter mapping and description, and ideal
results for use in implementing the system

47. araport.org
Developing a Data API
• In order, we prefer that you have ready
• Well-documented REST
• Moderately well-documented REST
• SOAP services (plus WSDL or WADL)
• Plain Old XML
• Plaintext CGI
• HTML CGI
• No web services at all
• Work with us to enroll your services as a data
source. This will involve a minor amount of
coding.

48. araport.org
Computational App Model (1)
Host file
systems
Host OS
Docker.io
Centos
6.4
custom-
repo
Container
/scratch
/database
Host FS (250 GB)
TACC Corral (PB+)
sftp
Agave apps, data, jobs
REST API x JSON objects

49. araport.org
Science Apps: Grid View
• Current Scheme
• 2-3 column view w
draggable apps
• Apps are normal, full-
size, or collapsed
• Single app screen
• Later in 2014
• N x X grid scheme
implementing resizable
app “tiles” like one sees
in Android or Win8.x
• App SDK libraries will
have “help” for enabling
resizable design
• Multiple app screens

50. araport.org
Data API Details (2)
• For service-specific parameters
– Provide human-readable names mapped to original
parameter names
– Offer minimal descriptive text
– Specify validation
• Cardinality
• Pattern validator (regex)
• Type (number, string, etc.)
– Indicate whether required
– Indicate whether they should be visible in a UI
– Specify reasonable default values
• Seems familiar?
– This approach is used to to abstract command line apps
– Allows automatic generation of minimally functional UI

51. araport.org
Data APIs: Response types (1)
• locus_relationship – pairwise
relationship between A and B
– Directionality
– Type
– Array of scores (weights, etc.)
• sequence_feature – positional attribute
– Extension of GFF model plus
– Build
– Attributes array

52. araport.org
Data APIs: Response types (2)
• locus_feature – key-value attributes per locus
– Optional controlled vocabulary* for keys
– Support for both slots and arrays
• raw – for returning images or other binary formats
– Source and other metadata carried in X-headers instead of
JSON result
– Outbound transformation still supported
– Not a preferred response mode
• text – returning either native service response or a
non-conformant JSON document
– Source and other metadata carried in X-headers instead of
JSON result
– Not a preferred response mode

53. araport.org
Data API Details (6)
• Transparent caching will compensate for
transient remote service failures
• Automatic indexing of certain response
types via ElasticSearch, allowing for
sophisticated global search
– ElasticSearch allows us to index everything
we “know about” and return it quickly
– iPlant uses it to live-index >700 TB user data

54. araport.org
Developing an app
• Understand and document the user stories you’re
addressing with your app
• Identify all requisite data sources AND
• Help us prepare them as Data APIs
– This may involve coding
• Understand the data integration or aggregation needs
of your app
– This may involve coding
• Develop the user interface(s) for your app using our
tool kits and suggested practices
– This will involve coding.
– But you will learn tools like jQuery, Bootstrap, & D3 and will
thus be eminently employable!