Slides from a presentation I gave at SciBarCamb 2011 (9th April, 2011) in Cambridge (UK). Basically it goes through some of the recent work and theory i've been doing to do with setting up a data portal using distributed web services, allowing easy data sharing and reduced effort in data maintenance.

1. Building a Distributed
Data Portal
Darren Oakley
SciBarCamb 2011

2. Background
•mouse informatics @ Sanger Institute
•work with lots of other groups
•need to share, integrate and represent
lots of datatypes
•both OUR and OTHER peoples data

3. •gene information (id‘s, location, GO etc)
•related human diseases (OMIM, GWAS)
•expression
•phenotyping
•mutant mouse breeding
•mutant es cells, vectors

4. that‘s a lot of stuff...

5. we can do this one of
two ways...

7. ‘Borg‘ Approach
• single group becomes
sole owner/curator
of portal and its data
• other groups feed
their data into portal
group

10. burp

11. Pros
•clearly defined centre to the universe
•provides central curation to all data

12. Cons
•huge effort to curate and maintain large
and diverse dataset
•hold / maintain your own db of
everything
•integrating totally new / different data
becomes a challenge
•single group becomes effective ‘owner‘
•can stifle innovation and new ideas

13. what happens when
more than one group
tries to do this?

15. “Hand over your data,
prepare to be assimilated”
“No, YOU hand over your data and
prepare to be assimilated”
“Ahem, both of you, prepare to be assimilated!”

16. “Hand over your data,
prepare to be assimilated”
“No, YOU hand over your data and
prepare to be assimilated”
g?
l Bor
e rea
u is th
o
ch of y
… whi
“Ahem, both of you, prepare to be assimilated!”

17. ‘Federation‘ Approach
• each group hosts
their own data and
exposes it via defined
services
• make a ‘clever‘ portal
that pulls these
resources together
• no single group is
totally in charge

27. Use data for a more
specialized purpose
Build own portal
competitor

28. The Tech
search engine data sources
web service

29. MartSearch / Portal

30. MartSearch / Portal

31. MartSearch / Portal
index searchable
terms

32. MartSearch / Portal
index searchable
terms

33. MartSearch / Portal
index searchable
terms

34. MartSearch / Portal
send users search term to Solr
index searchable
terms

35. MartSearch / Portal
send users search term to Solr
Solr returns groups of terms
to query data sources with
index searchable
terms

36. MartSearch / Portal
send users search term to Solr
Solr returns groups of terms
to query data sources with
send asynchronous requests to each of the
data sources for the data the user is interested in
index searchable
terms

38. User searches for ‘diabetes‘

39. User searches for ‘diabetes‘
Search for ‘diabetes‘

40. User searches for ‘diabetes‘
Search for ‘diabetes‘
JSON data containing information on
what to search each datasource by...

41. User searches for ‘diabetes‘
Search for ‘diabetes‘
JSON data containing information on
what to search each datasource by...
Search using query parameters
defined by Solr response

42. User searches for ‘diabetes‘
Search for ‘diabetes‘
JSON data containing information on
what to search each datasource by...
Search using query parameters
defined by Solr response
Render search results using templates

43. Pros
•easily extendable
•data curation done by primary data
producers / handlers
•YOU don‘t have to keep / maintain copies
of everything

44. Cons
•hard to avoid some data redundancy
•need common linking terms
•un-curated as a whole

45. Extending the Portal
•set-up or find a new datasource to add
•other web service
•another biomart
•write a simple config/adaptor to talk to it

46. www.knockoutmouse.org/martsearch
github.com/i-dcc/martsearch
@dazoakley