Edanz Journal Selector is a prototype journal search system built using Solr, Nutch, and Hadoop on Amazon Web Services. It indexes over 26,000 journal records from PubMed and journal tables of contents from over 1,600 publishers to provide a searchable index of academic journals. Data is obtained from various sources including PubMed, Springer, and journal TOCs and processed using Hadoop and indexed in Solr for search functionality powered by a Drupal front-end.

2. Edanz Journal Selector
a Prototype based on Solr/Nutch/Hadoop

3. Liang SHEN
@shenzhuxi
Web Developer
European Bioinformatics Institute
Drupal/Solr

4. Edanz Journal Selector (2011)

5. So many journals!

6. DEMO

7. Open Access
• By National Center for Biotechnology Information, U.S. National Library of Medicine
• Approximately 26,000 records are included in the PubMed journal lists

8. Feeds
Journal TOCs
• 21,498 journals from 1,677 publishers
• Institute for Computer Based Learning
• Heriot-Watt University

9. Springer
• Springer Metadata API
• Provides
metadata
for
over
5
million
online
documents
• Springer Open Access API
• Provides
metadata,
full-­‐text
content,
and
images
for
over
80,000
open
access
ar:cles

10. Open Source Stack
• Infrastructure: Amazon Web Service
• Data processing: Hadoop/Hive
• Index: Solr/Lucene
• Web service: Drupal
• Piwik

11. HDFS
Index
API
Feeds
Web

13. Springer Journal Selector

14. Chinese

15. Japanese

16. Scalability
• Shards

17. Internet vs. Intranet

18. Re-think after 3 years
Don't use Hadoop (<5TB)

19. Thanks!
Liang Shen