An update from the Center for Open Science for the SHARE Fall Meeting 2014.

1. SHARE NotificationServiceDevelopment
Update from the Center for Open Science

2. Center for Open Science

3. Community
Metascience
Infrastructure

4. http://osf.io

5. Modular
Reliable
Flexible
Scalable

6. COS-SHARE Development Team
Brian Geiger Chris Seto Coral Sheldon-Hess
Erin Braswell Fabian Von Feilitzsch

7. COS-SHARE Developer Alumni
Casey Rollins Erica Baranski Faye Huynh Lauren Revere Ginny Huang
Michelle Yao Peter Fan Saman Ehsan Xander Herrick

8. Gather Notify
Providers Consumers

9. How are we getting data?
Application
Programming
Interface

10. GET https://frogworld.
com/api/frogs
Request Response
[
{
name: Kermit,
color: green,
type: felt
},
{
name: Travis,
color: green,
type: tree
}
]

11. POST https://osf.io/api/share
Request Response
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
Success

15. Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
origin: frogworld,
id: 10.100/frogworld.102,
description:
Exploring greenness.
}

16. Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
name: Easy Being Green?,
contributors: [
Frog Scientists Intl,
Amphibians United,
],
origin: frogworld,
doi: 10.100/frogworld.102
}

17. Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
title: Easy Being Green?,
source: frogworld,
id: 10.100/frogworld.102
}

18. Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
title: Easy Being Green?,
contributors: [
No Contributors
],
source: frogworld,
id: 10.100/frogworld.102
}

19. Gather Notify
Providers Consumers

20. Request
GET
http://repository.stcloudstate.edu/do/oai/?
verb=ListRecords
&metadataPrefix=oai_dc
&from=2014-10-02T00:00:00Z

21. Request Response
GET
http://repository.stcloudstate.edu/do/oai/?
verb=ListRecords
&metadataPrefix=oai_dc
&from=2014-10-02T00:00:00Z
<record>
<date> … </date>
<title> … </title>
<contributors>
<name>...</name>
<name>...</name>
</contributors>
<description>
…
</description>
<id>...</id>
</record>

22. Provider:
stcloudstate

23. Provider:
stcloudstate
11:05

24. Provider:
stcloudstate
11:05
GET
stcloud.edu/api/

25. Provider:
stcloudstate
11:05
GET
stcloud.edu/api/
GET
plos.org/api/
GET
columbia.edu/api/
GET
mit.edu/api/
GET
vtech.edu/api/
GET
dataONE.org/api/
Queue

26. Provider:
stcloudstate
11:05
GET
stcloud.edu/api/
GET
plos.org/api/
GET
columbia.edu/api/
GET
mit.edu/api/
GET
vtech.edu/api/
GET
dataONE.org/api/
Queue
worker
server image by RRZEicons

27. Provider:
stcloudstate
11:05
GET
stcloud.edu/api/
GET
plos.org/api/
GET
columbia.edu/api/
GET
mit.edu/api/
GET
vtech.edu/api/
GET
dataONE.org/api/
Queue
worker
worker
worker
worker
server image by RRZEicons

28. GET
stcloud.edu/api/
Archive Response
Archive Data
Archive Request

29. GET
stcloud.edu/api/
Archive Response
Archive Data
Normalize
Archive Request

30. <record>
<header>
<identifier>oai:repository.stcloudstate.edu:comm_facpubs-1001</identifier>
<datestamp>2014-10-01T17:54:11Z</datestamp>
</header>
<metadata>
<oai_dc:dc>
<dc:title>The Press as an Elite Power Group in Japan</dc:title>
<dc:creator>Akhavan-Majid, Roya</dc:creator>
<dc:description>
A lengthy field study in Japan using interviews and other sources and focusing on the
nation's five leading national newspapers and Tokyo's major television stations finds
evidence of much overlap between industry and the news media, through interlocking
directorships and social club memberships
</dc:description>
<dc:date>1990-12-01T08:00:00Z</dc:date>
<dc:identifier>http://repository.stcloudstate.edu/comm_facpubs/3</dc:identifier>
<dc:source>Mass Communication Faculty Publications</dc:source>
<dc:publisher>the Repository at St. Cloud State</dc:publisher>
<dc:subject>Mass Communication</dc:subject>
</oai_dc:dc>
</metadata>
</record>

31. {
"source": "stcloud",
"description": "A lengthy field study in Japan using interviews and other sources and focusing
on the nation's five leading national newspapers and Tokyo's major television stations finds
evidence of much overlap between industry and the news media, through interlocking
directorships and social club memberships,",
"contributors": {
"given": "Roya",
"family": "Akhavan-Majid",
"middle": "",
"email": ""
},
"title": "The Press as an Elite Power Group in Japan",
"dateCreated": "1970-01-01T00:00:00",
"dateUpdated": "2014-10-01T17:54:11+00:00",
"tags": [ "mass communication"],
"id": {
"url": "http://repository.stcloudstate.edu/comm_facpubs/3",
"serviceID": "oai:repository.stcloudstate.edu:comm_facpubs-1001",
"doi": ""
}
}

32. GET
stcloud.edu/api/
Archive Response
Archive Data
Normalize
Send to OSF
Archive Request

33. Gather Notify
Providers Consumers

34. Notify
Consumers

35. OSF Metadata
OSF Metadata and Search APIs

36. Notify
Consumers
RSS
PubSubHubbub
ResourceSync
OSF Search
Engine
OSF API

37. RSS

38. /rss/?q=open AND science
AND source:scitech

39. PubSubHubbub

40. ResourceSync

41. /resync/resourcelist/?q=open AND
science AND source:scitech

45. Inside the Notification Service
3:30 in this room

46. Recap of lessons learned

47. Today….
Over 20 content providers
Robust backend providing:
➢ archive of all research release events
➢ monitoring and extensive logging
➢ recovery from service interruptions
Solid foundation to enable building of other services
like curation and networking of research projects

48. More institutional repositories
Tests with complementary types of content:
Providers able to push content to SHARE NS
Coming soon...

49. Today Early
2015
Fall
2015
Beta
release
Full
release
20 + service
providers
Robust backend
Output formats
RSS
ResourceSync
Search
More service providers
Complementary data
Refined metadata
schema and push of
content

50. Try it: http://share-dev.osf.io
Follow on GitHub: https://github.com/
CenterForOpenScience/SHARE
See more details:
https://osf.io/29ypx/wiki/

51. [ Intentionally Left Blank ]

52. InsidetheNotification
Service

53. Elasticsearch
RabbitMQ
MongoDB/
TokuMX
Celery Workers OSF App Servers
ScrAPI
Load Balancers
Caching Proxies
Raw
UserProvider Provider Provider
Fluentd
Elasticsearch
Kibana
UserUser

54. ● Reliable
○ Messages persist
○ Redundant message storage
● Scalable
○ easily clustered
● Supported
○ Most used message broker
RabbitMQ

55. Celery
● Asynchronous job execution
● Scalable
○ Allows for any number of worker
machines
● Fault resistant
● Large community
○ Tight integration with RabbitMQ

56. MongoDB/TokuMX
● Fast
● Transaction-based
● Improve document
storage
● Document-level
locking

57. Elasticsearch
● Flexible
○ Schema-less
● Built to be horizontally
extensible
● Robust API

58. Fluentd
● Most robust log
aggregation system
● Converts log messages to
JSON for easy processing
● Emphasizes simplicity

59. Sentry
● No exceptions uncaught
● Real-time logging
● Easy triage

60. Kibana
● Nice visualizations
● Easy setup
● Simple to use
● Stats at a glance
● In depth progress
monitoring

62. Harvesters
Gathering metadata from providers

64. RSS Feed

68. SHAREintheworld

69. SHARE on GitHub
https://github.com/CenterForOpenScience/SHARE

70. Ways to access SHARE

71. Search API

74. Knockout - Observables

77. Knockout - Functions

80. HTML