NiceCover: A Serverless Webapp for Crowdsourcing Data Extraction and Knowledge Generation on Top of Scientific Portals

.
Objectives
M.Zhanikeev -- maratishe@gmail.com -- NiceCover: A Serverless Webapp for ... on top of Scientific Portals 2/26
...
2/26

.
Objectives
1. facilitate true social collaboration of researchers
2. base your collaboration on existing scientific portals
◦ like IEICE iscover, IEEE Xplore, SpringerLink, ADL, etc.
3. create software for that
...
3/26

.
What's a Serverless Webapp?
...
4/26

.
Serverless Apps in the Cloud
.
Serverless Webapp...
..
.
... is a webapp that can run
without a web server
• entirely based on clouds
• hosted in cloud drives
• data management using
APIs
◦ Google Drive API, Dropbox
API
Service/Apps
Portal
The
Internet
Truly
Open
ClientsTraditional
Clients
Data
Data
Data
Apps
Sync
Data
BigData
User
FacilitatorsData
Hoarders
...
5/26

.
What's Social Collaboration
... in the world of webapps and cloud APIs?
...
6/26

.
Social Collaboration in the Cloud
• data can be yours only or it can be shared
• you can see all these models in
practice today
• the shared model is only
used by big players (big data)07
Data
App
Data
Data
App
Data
App
Data
App
App App
Data
Data
Data
Data
App
App
Data
App
07 R.Barros+4 "A Collaborative Approach to Building Evaluated Web Pages Datasets" Future Gen. Comp.Sys., vol.27(1) (2011)
...
7/26

.
Simple Example: maps2graphs
http://tinyurl.com/maps2graphs
• you can try it now! just let it run or commit before closing
.
Problem Statement
..
.
You need to create a graph where nodes are actual places on a map and
links are roads connecting them
• the above link: 300 Family Marts in Fukuoka
• 35k GoogleMaps API requests, assuming A→B = B→A routes
◦ free accounts are allowed 2.5k requests per day
• non-social solution: single client, 18 days
• social solution: 100 clients, a couple of hours
...
8/26

.
Step 1: Make Social Collaboration
Happen
...
9/26

.
Step 1 : Make Social Collaboration
Happen
1. is someone interested in graphs of actual locations?
◦ supply chains, road traffic optimization, etc.
2. is someone interested in research collaboration based on
content published in IEEE Xplore?
◦ post-publication discussion?
◦ professor-students collaboration in a lab?
...
10/26

.
Step 2: Distribute and Make it
Data-Centric
...
11/26

.
Step 2: Distribute and Make it
Data-Centric.
Problem
..
.
.... is that most existing software is all about big players and is completely
useless for real social collaboration
• problems that need solutions
1. client-side indexingwith cloud storage 12
• Lucene is a really bad choice! 13
2. seamlessoperation on top of scientific portals
• LinkedData APIs are not enough!
3. reasonable level of security
• your data should be safe!
12 myself "Stringex client" https://github.com/maratishe/stringex (current)
13 "Apache Lucene" lucene.apache.org (current)
...
12/26

.
Step 2: The Stringex Problem in
Client-Side Indexing
• JSON based
• index is created locally in
browser
• block-wise updates
• optimization problem --
minimize traffic exchange 11
◦ part of NiceCover, but have
PHP client as well 12
JSON { name: value1, age: value2, …}
Hash table
000 [ ]
001 …
…
#1 #2 …[ ]
Doc # JSON data
a123d …
53ffe3 { name: value1, age: value2, …}
…. ….
Per JSON key
…
hashing
Bit mask
Doc #Doc #
Cloud
storage
Local
storage
Realtime
Sync
name.block1
…
Block
Block
name.block2
age.block1
…
age.block2
docs.block1
…
docs.block2
Cloud Drive
API App Space
11 myself "The Stringex Problem: a New Formulation and Optimizations for Client-Side Cloud Applications" Springer (in review)
12 myself "Stringex client" https://github.com/maratishe/stringex (current)
...
13/26

.
Step 2: Stringex : Performance
• Lucene is too jittery
• Stringex client writes more
smaller files
◦ localized updates later on
• Stringex is more efficient
0 4000 8000 12000 16000 20000 24000
Transmitted traffic volume (kb)
0
10
20
30
40
50
60
Filecount
Stringex (keyHashMask=4; docHashMask=24)
Lucene
...
14/26

.
Step 3: Make it easy to Come By and
Use
...
15/26

.
Step 3: Chrome Extension
• 3 JS scripts as defined by
Chrome 03b
1. run for each page
2. run once, stays in
background
3. control panel
NiceCover
x
Browser
Extension
Control Panel
(user clickable)
Add-to-page
parsing script
Background
script
Parsing
report(popup)
Cloud Drive
writes onlyCloud Drive
reads/writes
Status
Pass data
for storage
03b "Google Chrome Extensions Developer Reference" http://developer.chrome.com/extensions/getstarted.html (current)
...
16/26

.
Step 3 : Full Automation
• will run with zero user
input
• user still has freedom to cancel
indexing of pages
• Stringex index and cloud sync
all happen in the background
◦ assuming you do not close your
browser...
NiceCover
(robot)
Users
Extract
data
URL Prefix Match
Wait
for input
Store
default
Hide popup
Timeout
Manual
input
User
click
...
17/26

.
Step 4: Make it Secure
...
18/26

.
OAuth-Based Communication
• NiceCover will only use what others shared with you
Cloud Drive API
Miner
Mapper
User’s own content
NiceCover webapp
NiceCover Public Data
NiceCover Private Datax
x
x
Miner
Mapper
...
19/26

.
Public and Private, ... or Both?
• shadow writes,
private + public
• simple protection
against
malicious
actions
◦ user confirms
deletes and
overwrites
◦ added data is
logged and
presented to user on
request
◦ rollback backups
NiceCover
Mapper
Cloud Drive
Provider
x
NiceCover
x
NiceCover
x
…
NiceCover
Mapper
x
NiceCover
Mapper
xPeer connection
Shadow
writes
...
20/26

.
A DEMO?
...
21/26

.
That’s all, thank you ...
...
22/26

.
[01] (current)
The enPiT Project
http://www.enpit.jp
[02] (current)
NiceCover Project Page
https://github.com/maratishe/nicecover
[03] (current)
Kontagent Homepage
http://www.kontagent.com/
[03b] (current)
Google Chrome Extensions Developer Reference
http://developer.chrome.com/extensions/getstarted.html
[04] (current)
Dropbox Homepage
www.dropbox.com/
...
22/26

.
[05] (current)
IEEE Xplore Terms of Use
http://ieeexplore.ieee.org/xpl/termsOfUse.jsp
[06] (current)
IEICE iSCover Search Engine
http://i-scover.ieice.org/
[07] R.Barros+4 (2011)
A Collaborative Approach to Building Evaluated Web Pages Datasets
Future Gen. Comp.Sys., vol.27(1)
[08] myself (in review)
On Metro Maps versus Ontology Graphs in Assisted Context Creation and
Browsing
IEICE Tran. Info.
[09] K.Nesbitt (2004)
Getting to more abstract places using the metro map metaphor
Conf. on Info. Visual. (IV)
...
22/26

.
[10] myself (current)
Metromaps Project
https://github.com/maratishe/metromaps
[11] myself (in review)
The Stringex Problem: a New Formulation and Optimizations for Client-Side Cloud
Applications
Springer
[12] myself (current)
Stringex client
https://github.com/maratishe/stringex
[13] (current)
Apache Lucene
lucene.apache.org
...
23/26

.
Q/A 1: Structure/Visualization
• currently, NiceCover only provides
raw metadata
• need to connect
• metromaps is a good way
to do that 08 09 10
• otherwise, ontologies
viz
pcapca
dependency
test
visualization
multidimensional
softwaresoftwaresoftwaresoftware
models
kbseken
smil
multimedia
session
performance
todotodotodotodotodo testing
modeling
benchmark
vne
optimization
cloud
ieiceconf
ospf
crosslayer
game
theory
opportunetstic
opportunistic
networking
multiflow
aggregationaggregation
hardware
p2pwifi
wifi
direct
multiple
connectivity
google
wirelesswireless
drive
content
rcskenrcsken
virtual
access
adhoc
aodv
e2e
path
establishment
battery
differential
backup
cost
budget
depletion
energy
efficiency
tomo
tomography
network
coordinates
endtoend
delay
networks
matrix
missingmissingmissingmissingmissingmissingmissing
values
08 myself "On Metro Maps versus Ontology Graphs in Assisted Context Creation and Browsing" IEICE Tran. Info. (in review)
09 K.Nesbitt "Getting to more abstract places using the metro map metaphor" Conf. on Info. Visual. (IV) (2004)
10 myself "Metromaps Project" https://github.com/maratishe/metromaps (current)
...
23/26

.
Q/A 2: Why Not Wikipedia Style?
1. Wikipedia is few create, many watch model
◦ collaborators are roughly equal in NiceCover, you contribute by browsing
◦ some may contribute more (professors versus students) but the effort gap is not
extreme
2. Wikipedia is a centralized place
◦ NIceCover is a bottom-up aggregation
◦ there is a hiararchy
◦ sideways (peer) connections are also important
3. Wikipedia pages are about one thing with references to other things
◦ NIceCover nodes are aggregates of things under a rough common intersection of topics
◦ at least they are supposed to be by design
◦ see metromaps 08 09 10
08 myself "On Metro Maps versus Ontology Graphs in Assisted Context Creation and Browsing" IEICE Tran. Info. (in review)
09 K.Nesbitt "Getting to more abstract places using the metro map metaphor" Conf. on Info. Visual. (IV) (2004)
10 myself "Metromaps Project" https://github.com/maratishe/metromaps (current)
...
24/26

.
Q/A 3: Is It Even Legal?
• not working with files, only metadata freely availabled on websites
◦ think about this as group browsing
• does not violate existing terms of use
◦ IEEE Xplore 05
◦ IECE iSCover seems to have no terms of use
• no problem with Dropbox -- we are just storing some data
05 "IEEE Xplore Terms of Use" http://ieeexplore.ieee.org/xpl/termsOfUse.jsp (current)
...
25/26

.
Q/A 4: Why Bother? IEEE Xplore
Does it Better!
• you seem to have missed the entire point
• IEEE Xplore: all about big information
◦ big data: big aggregates of information -- papers, presentations, etc.
◦ with growing volume, precision/focus/relevance is lost
• NiceCover: all about specific information
◦ focus is as big as your collaboration -- which is normally comparatively small
• big portals have poor APIs -- not suitable for collaboration, search only
...
26/26

NiceCover: A Serverless Webapp for Crowdsourcing Data Extraction and Knowledge Generation on Top of Scientific Portals

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (11)

Similar to NiceCover: A Serverless Webapp for Crowdsourcing Data Extraction and Knowledge Generation on Top of Scientific Portals

Similar to NiceCover: A Serverless Webapp for Crowdsourcing Data Extraction and Knowledge Generation on Top of Scientific Portals (20)

More from Tokyo University of Science

More from Tokyo University of Science (20)

Recently uploaded

Recently uploaded (20)

NiceCover: A Serverless Webapp for Crowdsourcing Data Extraction and Knowledge Generation on Top of Scientific Portals