LiquidPub: Services at the service of
science
Marcos Baez, Fabio Casati , and the LiquidPub
                    team
  Dep...
In a nutshell: the goal
Capture the lessons learned and opportunities provided by the Web
and open source, agile developme...
Objective
                  Understand                  Improve
         Dissemination models and      Better ways to do...
Understanding




project.liquidpub.org   4
PR: Initial Goals

 Understand how well peer
  review works

 Understand how to
  improve the process

 Metrics + Analy...
Comparing rankings
 28       33
 17       89
  2        2
 45       17
 67       67
 ..       ..
 ..       28
 89       .....
Comparing rankings

               T=3




               N=10




           Divergence: Div (t,N)
           Kendall τ
8
Results:
      peer review ranking vs. citation
                  count
                    Divergence
Div




           ...
Quality-related Metrics: Statistics
                       Distribution of marks (integer marks)
              0.18
      ...
Disagreement vs number of
         reviews

  2.5
   2
  1.5
   1
        R1   R2    R3+
Fairness

 Definition: A review process is fair if and only of
  the acceptance of a contribution does not
  depend on th...
The road ahead
About citations…

project.liquidpub.org
Mikalai Krapivin, Maurizio Marchese, Fabio Casati: Exploring and
Understanding Scientific Metrics in Citation Networks. Co...
Positional effect on citation and readership
17




     in arXiv, by Haque and Ginsparg
project.liquidpub.org
Fabio Casati - ECOWS 2009




                            Positional effect on citation and readersh...
Improving

                  Principles
                   Models
                 IT Services


project.liquidpub.org    ...
Principles and objectives

 Everything counts!! (not just papers, not just ―innovations‖)

 Minimal dissemination overhe...
Model:
Liquid Journals
Liquid Journals




project.liquidpub.org   Fabio Casati - ECOWS 2009   22
project.liquidpub.org   Fabio Casati - ECOWS 2009   23
Internet

                                                      Data


project.liquidpub.org     Fabio Casati - ECOWS 2009...
Data




project.liquidpub.org   Fabio Casati - ECOWS 2009          25
project.liquidpub.org   Fabio Casati - ECOWS 2009   26
3. What you can do with it




project.liquidpub.org   Fabio Casati - ECOWS 2009   27
2. How it gets filled…

 Can be edited, but can be (semi-)automated

 Topics and tags (+ tags by specific people)

 Sub...
Results
         Everybody becomes a journal editor (selfishly)

         No gatekeeping, quality enforced by the commun...
IT Services: LiquidPub
    infrastructure



project.liquidpub.org   Fabio Casati - ECOWS 2009   30
Engineer a system that

         Accesses heterogeneous src w/ no API

         Is modular so that functions can be reus...
Liquid Journals dashboard


                   Liquid journals API



     Search                   Personalized
         ...
UI
                                                 A
                        Research services        P
                 ...
Karaku: Uniform access
Repositories                     Documents                       (CRUD)

 Resource       1..N     1...
Vertical services


 Liquid journals                Liquid books                    …




                                ...
[Insert header here]   [Insert footer]
Gelee
  Paper at ICDE 09. Video avail online and demo at icsoc



[Insert header here]                [Insert footer]
Mashart
  Paper at ER‘09, demo at icsoc ‗09



project.liquidpub.org                 38
RESEVAL     http://demo.liquidpub.org/reseval/ complex’09
project.liquidpub.org                                           ...
Group Management
Group Management



               Compare the metrics
                of people in
                group.
              ...
comparison




   ―Search comparison‖ shows all the comparison made.



    We can see and recompute them (for updated da...
Communities




project.liquidpub.org   43
Overview of the Community Network

                                                              SOFTWARE ENGINEERING(kbse...
Community Comparison and
                              normalization
          50


          45


          40


        ...
Hi Alex!




Create Journal



                 U   You haven’t created any LJ yet.
                     Start sharing sci...
Hi Alex!




                   Untitled
1   General Info
                    Journal Name      Peer-review journal
    So...
Hi Alex!




                   Peer-review journal                                                                       ...
Hi Alex!




                   Peer-review journal
1   General Info


    Sources to         Traditional journal process
...
Hi Alex!




                           Peer-review journal
     Create Issue                                             ...
Hi Alex!




                     Peer-review journal
     Create Issue                                                   ...
Hi Alex!




                             Peer-review journal » Issue on new review models
    Create Issue             Pu...
Take-Home Message

         Flaws of current practices (or, lack of evidence
              that they work as expected)

 ...
 Google liquidpub-announce and subscribe if
              you are interested (VERY low traffic)

         First release ...
How to collaborate

 Liquid journals server
   http://demo.liquidpub.org/liquidjournals/

 Liquid journals dashboard
  ...
How to collaborate

         http://project.liquidpub.org/karaku/

         http://project.liquidpub.org/resman/




pro...
57
LiquidPub: Services at Service of Science
Upcoming SlideShare
Loading in …5
×

LiquidPub: Services at Service of Science

671 views
605 views

Published on

"LiquidPub: Services at Service of Science". Invited talks of Fabio Casati at the European Conference on Web Services 2009 and in the Politechnico di Milano

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
671
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

LiquidPub: Services at Service of Science

  1. 1. LiquidPub: Services at the service of science Marcos Baez, Fabio Casati , and the LiquidPub team Department of Information Engineering and Computer Science University of Trento ECOWS 2009 – Eindhoven, The Netherlands – 10 Nov 2009
  2. 2. In a nutshell: the goal Capture the lessons learned and opportunities provided by the Web and open source, agile development to develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming scientific knowledge. Understand what’s good for science, and make it happen project.liquidpub.org •Publish and perish: why the current publication and review model is killing research and wasting your money (ACM Ubiquity 8(3), Feb 2007), and •Liquid Publications: Scientific Publications meet the Web With Fausto Giunchiglia and Maurizio Marchese.
  3. 3. Objective Understand Improve  Dissemination models and  Better ways to do the overhead same things  Evaluation processes, and  Different things quality/impact of research (people, papers, projects)  Principles  Peer review and  Models innovation  IT services  Scientific communities  Funding models and processes project.liquidpub.org 3
  4. 4. Understanding project.liquidpub.org 4
  5. 5. PR: Initial Goals  Understand how well peer review works  Understand how to improve the process  Metrics + Analysis  (refer to liquid doc)  Focus only on “Not everything that can be counted counts, gatekeeping aspect and not everything that counts can be counted.” -- Albert Einstein
  6. 6. Comparing rankings 28 33 17 89 2 2 45 17 67 67 .. .. .. 28 89 .. 33 .. .. 45
  7. 7. Comparing rankings T=3 N=10 Divergence: Div (t,N) Kendall τ
  8. 8. 8
  9. 9. Results: peer review ranking vs. citation count Divergence Div Normalized t 9
  10. 10. Quality-related Metrics: Statistics Distribution of marks (integer marks) 0.18 0.16 0.14 0.12 Probability 0.1 0.08 0.06 0.04 0.02 0 0 1 2 3 4 5 6 7 8 9 10 Marks Also: (-3,3) very different from (1,7) 11
  11. 11. Disagreement vs number of reviews 2.5 2 1.5 1 R1 R2 R3+
  12. 12. Fairness  Definition: A review process is fair if and only of the acceptance of a contribution does not depend on the particular set of PC members that reviews it  The key is in the assignment of a paper to reviewers: A paper assignment is unfair if the specific assignment influences (makes more predictable) the fate of the paper.
  13. 13. The road ahead
  14. 14. About citations… project.liquidpub.org
  15. 15. Mikalai Krapivin, Maurizio Marchese, Fabio Casati: Exploring and Understanding Scientific Metrics in Citation Networks. Complex 2009 16
  16. 16. Positional effect on citation and readership 17 in arXiv, by Haque and Ginsparg
  17. 17. project.liquidpub.org Fabio Casati - ECOWS 2009 Positional effect on citation and readership 18 in arXiv, by Haque and Ginsparg
  18. 18. Improving Principles Models IT Services project.liquidpub.org Fabio Casati - ECOWS 2009 19
  19. 19. Principles and objectives  Everything counts!! (not just papers, not just ―innovations‖)  Minimal dissemination overhead  Early sharing  Early feedback/interaction  Find diversity  Interestingness and sharing as measures of reputation  No gatekeeping. Use the filtering power of the community  We are not necessarily right!! Let the community select the principles and models
  20. 20. Model: Liquid Journals
  21. 21. Liquid Journals project.liquidpub.org Fabio Casati - ECOWS 2009 22
  22. 22. project.liquidpub.org Fabio Casati - ECOWS 2009 23
  23. 23. Internet Data project.liquidpub.org Fabio Casati - ECOWS 2009 24
  24. 24. Data project.liquidpub.org Fabio Casati - ECOWS 2009 25
  25. 25. project.liquidpub.org Fabio Casati - ECOWS 2009 26
  26. 26. 3. What you can do with it project.liquidpub.org Fabio Casati - ECOWS 2009 27
  27. 27. 2. How it gets filled…  Can be edited, but can be (semi-)automated  Topics and tags (+ tags by specific people)  Subscriptions/preferences  Journals—editing power of the community!!  Authors  Wisdom of people you trust (as editor)  Wisdom of the crowd (if any)  Diversity!!!
  28. 28. Results  Everybody becomes a journal editor (selfishly)  No gatekeeping, quality enforced by the community (gatekeeping is the noise)  You get things you want to read (or you want your group to read)  You get diversity  You leverage the selection work of your friends  Measure interestingness and quality by sharing  Reduce (optimize) dissemination overhead and encourage early sharing project.liquidpub.org Fabio Casati - ECOWS 2009 29
  29. 29. IT Services: LiquidPub infrastructure project.liquidpub.org Fabio Casati - ECOWS 2009 30
  30. 30. Engineer a system that  Accesses heterogeneous src w/ no API  Is modular so that functions can be reused  In a world where modularity had little success  Facilitates the creation of arbitrary dissemination and evaluation models (by non-programmers)  Provides commonly needed research services  Implements liquid journals (agile – our main macro- story)  Simple and lightweight project.liquidpub.org Fabio Casati - ECOWS 2009 31
  31. 31. Liquid Journals dashboard Liquid journals API Search Personalized Sharing computing tagging Diversity Caching Reputation Cached metadata Search by topic Crawling Access Disambigua tion
  32. 32. UI A Research services P I Crawling Karaku client-code Cache mgmt Subscriptions/stream Disambiguation Personalized tagging REST API Liquidity Karaku Rendering REST API Cached SKOs ResMan Adapters project.liquidpub.org Fabio Casati - ECOWS 2009 REST API REST API 33
  33. 33. Karaku: Uniform access Repositories Documents (CRUD) Resource 1..N 1..N 1..N 1..N Resource type Action type Manager type Abstraction Resource 1..1 1..N 1..1 1..N Resource Actions Manager Google Docs Documents in Google Docs GDocs-specific Operations  Handling semantically-equivalent resources with the same set of actions and interface  Interoperability at the type level  The semantics are defined by the type
  34. 34. Vertical services Liquid journals Liquid books … programming Collaborative Gelee Search computing Mashart Community TopicSearch Reseval discovery project.liquidpub.org Fabio Casati - ECOWS 2009 35
  35. 35. [Insert header here] [Insert footer]
  36. 36. Gelee Paper at ICDE 09. Video avail online and demo at icsoc [Insert header here] [Insert footer]
  37. 37. Mashart Paper at ER‘09, demo at icsoc ‗09 project.liquidpub.org 38
  38. 38. RESEVAL http://demo.liquidpub.org/reseval/ complex’09 project.liquidpub.org 39
  39. 39. Group Management
  40. 40. Group Management  Compare the metrics of people in group.  Total citations  Total publications  H-index  G-index  Noise ratio
  41. 41. comparison  ―Search comparison‖ shows all the comparison made. We can see and recompute them (for updated data).
  42. 42. Communities project.liquidpub.org 43
  43. 43. Overview of the Community Network SOFTWARE ENGINEERING(kbse,icse) DIST. SYSTEM/COMPILER(ipps,iccS) APPLIED COMPUTING/CRYPTO(sac,compsac) TELETEACHING/HUM_INT (chi,hicss) TELECOM (icc,globecom) HUMMAN –COMP INTER(icchp,hci) GENETIC AND EVO ALG(cec,gecco) AI/DB (icai,aaai) ROBOTIC/M.MEDIA (icra,icpr)
  44. 44. Community Comparison and normalization 50 45 40 35 30 [icc- globecom] h-index [ijcai- aaai] 25 [icra- icpr] 20 [iscas- date] [kbse- icse] 15 10 5 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 Author
  45. 45. Hi Alex! Create Journal U You haven’t created any LJ yet. Start sharing scientific content!
  46. 46. Hi Alex! Untitled 1 General Info Journal Name Peer-review journal Sources to 2 overlay Description Journal that groups the relevant Information about peer-review 3 Process Editorial board Joe Alex Joe Cancel Next
  47. 47. Hi Alex! Peer-review journal To be included in your 1 General Info journal I WANT Sources to peer review Test • papers, 2 • blogs, overlay • datasets TALKING ABOUT 3 Process Contributions Sources peer-review papers Springer FROM blogs Arxiv Springer dataset Scopus Arxiv wikis Scholar UPDATE RATE weekly Edit relations | Advanced filters Rank by citations Peer review, an overview [none] Springer Peter, Pablo, Pedro (0) LJ references The world of scientif ic publications has been largely… (4) citations Peer review, scientific publications Rethinking peer review [Carl’s LJ] Arxiv Joe (1) LJ references Exploring new ideas f or peer review in physics (3) citations Peer review, open access Anonymized review data [Liquidpub LJ] Arxiv Juan, Alejandro (1) LJ references Review data f rom 10 conf erences in biology (1) citations Peer review, Cancel Next
  48. 48. Hi Alex! Peer-review journal 1 General Info Sources to Traditional journal process 2 overlay Model f or traditional journals 3 Process Publication via voting “Democratic model” f or agreement Total agreement Full agreement model Cancel Done
  49. 49. Hi Alex! Peer-review journal Create Issue Update Now Latests activities Journal Created Home Springer 11 Apr 9.00am Peer review, an overview [none] Content Issues Peter, Pablo, Pedro (0) LJ references The world of scientif ic publications has … (4) citations Stats Latests subscribers Settings Peer review, scientific publications, [no subscribers yet] Rethinking peer review [Carl’s LJ] Arxiv Joe (1) LJ references General Info (3) citations Exploring new ideas f or peer review in … Created on 11 Apr 9.00am Owner Alex Peer review, open access Subscribers 0 Anonymized review data [Liquidpub LJ] Arxiv Editors Juan, Alejandro (1) LJ references Review data f rom 10 conf erences in … (1) citations Alex Peer review, Joe Settings Issues [no issues yet] U We are still learning from you. Soon you’ll get our suggestions :) Disable Settings U No suggestions from your friends Disable
  50. 50. Hi Alex! Peer-review journal Create Issue Add Contribution Citations Latests activities Joe joined the LJ Home Springer 11 Apr 9.02am Peer review, an overview [none] Content Journal Created by Alex Issues Peter, Pablo, Pedro (0) LJ references 11 Apr 9.00am The world of scientif ic publications has … (4) citations Latests subscribers Stats Settings Peer review, scientific publications, [no subscribers yet] Rethinking peer review [Carl’s LJ] Arxiv Editors Joe (1) LJ references Exploring new ideas f or peer review in … (3) citations Popular Tags Alex Peer review, open access Peer review Scientific publications Joe Anonymized review data [Liquidpub LJ] Arxiv Open access Juan, Alejandro (1) LJ references Review data f rom 10 conf erences in … (1) citations Issues Peer review, [no issues yet]
  51. 51. Hi Alex! Peer-review journal » Issue on new review models Create Issue Published Latests activities “Issue on new review models” moved to Home Springer the Publication phase, by Joe Peer review, an overview [none] 11 Apr 9.09am Content Issues Peter, Pablo, Pedro (0) LJ references (1) Contributions added to the Issue on The world of scientif ic publications has … (4) citations new review models” by Alex Stats 11 Apr 9.08am Settings Peer review, scientific publications, by Alex “Issue on new review models” moved to the Selection phase, by Alex Rethinking peer review [Carl’s LJ] Arxiv 11 Apr 9.07am Editors Joe (1) LJ references Latests subscribers Exploring new ideas f or peer review in … (3) citations Alex Peer review, open access by Alex [no subscribers yet] Joe Benefits of open peer review [none] Arxiv Charles Popular Tags This paper illustrates the benef its of … (0) LJ references (2) citations Peer review Issues Peer review, open access Scientific publications Issue on new by Joe Open access review models (4) Peer review, past and future [none] Ronald, James Scopus (0) LJ references This paper illustrates the benef its of … (1) citations Peer review by Alex
  52. 52. Take-Home Message  Flaws of current practices (or, lack of evidence that they work as expected)  Research services for novel dissemination model  Principles, models, composable IT services in a restricted domain  Liquid journals: Interestingness, reward innovation, sharing, everything contributes, use the filtering power of the community project.liquidpub.org Fabio Casati - ECOWS 2009 53
  53. 53.  Google liquidpub-announce and subscribe if you are interested (VERY low traffic)  First release at the end of the month project.liquidpub.org Fabio Casati - ECOWS 2009 54
  54. 54. How to collaborate  Liquid journals server  http://demo.liquidpub.org/liquidjournals/  Liquid journals dashboard  http://demo.liquidpub.org/liquidjournals/web/work space/  Liquid journals SVN  https://dev.liquidpub.org/svn/liquidpub/prototype /liquidjournals/
  55. 55. How to collaborate  http://project.liquidpub.org/karaku/  http://project.liquidpub.org/resman/ project.liquidpub.org Fabio Casati - ECOWS 2009 56
  56. 56. 57

×