Reshaping Scientific Knowledge
 Dissemination and Evaluation in the Age of
 the Web




Maurizio Marchese, Aliaksandr Biru...
Challenge: doing science in the 21st
• The Web has changed many fields:
  ▫   News (blogs, RSS feeds, ...)
  ▫   Music (p2...
Challenge: doing science in the 21st
• Yes ! But - so far - mainly
  ▫ new and faster dissemination access channels
  ▫ di...
We have a
  dream
Capture the lessons learned and opportunities
provided by the Web and open source, agile
development to ...
5




Objectives
Understand                       Improve
  ▫ Peer review and innovation     ▫ Better ways to do the same
...
PR: Initial Goals
• Understand how well peer
  review works
• Metrics + Analysis
• Understand how to improve
  the process...
Metric Dimensions
                                Quality


                       Divergence
                            ...
Quality-related Metrics:
real vs. ideal
• Real peer review ranking vs. ideal ranking
  ▫ Ideal ?
      Subjective vs. Obje...
Comparing rankings
Comparing rankings
Divergence Metric



When the second ranking is random, we have:
                         t
 NDivρ i ρ a (t, n, C ) = ∑ pt...
prior vs. after discussion




                                ca. 74 (36%) contributions have been effected
   NDiv(53, 2...
Results:
 peer review ranking vs. citation count
Div




                   Normalized t




                             ...
Fairness

• Definition: A review process is fair if and only
  of the acceptance of a contribution does not
  depend on th...
Computed Normalized Rating Biases

                             C1               C2           C3               C4

 top ac...
18/13




Disagreement metric
• Through this metric we compute the similarity
  between the marks given by the reviewers o...
19/13




Normalized Disagreement
Disagreement vs number of reviews
Estimation of the "optimized”
number of proposals per reviewer
• There are different groups of papers, for
  example: bad,...
Estimation of the "optimized”
number of papers per reviewer
The road ahead
Real-Time accuracy estimation




                                Ranking vs. marking




      Speed Ranki...
24




Principles
Models
IT Services
Principles and objectives
• Everything counts!! (not just papers, not just
  “innovations”)
• Minimal dissemination overhe...
A large number of technologies are out there
Blogs



Wikis



Collaborative tagging and social bookmarking



Scientific Search Services



Journals with collaborativ...
But how they can be effectively
used ?
• Let’s explore some dimensions of the issue

 ▫ Agile, Collaborative, Open Source ...
29
Liquid Books Initial Definition
A Scenario: Text books
LiquidBook Structure
LiquidBook Lifeycycle
Liquid Book: State of the art and
what’s new
o WikiBooks: open‐content textbooks ‐ community for creating a free library o...
Examples of collaboratively written 
books
o How to Think Like a Computer Scientist series of 
  publications by Green Tea...
36
Journals today

• Based on traditional
  notion of paper
• Traditional peer-review
• Solid in nature
• Established reputat...
The Web Era
  The Web has changed the way we get, share, produce and consume
    scientific content




                  ...
Journals: revisited
• Original reasons for the current model are gone
• Back to the roots: How to provide interesting
  co...
Liquid journals: Proposal
Scientific contributions
• Different types, maturity and certification levels
Filling a LJ
• Editor decides what to put



                                        liquid




                          ...
Consuming a LJ




                 Readers
Demo at
http://project.liquidpub.org/research-areas/liquid-journal
Hi Alex!




                           Peer-review journal
     Create Issue                                             ...
Liquid journals: Characteristics

• Filled Semi-automatically (query based)
• Multi-faceted content
• Separation of knowle...
Liquid journals: Benefits
• Everybody becomes a journal editor (selfishly)
• No gatekeeping, quality enforced by the
  com...
48



Related work
• Reference management tools (Mendeley/CiteULike
  collections)
  ▫ A way to share set of papers among ...
Engineer a system that
• Accesses heterogeneous src w/no API
• Is modular so that functions can be reused
 ▫ In a world wh...
Liquid Journals dashboard


                   Liquid journals API



     Search                   Personalized 
        ...
UI
                     A
Research services    P
                     I   Crawling
                         Cache mgmt
Kar...
[Insert header here]   [Insert footer]
Vertical services
 Liquid journals      Liquid books             …




                                                   ...
[Insert footer]   [Insert header
                                                                   here]




Gelee
Paper ...
Mashart
Paper at ER’09, demo at icsoc ‘09
project.liquidpu
                                                b.org




RESEVAL    http://demo.liquidpub.org/reseval/  ...
Group Management
Overview of the Community Network

                                                              SOFTWARE ENGINEERING(kbse...
Take-Home Message
• Flaws of current practices (or, lack of evidence that
  they work as expected)
• Research services for...
Collabiration with ICST.org
• The Institute for Computer Sciences, Social-
  Informatics and Telecommunications Engineerin...
Collaboration with other projects
http://project.liquidpub.org/collaboration/collaboration
63




To know more
• Google liquidpub-announce and subscribe if you
  are interested (VERY low traffic)
• First release a...
Thank you
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web
Upcoming SlideShare
Loading in …5
×

Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web

1,274 views

Published on

This talk tries to unveil some of the problems inherent in the current knowledge creation, dissemination, and evaluation practices, also based on models and quantitative analyses of the effectiveness of peer review as gatekeeping/assessment method and of citations as measure of impact. The speaker will present the recent research and development threads aiming at making the knowledge generation and dissemination process efficient, and the evaluation process (more) fair and accurate. He will in particular present the models and tools being developed to this end, which are essentially based on applying to knowledge dissemination the lessons learned from open source development and the social web. The presentation will be interactive and discussion-oriented.

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,274
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
36
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web

  1. 1. Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web Maurizio Marchese, Aliaksandr Birukou, Fabio Casati and the LiquidPub team Department of Information Engineering and Computer Science - DISI University of Trento, Italy The Tuesday Conversation, January 12, 2010 – DG-INFSO-Brussels
  2. 2. Challenge: doing science in the 21st • The Web has changed many fields: ▫ News (blogs, RSS feeds, ...) ▫ Music (p2p networks, iTunes, lastFM, …) ▫ Travel (Orbiz, Google maps,…) ▫ Photos (Flikr, …) ▫ … • Has it changed also scientific knowledge production and dissemination processes ?
  3. 3. Challenge: doing science in the 21st • Yes ! But - so far - mainly ▫ new and faster dissemination access channels ▫ distributed working environment ▫ … • Scientific knowledge processes are still based on the traditional notion of “paper” publication and on peer review as quality assessment method ▫ Philosophical Transactions of the Royal Society of London— founded in 1665 ▫ Journal des scavans— 1665 ▫ Royal Society of Edinburgh’s Medical Essays and Observations, - 1731, introduces peer review as we would recognize it today
  4. 4. We have a dream Capture the lessons learned and opportunities provided by the Web and open source, agile development to develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming scientific knowledge. Understand what’s good for science, and make it happen project.liquidpub.org •Publish and perish: why the current publication and review model is killing research and wasting your money (ACM Ubiquity 8(3), Feb 2007), and •Liquid Publications: Scientific Publications meet the Web From. www.52en.com/img/dream_01.jpg Fabio Casati, Fausto Giunchiglia and Maurizio Marchese.
  5. 5. 5 Objectives Understand Improve ▫ Peer review and innovation ▫ Better ways to do the same ▫ Evaluation processes, and things quality/impact of research ▫ Better ways to do different (people, papers, projects) and new things ▫ Dissemination models and overhead ▫ Principles ▫ Scientific communities ▫ Models ▫ IT services
  6. 6. PR: Initial Goals • Understand how well peer review works • Metrics + Analysis • Understand how to improve the process • Gatekeeping aspects (in/out) • Quality improvements “Manuscript Quality before and after Peer Review and Editing at Annals of “Not everything that can be counted counts, Internal Medicine” and not everything that counts can Goodman S.N., Berlin J., Fletcher be counted.” -- Albert Einstein S.W., Fletcher R.H.
  7. 7. Metric Dimensions Quality Divergence Kendall Distance Disagreement Robustness Effort vs. quality Statistics Effort invariant alternatives Unbiasing Biases Min. criteria Fairness Efficiency
  8. 8. Quality-related Metrics: real vs. ideal • Real peer review ranking vs. ideal ranking ▫ Ideal ? Subjective vs. Objective But each process could/should define approximate indicators of quality like: citations, downloads, community voting, success in a second phase, publication, citations, patents… • IF an approximate ideal ranking is available we can measure the difference in various ways, e.g ▫ Kendall τ distance / Kendall τ rank correlation ▫ Divergence metric
  9. 9. Comparing rankings
  10. 10. Comparing rankings
  11. 11. Divergence Metric When the second ranking is random, we have: t NDivρ i ρ a (t, n, C ) = ∑ pt (i)wi i=0 ©Ndiv(1,n, C) = n-1/n inv. correlated e.g. √ N-Divergence ⎛ 1⎞ ⎛ 0⎞ NDivρ i ρ a (1, n, C ) = p1 (0) + p1 (1) ⎝ 1⎠ ⎝ 1⎠ indipendent n − 1 ⎛ 1⎞ = n ⎝ 1⎠ correlated .... Cit C tn−t t −i © pt (i) = −i ;wi = √ n Ct t Nt = 1/n Nt = n/n Normalized t
  12. 12. prior vs. after discussion ca. 74 (36%) contributions have been effected NDiv(53, 206, 206) = 0, 36 by the discussion phase
  13. 13. Results: peer review ranking vs. citation count Div Normalized t 14
  14. 14. Fairness • Definition: A review process is fair if and only of the acceptance of a contribution does not depend on the particular set of PC members that reviews it • The key is in the assignment of a paper to reviewers: a paper assignment is unfair if the specific assignment influences (makes more predictable) the fate of the paper.
  15. 15. Computed Normalized Rating Biases C1 C2 C3 C4 top accepting 2,66 3,44 1,52 1,17 top rejecting -1,74 -2,78 -2,06 -1,17 > + |min bias| 13% 5% 9% 7% < - |min bias| 12% 4% 8% 7% C1 C2 C3 C4 Unbiasing effect (divergence) 13% 9% 11% 14% Unbiasing effect (reviewers affected) 10 16 5 4
  16. 16. 18/13 Disagreement metric • Through this metric we compute the similarity between the marks given by the reviewers on the same contribution. • The rationale behind this metric is that in a review process we expect some kind of agreement between reviewers.
  17. 17. 19/13 Normalized Disagreement
  18. 18. Disagreement vs number of reviews
  19. 19. Estimation of the "optimized” number of proposals per reviewer • There are different groups of papers, for example: bad, moderate and good papers • We want to distribute the papers among reviewers in a way such that each reviewer will have statistically at least one paper from each group ▫ In this way the reviewer will have a better view of the overall quality of the papers
  20. 20. Estimation of the "optimized” number of papers per reviewer
  21. 21. The road ahead Real-Time accuracy estimation Ranking vs. marking Speed Ranking
  22. 22. 24 Principles Models IT Services
  23. 23. Principles and objectives • Everything counts!! (not just papers, not just “innovations”) • Minimal dissemination overhead • Early sharing • Early feedback/interaction • Find diversity • Interestingness and sharing as measures of reputation • No gatekeeping. Use the filtering power of the community ▫ We are not necessarily right!! Let the community select the principles and models
  24. 24. A large number of technologies are out there
  25. 25. Blogs Wikis Collaborative tagging and social bookmarking Scientific Search Services Journals with collaborative peer review processs More Complex Systems
  26. 26. But how they can be effectively used ? • Let’s explore some dimensions of the issue ▫ Agile, Collaborative, Open Source “scientific” processes LiquidBook ▫ New models for dissemination, sharing, interactions, evaluation LiquidJournal
  27. 27. 29
  28. 28. Liquid Books Initial Definition
  29. 29. A Scenario: Text books
  30. 30. LiquidBook Structure
  31. 31. LiquidBook Lifeycycle
  32. 32. Liquid Book: State of the art and what’s new o WikiBooks: open‐content textbooks ‐ community for creating a free library of  educational textbooks that anyone can edit o Differently from WikiBook you can have different roles in the community: o Authors o Contributors o People who just rate, write comments/reviews o We want to offer a (legal) framework to authors to easier collaboration o Tailored material for different needs (classes, professionals). Several Personalized Editions which stay up‐to‐date with the current state of the art. o Multi‐Faceted Content (presentations, excercises are available too) o Sharing and reusing of content among a trusted network of authors, who  guarantee the quality
  33. 33. Examples of collaboratively written  books o How to Think Like a Computer Scientist series of  publications by Green Tea Press, where the same  core programming text has been adapted to several  different programming languages o 97 Things Every Programmer Should Know example  of collaborative written book, with hundreds of  contributors o Business Model Generation example of collaborative  written book, with 470 co‐authors and without a  publisher
  34. 34. 36
  35. 35. Journals today • Based on traditional notion of paper • Traditional peer-review • Solid in nature • Established reputation
  36. 36. The Web Era The Web has changed the way we get, share, produce and consume scientific content Internet How do I get  How do I make  interesting  my work visible! Authors content! Readers
  37. 37. Journals: revisited • Original reasons for the current model are gone • Back to the roots: How to provide interesting content? blogs papers datasets
  38. 38. Liquid journals: Proposal
  39. 39. Scientific contributions • Different types, maturity and certification levels
  40. 40. Filling a LJ • Editor decides what to put liquid Editor
  41. 41. Consuming a LJ Readers
  42. 42. Demo at http://project.liquidpub.org/research-areas/liquid-journal
  43. 43. Hi Alex! Peer-review journal Create Issue Update Now Latests activities Journal Created Home 11 Apr 9.00am Peer review, an overview [none] Springer Content Issues Peter, Pablo, Pedro (0) LJ references Stats The world of scientific publications has … (4) citations Latests subscribers Settings Peer review, scientific publications, [no subscribers yet] Rethinking peer review [Carl’s LJ] Arxiv Joe (1) LJ references General Info Exploring new ideas for peer review in … (3) citations Created on 11 Apr 9.00am Owner Alex Peer review, open access Subscribers 0 Anonymized review data [Liquidpub LJ] Arxiv Editors Juan, Alejandro (1) LJ references Review data from 10 conferences in … (1) citations Alex Peer review, Joe Settings Issues [no issues yet] We are still learning from you. Disable Soon you’ll get our suggestions :) Settings No suggestions from your friends Disable
  44. 44. Liquid journals: Characteristics • Filled Semi-automatically (query based) • Multi-faceted content • Separation of knowledge production from publication • Use the editing power of the community • Subscribe to the editors you trust • Use the wisdom of the community
  45. 45. Liquid journals: Benefits • Everybody becomes a journal editor (selfishly) • No gatekeeping, quality enforced by the community (gatekeeping is the noise) • You get things you want to read (or you want your group to read) • You get diversity • You leverage the selection work of your friends • Measure interestingness and quality by sharing • Reduce (optimize) dissemination overhead and encourage early sharing
  46. 46. 48 Related work • Reference management tools (Mendeley/CiteULike collections) ▫ A way to share set of papers among a group of peers ▫ LJ are not just manually edited collections We provide automatic feed of new content using “liquidity” of queries Editor can organize contributions in issues, just like in traditional journals Flexible workflow (maintained by Gelee) • Overlaid journals (RIOJA) ▫ Provide an interface on top of public (open access) repositories LJ does not only focus on open access articles, but includes also articles from different digital libraries and preliminary ideas from blogs ▫ Whole idea is even older – see (Smith 2000)
  47. 47. Engineer a system that • Accesses heterogeneous src w/no API • Is modular so that functions can be reused ▫ In a world where modularity had little success • Facilitates the creation of arbitrary dissemination and evaluation models (by non- programmers) • Provides commonly needed research services • Implements liquid journals (agile – our main macro-story) • Simple and lightweight
  48. 48. Liquid Journals dashboard Liquid journals API Search  Personalized  Sharing computing tagging Diversity Caching Reputation Cached metadata Search by  topic Crawling Access Disambigua tion
  49. 49. UI A Research services P I Crawling Cache mgmt Karaku client‐code Subscriptions/stream Disambiguation Personalized tagging REST API Liquidity Karaku Rendering REST API Cached SKOs ResMan Adapters REST REST API API REST REST API API Fabio Casati - ECOWS 2009 52
  50. 50. [Insert header here] [Insert footer]
  51. 51. Vertical services Liquid journals Liquid books … Collaborative  programming Gelee Search computing Mashart Community  TopicSearch Reseval discovery
  52. 52. [Insert footer] [Insert header here] Gelee Paper at ICDE 09. Video avail online and demo at icsoc
  53. 53. Mashart Paper at ER’09, demo at icsoc ‘09
  54. 54. project.liquidpu b.org RESEVAL    http://demo.liquidpub.org/reseval/ complex’09
  55. 55. Group Management
  56. 56. Overview of the Community Network SOFTWARE ENGINEERING(kbse,icse) DIST. SYSTEM/COMPILER(ipps,iccS) APPLIED COMPUTING/CRYPTO(sac,compsac) TELETEACHING/HUM_INT (chi,hicss) TELECOM (icc,globecom) HUMMAN –COMP INTER(icchp,hci) GENETIC AND EVO ALG(cec,gecco) AI/DB (icai,aaai) ROBOTIC/M.MEDIA (icra,icpr)
  57. 57. Take-Home Message • Flaws of current practices (or, lack of evidence that they work as expected) • Research services for novel dissemination model • Principles, models, composable IT services in a restricted domain • Use cases: ▫ LiquidBook: Sharing and reusing of content among a trusted network of authors, who guarantee the quality ▫ LiquidJournals: Interestingness, reward innovation, sharing, everything contributes, use the filtering power of the community
  58. 58. Collabiration with ICST.org • The Institute for Computer Sciences, Social- Informatics and Telecommunications Engineering: an international society, focuses on ICT in its broadest sense • Supports research, innovation and technology transfer in IT • Collaboration on: ▫ LiquidJournals will be fed with data from ICST’s eScripts (electronic journals) and distributed via PeerNet (social network) ▫ Courseware platform – integration with LiquidBooks ▫ Review analysis – using Assyst (conf management system) as a data source
  59. 59. Collaboration with other projects http://project.liquidpub.org/collaboration/collaboration
  60. 60. 63 To know more • Google liquidpub-announce and subscribe if you are interested (VERY low traffic) • First release at the end of the month
  61. 61. Thank you

×