Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web
Upcoming SlideShare
Loading in...5

Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web



This talk tries to unveil some of the problems inherent in the current knowledge creation, dissemination, and evaluation practices, also based on models and quantitative analyses of the effectiveness ...

This talk tries to unveil some of the problems inherent in the current knowledge creation, dissemination, and evaluation practices, also based on models and quantitative analyses of the effectiveness of peer review as gatekeeping/assessment method and of citations as measure of impact. The speaker will present the recent research and development threads aiming at making the knowledge generation and dissemination process efficient, and the evaluation process (more) fair and accurate. He will in particular present the models and tools being developed to this end, which are essentially based on applying to knowledge dissemination the lessons learned from open source development and the social web. The presentation will be interactive and discussion-oriented.



Total Views
Views on SlideShare
Embed Views



4 Embeds 10 5 3 1 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web Presentation Transcript

  • Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web Maurizio Marchese, Aliaksandr Birukou, Fabio Casati and the LiquidPub team Department of Information Engineering and Computer Science - DISI University of Trento, Italy The Tuesday Conversation, January 12, 2010 – DG-INFSO-Brussels
  • Challenge: doing science in the 21st • The Web has changed many fields: ▫ News (blogs, RSS feeds, ...) ▫ Music (p2p networks, iTunes, lastFM, …) ▫ Travel (Orbiz, Google maps,…) ▫ Photos (Flikr, …) ▫ … • Has it changed also scientific knowledge production and dissemination processes ?
  • Challenge: doing science in the 21st • Yes ! But - so far - mainly ▫ new and faster dissemination access channels ▫ distributed working environment ▫ … • Scientific knowledge processes are still based on the traditional notion of “paper” publication and on peer review as quality assessment method ▫ Philosophical Transactions of the Royal Society of London— founded in 1665 ▫ Journal des scavans— 1665 ▫ Royal Society of Edinburgh’s Medical Essays and Observations, - 1731, introduces peer review as we would recognize it today
  • We have a dream Capture the lessons learned and opportunities provided by the Web and open source, agile development to develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming scientific knowledge. Understand what’s good for science, and make it happen •Publish and perish: why the current publication and review model is killing research and wasting your money (ACM Ubiquity 8(3), Feb 2007), and •Liquid Publications: Scientific Publications meet the Web From. Fabio Casati, Fausto Giunchiglia and Maurizio Marchese.
  • 5 Objectives Understand Improve ▫ Peer review and innovation ▫ Better ways to do the same ▫ Evaluation processes, and things quality/impact of research ▫ Better ways to do different (people, papers, projects) and new things ▫ Dissemination models and overhead ▫ Principles ▫ Scientific communities ▫ Models ▫ IT services
  • PR: Initial Goals • Understand how well peer review works • Metrics + Analysis • Understand how to improve the process • Gatekeeping aspects (in/out) • Quality improvements “Manuscript Quality before and after Peer Review and Editing at Annals of “Not everything that can be counted counts, Internal Medicine” and not everything that counts can Goodman S.N., Berlin J., Fletcher be counted.” -- Albert Einstein S.W., Fletcher R.H.
  • Metric Dimensions Quality Divergence Kendall Distance Disagreement Robustness Effort vs. quality Statistics Effort invariant alternatives Unbiasing Biases Min. criteria Fairness Efficiency
  • Quality-related Metrics: real vs. ideal • Real peer review ranking vs. ideal ranking ▫ Ideal ? Subjective vs. Objective But each process could/should define approximate indicators of quality like: citations, downloads, community voting, success in a second phase, publication, citations, patents… • IF an approximate ideal ranking is available we can measure the difference in various ways, e.g ▫ Kendall τ distance / Kendall τ rank correlation ▫ Divergence metric
  • Comparing rankings
  • Comparing rankings
  • Divergence Metric When the second ranking is random, we have: t NDivρ i ρ a (t, n, C ) = ∑ pt (i)wi i=0 ©Ndiv(1,n, C) = n-1/n inv. correlated e.g. √ N-Divergence ⎛ 1⎞ ⎛ 0⎞ NDivρ i ρ a (1, n, C ) = p1 (0) + p1 (1) ⎝ 1⎠ ⎝ 1⎠ indipendent n − 1 ⎛ 1⎞ = n ⎝ 1⎠ correlated .... Cit C tn−t t −i © pt (i) = −i ;wi = √ n Ct t Nt = 1/n Nt = n/n Normalized t
  • prior vs. after discussion ca. 74 (36%) contributions have been effected NDiv(53, 206, 206) = 0, 36 by the discussion phase
  • Results: peer review ranking vs. citation count Div Normalized t 14
  • Fairness • Definition: A review process is fair if and only of the acceptance of a contribution does not depend on the particular set of PC members that reviews it • The key is in the assignment of a paper to reviewers: a paper assignment is unfair if the specific assignment influences (makes more predictable) the fate of the paper.
  • Computed Normalized Rating Biases C1 C2 C3 C4 top accepting 2,66 3,44 1,52 1,17 top rejecting -1,74 -2,78 -2,06 -1,17 > + |min bias| 13% 5% 9% 7% < - |min bias| 12% 4% 8% 7% C1 C2 C3 C4 Unbiasing effect (divergence) 13% 9% 11% 14% Unbiasing effect (reviewers affected) 10 16 5 4
  • 18/13 Disagreement metric • Through this metric we compute the similarity between the marks given by the reviewers on the same contribution. • The rationale behind this metric is that in a review process we expect some kind of agreement between reviewers.
  • 19/13 Normalized Disagreement
  • Disagreement vs number of reviews
  • Estimation of the "optimized” number of proposals per reviewer • There are different groups of papers, for example: bad, moderate and good papers • We want to distribute the papers among reviewers in a way such that each reviewer will have statistically at least one paper from each group ▫ In this way the reviewer will have a better view of the overall quality of the papers
  • Estimation of the "optimized” number of papers per reviewer
  • The road ahead Real-Time accuracy estimation Ranking vs. marking Speed Ranking
  • 24 Principles Models IT Services
  • Principles and objectives • Everything counts!! (not just papers, not just “innovations”) • Minimal dissemination overhead • Early sharing • Early feedback/interaction • Find diversity • Interestingness and sharing as measures of reputation • No gatekeeping. Use the filtering power of the community ▫ We are not necessarily right!! Let the community select the principles and models
  • A large number of technologies are out there
  • Blogs Wikis Collaborative tagging and social bookmarking Scientific Search Services Journals with collaborative peer review processs More Complex Systems
  • But how they can be effectively used ? • Let’s explore some dimensions of the issue ▫ Agile, Collaborative, Open Source “scientific” processes LiquidBook ▫ New models for dissemination, sharing, interactions, evaluation LiquidJournal
  • 29
  • Liquid Books Initial Definition
  • A Scenario: Text books
  • LiquidBook Structure
  • LiquidBook Lifeycycle
  • Liquid Book: State of the art and what’s new o WikiBooks: open‐content textbooks ‐ community for creating a free library of  educational textbooks that anyone can edit o Differently from WikiBook you can have different roles in the community: o Authors o Contributors o People who just rate, write comments/reviews o We want to offer a (legal) framework to authors to easier collaboration o Tailored material for different needs (classes, professionals). Several Personalized Editions which stay up‐to‐date with the current state of the art. o Multi‐Faceted Content (presentations, excercises are available too) o Sharing and reusing of content among a trusted network of authors, who  guarantee the quality
  • Examples of collaboratively written  books o How to Think Like a Computer Scientist series of  publications by Green Tea Press, where the same  core programming text has been adapted to several  different programming languages o 97 Things Every Programmer Should Know example  of collaborative written book, with hundreds of  contributors o Business Model Generation example of collaborative  written book, with 470 co‐authors and without a  publisher
  • 36
  • Journals today • Based on traditional notion of paper • Traditional peer-review • Solid in nature • Established reputation
  • The Web Era The Web has changed the way we get, share, produce and consume scientific content Internet How do I get  How do I make  interesting  my work visible! Authors content! Readers
  • Journals: revisited • Original reasons for the current model are gone • Back to the roots: How to provide interesting content? blogs papers datasets
  • Liquid journals: Proposal
  • Scientific contributions • Different types, maturity and certification levels
  • Filling a LJ • Editor decides what to put liquid Editor
  • Consuming a LJ Readers
  • Demo at
  • Hi Alex! Peer-review journal Create Issue Update Now Latests activities Journal Created Home 11 Apr 9.00am Peer review, an overview [none] Springer Content Issues Peter, Pablo, Pedro (0) LJ references Stats The world of scientific publications has … (4) citations Latests subscribers Settings Peer review, scientific publications, [no subscribers yet] Rethinking peer review [Carl’s LJ] Arxiv Joe (1) LJ references General Info Exploring new ideas for peer review in … (3) citations Created on 11 Apr 9.00am Owner Alex Peer review, open access Subscribers 0 Anonymized review data [Liquidpub LJ] Arxiv Editors Juan, Alejandro (1) LJ references Review data from 10 conferences in … (1) citations Alex Peer review, Joe Settings Issues [no issues yet] We are still learning from you. Disable Soon you’ll get our suggestions :) Settings No suggestions from your friends Disable
  • Liquid journals: Characteristics • Filled Semi-automatically (query based) • Multi-faceted content • Separation of knowledge production from publication • Use the editing power of the community • Subscribe to the editors you trust • Use the wisdom of the community
  • Liquid journals: Benefits • Everybody becomes a journal editor (selfishly) • No gatekeeping, quality enforced by the community (gatekeeping is the noise) • You get things you want to read (or you want your group to read) • You get diversity • You leverage the selection work of your friends • Measure interestingness and quality by sharing • Reduce (optimize) dissemination overhead and encourage early sharing
  • 48 Related work • Reference management tools (Mendeley/CiteULike collections) ▫ A way to share set of papers among a group of peers ▫ LJ are not just manually edited collections We provide automatic feed of new content using “liquidity” of queries Editor can organize contributions in issues, just like in traditional journals Flexible workflow (maintained by Gelee) • Overlaid journals (RIOJA) ▫ Provide an interface on top of public (open access) repositories LJ does not only focus on open access articles, but includes also articles from different digital libraries and preliminary ideas from blogs ▫ Whole idea is even older – see (Smith 2000)
  • Engineer a system that • Accesses heterogeneous src w/no API • Is modular so that functions can be reused ▫ In a world where modularity had little success • Facilitates the creation of arbitrary dissemination and evaluation models (by non- programmers) • Provides commonly needed research services • Implements liquid journals (agile – our main macro-story) • Simple and lightweight
  • Liquid Journals dashboard Liquid journals API Search  Personalized  Sharing computing tagging Diversity Caching Reputation Cached metadata Search by  topic Crawling Access Disambigua tion
  • UI A Research services P I Crawling Cache mgmt Karaku client‐code Subscriptions/stream Disambiguation Personalized tagging REST API Liquidity Karaku Rendering REST API Cached SKOs ResMan Adapters REST REST API API REST REST API API Fabio Casati - ECOWS 2009 52
  • [Insert header here] [Insert footer]
  • Vertical services Liquid journals Liquid books … Collaborative  programming Gelee Search computing Mashart Community  TopicSearch Reseval discovery
  • [Insert footer] [Insert header here] Gelee Paper at ICDE 09. Video avail online and demo at icsoc
  • Mashart Paper at ER’09, demo at icsoc ‘09
  • project.liquidpu RESEVAL complex’09
  • Group Management
  • Overview of the Community Network SOFTWARE ENGINEERING(kbse,icse) DIST. SYSTEM/COMPILER(ipps,iccS) APPLIED COMPUTING/CRYPTO(sac,compsac) TELETEACHING/HUM_INT (chi,hicss) TELECOM (icc,globecom) HUMMAN –COMP INTER(icchp,hci) GENETIC AND EVO ALG(cec,gecco) AI/DB (icai,aaai) ROBOTIC/M.MEDIA (icra,icpr)
  • Take-Home Message • Flaws of current practices (or, lack of evidence that they work as expected) • Research services for novel dissemination model • Principles, models, composable IT services in a restricted domain • Use cases: ▫ LiquidBook: Sharing and reusing of content among a trusted network of authors, who guarantee the quality ▫ LiquidJournals: Interestingness, reward innovation, sharing, everything contributes, use the filtering power of the community
  • Collabiration with • The Institute for Computer Sciences, Social- Informatics and Telecommunications Engineering: an international society, focuses on ICT in its broadest sense • Supports research, innovation and technology transfer in IT • Collaboration on: ▫ LiquidJournals will be fed with data from ICST’s eScripts (electronic journals) and distributed via PeerNet (social network) ▫ Courseware platform – integration with LiquidBooks ▫ Review analysis – using Assyst (conf management system) as a data source
  • Collaboration with other projects
  • 63 To know more • Google liquidpub-announce and subscribe if you are interested (VERY low traffic) • First release at the end of the month
  • Thank you