Badges to Acknowledge Open Practices

642 views

Published on

Presented at IASSIST 2014 meeting, Toronto, Canada.

Published in: Science, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
642
On SlideShare
0
From Embeds
0
Number of Embeds
95
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Merton’s norms of science (1942) - http://iie.fing.edu.uy/ense/asign/hciencia/trabs2001/victor/docs/merton.html

    We can understand the nature of the challenge with existing psychological theory. For example:
    1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman)
    2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others.
    3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias.
    4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. …
    5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay.


    Communality – open sharing with colleagues; Secrecy
    Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
    Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
    Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
    Quality – seek quality contributions; Quantity – seek high volume
  • 3,247 mid- and early-career scientists who had research funding from NIH.

    ideal to which most scientists subscribe

    scientists perceptions of their own behavior

    scientists perceptions of their peer’s behaviors

    self-regulation, substantiall autonomy, the complexity of scientific projects, professional expertise, innovative work on cutting-edge problems, and a system of largely voluntary compliance with regulation and codes of ethics all point to the futility and inadvisability of direct administrative control over scientists’ behavior

    Anderson MS, Martinson BC, De Vries R. Normative dissonance in science: Results from a national survey of U.S. scientists. Journal of Empirical Research on Human Research Ethics.2007;2(4):3–14.
  • We can understand the nature of the challenge with existing psychological theory. For example:
    1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman)
    2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others.
    3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias.
    4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. …
    5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay.
  • think about very simple interventions that people can adopt

    top down versus bottom up
    community bottom up
    exposing users to good practices
    training, workshops,
    active in community with publishing,
    going to where the user is now,
    stat consultant “shock troops”,
    help reduce barriers and connect people to OSF tools that help them get there
    community top down
    committees,
    journals,
    feds,
    etc.
  • Badges to Acknowledge Open Practices

    1. 1. Badges to Acknowledge Open Practices Andrew Sallans Center for Open Science http://cos.io IASSIST June 2014
    2. 2. Openness is a core value of scientific practice
    3. 3. Challenges: Perceived norms Norms Communality Open Sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Counternorms Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity
    4. 4. Anderson, Martinson & DeVries, 2007
    5. 5. Challenges ● Perceived norms (Anderson, Martinson & DeVries, 2007) ● Motivated reasoning (Kunda, 1990) ● Minimal accountability (Lerner & Tetlock, 1999) ● I am busy (Everyone)
    6. 6. How might this change?
    7. 7. Top-down ● Committees ● Journal Editors ● Federal Agencies ● University administrators Bottom-up ● Exposing users to good practices ● Training sessions ● Workshops ● Spreading the message through active publishing ● Going to where the user is now ● Consultants serving as “Shock troops” Focus on simple interventions that researchers can actually adopt.
    8. 8. How to get a badge... 1. Disclosure --- author provides public statement that they’ve met badge criteria 1. Peer Reviewed --- independent review of author’s public statement and meeting of criteria
    9. 9. What are the criteria? ● Persistent path to the data, materials, pre registration, etc. ● Sufficient information for an independent person to reproduce the results ● Other more specialized items...
    10. 10. Who endorses these badges?
    11. 11. Which journals are adopting?
    12. 12. Going further…baking badges
    13. 13. What comes next? ● Improve the badge issuing workflow ● Refine integration into peer review process ● Gain adoption of more journals/organizations ● Measure effect
    14. 14. Lastly, this is a community effort... Ben B. Blohowiak | Johanna Cohoon | Lee de-Wit | Eric Eich | Frank J. Farach | Roger Giner-Sorolla | Fred Hasselman | Alex O. Holcombe | Macartan Humphreys | Melissa Lewis | Brian A. Nosek | Jonathan Peirce | Andrew Sallans | Jeffrey R. Spies | Chris Seto | Sara Bowman Note: bold are non-COS
    15. 15. Thanks! Find out more and get involved here: https://osf.io/tvyxz/ Email me: andrew@cos.io Twitter: @asallans

    ×