Communicating trust, enabling criticism


Published on

Talk given at the Research Without Borders panel on Communicating Computational Research at Columbia University on 4th April 2013.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • In this talk I’ll be considering where I think software needs to get to so that we can be “Doing Research Properly In A Digital World”.In particular, I’ll look at the increasing complexity of systems and what this means for understanding, and assessing whether the new generation of data sharers might change the way in which we collaborate.
  • The Software Sustainability Institute can help with: software reviews and refactoring, collaborations to develop your project, guidance and best practice on software development, project management, community building, publicity and more…Drawing on pool of specialists to drive the continued improvement and impact of research software developed by and for researchersProviding services for research software users and developersDeveloping research community interactions and capacityPromoting research software best practice and capability
  • Criticism is a key concept in research. Intellectual debate drives research forward.I propose that just as we must be able to peruse, criticise and build on traditional research results, the same must be true of those created digitally.This does not necessarily mean that researchers must make their work accessible to everyone – there is a mutual relationship that must be built such that the critic can claim to be qualified to understand and comment on the subject in the same way that the researcher must make their results open to scrutiny.“Episodis Critics”Photo by Camilay el ArteCC-BY-NC-SA
  • We are increasingly seeing the coupling of models, each a complex simulation in their own right.Sensors and instruments are providing ever increasing amounts of data to be analysed but it is about extracting knowledge, and specialists are increasingly necessary.Noone can grasp the entirety of the system, which is a cause for concern. Where is your gedanken now?“Paint by Particle – aerosol movement”Photo by NASA Goddard Photo and VideoCC-BYWinds disperse vast quantities of dust (red), sea salt (blue), sulphate (white) and black and organic carbon (green) around the world.
  • Code is complex and often treated like a black box. It has helped us to improve the efficiency and accuracy of our research, but it also means that there are plenty of black boxes.
  • Simulation of tamiflu bound to a drug-resistant mutant of influenza neuraminidase. The red parts of the protein prefer to be bound to the drug than water, while the blue parts prefer the water. At the start of the movie there are two blue residues at the top, with a red residue between them. The blue residue on the left is the mutant residue that confers drug resistance to the virus.
  • How much must we take on trust to allow knowledge to scale?The very fact that I can show you this piece of code is a huge start – that it is being reviewed is another.In this case the simulations are completely reproducible as they are based on a set random number seed and are thus deterministic. However, changes in the software may change the order in which random numbers are drawn, so can only guarantee that the simulation is reproducible for this version of the software.
  • A common issue is the question of “how much is enough” and whose job is it to provide it“Social Media Information Overload”Photo by Joe Intersection ConsultingCC-BY-NC
  • This question of minimal requiremements ducks the issue of who is to enforce them.
  • Star graphic modified under CC-BY fromSsolbergjC.f.5 Stars of Linked Data (Berners-Lee):Available w/ open license, machine-readable, non-proprietary format, open standards, linked to provide context 5 Stars of Online Journals (Shotton):Peer Review, Open Access, Enriched Content, Available Datasets, Machine-readable metadataWhat about community?
  • We are afraid of what might happen if we choose to share too widely.An oft misused example is that of Geoffrey Chang, whose internal data analysis software was wrong and he was forced to retract work in Science. But he has continued to publish and been awarded a EUREKA grant, "for exceptionally innovative research projects that could have an extraordinarily significant impact on many areas of science" and, importantly, he had *not* shared his software – perhaps if he’d shared earlier the mistake could have been corrected. After all, his science method was fundamentally sound.“Scared”Photo by Capture QueenCC-BY
  • The best thing that a young researcher can do.Recent studies in The Atlantic suggest that getting an academic career is no easier now than 10 years ago. Getting a name for yourself is important, and sharing knowledge (and criticism) improves your profile (see work from HeatherPiwowar). Indeed it cannot be long before academic credit is gained for reviewing others work – signed reviews are the first step. Sharing is required by fundersAs well as enabling critique, discussion, repeatibility, reproducibility,reusabilityIt’s about reuse for improved efficiency and productivity
  • A metajournal which encourages the publication of information that encourages the reuse of software.A way of using the current tools and practices to make software betterrecognised.
  • The social stigma of sharing data is very different in the next generation. Can we ensure it doesn’t get driven out of them by the generation above?“Lady and the Tramp”Photo by Joe ShlabotnikCC-BY-NC-SA
  • Some further resources:
  • Data Sharing and Management Snafu in 3 Short Actsby Karen Hanson, Alisa Surkis and Karen Yacobucci of NYU Health Sciences LibrariesLicensed under CC-BY
  • Communicating trust, enabling criticism

    1. 1., enablingcriticismCommunicating Computational Research4 April 2013, Columbia University, New YorkNeil Chue Hong (@npch)ORCID: Institute Software Where indicated slides licensed under
    2. 2. The Software Sustainability Institute national facility for cultivating world-class research through software• Better software enables better research• Software reaches boundaries in its development cycle that prevent improvement, growth and adoption• Providing the expertise and services needed to negotiate to the next stage• Developing the policy and tools to support the community developing and using research software Supported by EPSRC Software Sustainability Institute Grant EP/H043160/1
    3. 3. “Episodis Critics”Photo by Camilay el ArteCC-BY-NC-SA If we cannot be critical, we cannot be researchers Software Sustainability Institute
    4. 4. systems are complex “Paint by Particle – aerosol movement” Photo by NASA Goddard Photo and Video Software Sustainability Institute CC-BY
    5. 5. Function … and software is always growing Library / Suite / Package AlgorithmProgram … Software Sustainability Institute
    6. 6. Water SwapReaction Coordinate A water-swap reaction coordinate for the calculation of absolute protein-ligand binding free energies Woods CJ, Malaisree M, Hannongbua S, Mulholland AJ J. Chem. Phys. (2011) vol. 134, pp. 054114 Software Sustainability Institute
    7. 7. How can I trust this code? = Symbol("lambda")S_sym = Symbol("S")S_scl = S_sym - 4*(S_sym-1)*(lam-0.5)**2 Opennessligand_bound_nrg_sym = Symbol("E_{ligand:bound}")ligand_bound_nrg = ligand_intraclj.components().total() + ligand_intraff.components().total() + bound_ligand_mobile.components().total(0) + Reviewbound_ligand_fixed_nrgswap_bound_nrg_sym = Symbol("E_{swap:bound}")swap_bound_nrg = ((S_scl) * swap_interclj.components().coulomb()) + Testsswap_interclj.components().lj() + bound_swap_fixed_nrg bound_swap_mobile.components().total(0) + bound_nrg_sym = Symbol("E_{bound}") Automationbound_nrg = bound_bound_nrg_sym + ((1-lam) * ligand_bound_nrg_sym) + (lam * swap_bound_nrg_sym)free_nrg_sym = Symbol("E_{free}")free_nrg = free_free_nrg_sym + (lam * ligand_free_nrg_sym) + ((1-lam) * swap_free_nrg_sym)total_nrg_sym = system.totalComponent()total_nrg = bound_nrg_sym + free_nrg_symsystem.setComponent(total_nrg_sym, total_nrg) Software Sustainability Institute
    8. 8. “Social Media Information Overload”Photo by Joe Intersection ConsultingCC-BY-NC Increase the amount of information on the software (without drowning in it) increase the trust Software Sustainability Institute
    9. 9. Academic Software Should Not Suck of C. Titus Brown Software Sustainability Institute
    10. 10. 5 Stars of Research Software • Community  There is a community infrastructure C • Open R O  Software has permissive license • Defined  Accurate metadata for the software E D • Extensible  Usable, modifiable for my purpose c.f. • Runnable 5 Stars of Linked Data (Berners-Lee)  I can access and run software 5 Stars of Online Journals (Shotton)“Golden Star”Originally bySsolbergjCC-BY Software Sustainability Institute
    11. 11. “Scared”Photo by Capture QueenCC-BY Isn’t sharing scary? Software Sustainability Institute
    12. 12. There is only one thing in life than beingtalked about, andthat is not beingtalked about. – Oscar Wilde Software Sustainability Institute
    13. 13. Journal of OpenResearch Software Software Sustainability Institute
    14. 14. “Lady and the Tramp”Photo by Joe ShlabotnikCC-BY-NC-SA The stigma of sharing will disappear Software Sustainability Institute
    15. 15. Communication’t just questions + answers… it’s about sharing knowledge Software Sustainability Institute
    16. 16. SSI Organisation• Community Engagement (Lead: Shoaib Sufi)  Fellowship Programme  Events and Roadshows• Consultancy (Lead: Steve Crouch)  Open Call for Projects / Funded Collaborations  Software Evaluation• Policy and Publicity (Lead: Simon Hettrick)  Guides and Case Studies  Best Practice and Policy• Training (Lead: Mike Jackson)  Software Carpentry (500+ students/year)  Software Surgeries• Collaboration between universities of Edinburgh, Manchester, Oxford and Southampton. 9.5 FTEs for 5 yrs supplemented by additional project funding. Software Sustainability Institute