Software Sustainability in e-Research: Dying for a Change

Software Sustainability in e-Research: Dying for a Change?
Neil Chue Hong
[email_address]
e-Research Australasia 11 November 2010

Software underpins e-Research

UK e-Science Programme: Preparing the Ground
“ e-Science is about global collaboration in key areas of science, and the next generation of infrastructure that will enable it”
John Taylor, D-G RCUK
e-Science Centres
e-Science Pilot Projects

OMII: Sowing the first seeds 11 initial projects funded by Managed Programme Many projects flourished But some wilted and decayed OMII setup to harvest and maintain software output of UK e-Science Core Programme

OMII-UK: Cultivating and Nurturing
Emphasis on helping existing software grow
Extra gardeners brought in (Edinburgh and Manchester) with their own plant stock
Making the garden public through initiatives like Google Summer of Code and ENGAGE
Inviting specialists through the PALs scheme
Cultivate and sustain community software important to research

Software Sustainability Institute: pruning, staking, grafting
Working with research software users and developers
Helping review and refactor
Providing support and skills
Identifying areas of convergence
Producing strong, capable software able to live long and be successfully built on

SSI Pilot Projects
Pilot collaborators:
Fusion Energy
Climate Policy
Geospatial Linked Data
Crystal Structure
Brain Imaging
Scholarly Journals

No one starts writing software intending to abandon it completely

… and yet …
“ Sustainability is not an issue for researchers, we want others to take the software over subject to IP issues”
“ Sustainability is a big issue, we are producing complex tools we want to continue to use but it’s not clear how they’ll be sustained”
Quotes taken from ENGAGE interviews

Just the Nature of the problem?
Maintenance is not fun
Hacking is fun
Statistics courtesy of Greg Wilson, Software Carpentry, from Nature article Published online 13 October 2010 | Nature 467, 775-777 (2010) doi:10.1038/467775a

A quick comparison
Basic training for kitchen chef: 3-4 years
Head chef: 10 years
Basic training for s/w engineer: 3-4 years
Architect: 10 years
Photo by ZagatBuzz
Training in S/W Dev in UG Physics: 140 hours
Training in S/W Dev in UG Geography: 0 hours

Software development comes in stages
Bridging criteria: strength of team; strength of market; proximity of software to market
An idea to solve a problem Scaling to work for others Understand the functionality Allow others to participate Idea Prototype Research Idea Prototype Idea Idea Prototype Research Supported Product Idea Prototype Research Supported

e-Research is multidisciplinary, timescales don’t synchronise Cutting Edge Research Applied Research
Timescales vary:
ARIES (1989 – 1994)
Giant Magnetoresistance (1988 – 1999)
Frequency Hopping (1903/1942 – 1976)
Bayesian statistics (1763 – 1996)
Gap in Interest?

Social and Technical Infrastructure
Human infrastructure often an afterthought:
too late in the process to be effective
no conceptual framework
social science contributions ill-matched
need hybrid skills and truly interdisciplinary approaches
How you manage phase change is important

Software is no longer easy to define, let alone sustain

What do we sustain:
Workflow?
Software that runs workflow?
Software referenced by workflow?

Novel reuse of public sector data http://www.mysociety.org
What do we sustain:
Map?
Software that creates map?

Sustaining Function or Form
What do we sustain:
Function?
Form?

Repositories are not the solution… … only a mechanism

re⋅pos⋅i⋅tor⋅y
/noun/ [ri- poz -i-tawr-ee]
1. a receptacle or place where things are deposited, stored, or offered for sale.
2. a burial place; sepulchre.

Preserving something …

… is not the same as sustaining it

The Zombie Effect
Software not always fully alive when you reanimate it!
Complex set of dependencies
Significant Properties of Software
Purposes and benefits of software preservation
http://www.jisc.ac.uk/media/documents/programmes/preservation/significantpropertiesofsoftware-final.doc http://softwarepreservation.jiscinvolve.org/wp/

Open Source is not the solution… … only a mechanism

Open Source Software is Free...

Free as in Puppy...
Long term costs
Needs love and attention
May lose charm after growing up
Occasional clean-ups required
Many left abandoned by their owners
Inspired by Scott McNealy Photos of Great Pyrenees from Jen Schopf

Open Source is not just a licence Open Source Process is rarely completed Committer Policies Contributor Agreements Project Infrastructure Licence Resolution

Exploiting software for sustainability
Models
Grant Mosaic
Institutional support
Fully Costed Service
External Enterprise / Consultancy
Royalties and Fees
Donations
Advertising
T-shirt (spinoff merchandising)
Vehicles
University based
Spin out company
Consultancy and Customisation
Industrial knowledge transfer
Contracts
Licensing
Certification
Support services / training
Software as a Service
Software Foundation

Software sustainability is part of the process
Comparable to risk management
No one right “solution” but many examples of best practice and process
Plan from before the start if possible
But must be reviewed regularly
No longer considering timescales bounded by a project , but considering the product

Sustainability in Context Product Management Funding/ Effort Community Engagement Software Engineering Support / Contributions Market Development Software Sustainability

Communities are important… you are the solution!

Sustainable Communities
Cohesion and Identity: Creating a community
Tolerance and Diversity: Smart growth through collaboration
Efficient use of resources: Leveraging infrastructure
Adaptability to change: Governing sustainably

A: Analyse Customers Slide from Jim Austin

B: Build Community
Building intellectual access ramps to support incremental engagement – building capacity and capability
Individual
Group
Consortium
W/ industry
Community
Active
Teams change as project matures
http://www.castep.org/

C: Cultivate Contributors
Basics: Website, mailing list, code repository, issue resolution
Remove barriers to participation, increase efficiency
1993: First public release; 2 devs
1995: Code open sourced; 3 devs
1996: r-testers list set up
1997: lists split: r-announce, r-help, r-devel; public CVS; 11 devs
2000: CRAN split and mirror
2001: BioConductor
2003: Namespaces
2005: I8n, L8n
2007: R-Forge
Today: BioConductor (33 core devs), R-Forge (532 projects, 1562 devs), CRAN (1400+ packages)
http://cran.r-project.org/doc/html/interface98-paper/paper_2.html

Reuse, not reinvent Innovate, not replicate

The Software Maturity Curve Software proliferation Time Customisation Innovation Consolidation Portals Quantum chemistry Cloud Computing RDBMS Social Simulation Workflows Spatio- Temporal viz Molecular Dynamics Geospatial viz Digitised Doc Analysis Digital repositories

Enabling Innovation
Supporting emergent disciplines
Needs recognition of innovative software development as part of funding
Breaking down barriers
We cannot assume that the way people interact with resources will conform to expectations
e.g. researchers will use/store files outside of universities
Researchers will do whatever they can to get an edge – they will not always conform
We need to understand the relationship between e-Research and e-learning frameworks (VREs/VLEs?)

Supporting Consolidation
“ e-Science is an organic, emergent process requiring ongoing, coordinated investment from multiple funders and coordinated action by multiple research and infrastructure communities. It is both an enabler of research and an object of research ” – RCUK Review of e-Science
Bridging the expectation gaps between participants
Maintenance vs. research
Different timescales for “exciting” work
Well supported open platforms are the key in the age of the research mashup
Platforms to enable bottom-up innovation
Platforms to enable citizen participation
Competition/innovation built on top c.f. industry

Sustaining Customisation
“ The time constants for real transformative impact and significant competitive advantage is decades ” – RCUK Review of e-Science
Sustain software infrastructure in the long term
Differing models: through centres; within institutions; distributed
Need to change perceptions so that software is seen as valuable! (and not just invaluable)
Lower barriers to community growth and participation
Increase value of providing services
Virtually merge + map small amounts of effort / funding

Invest in people
People are the most important investment
Adaptability, ability to recognise transferable skills, not strict career paths
Software developers come from many backgrounds
If e-Science is multi-disciplinary, multi-institution, multi-scale then make it easier to recognise peoples efforts as they move
University structures do not make it easy
These people are key to effective e-Science as they bridge the gap between other participants

Case Study: NeISS
Evaluate impact of traffic control measures over next 5/10/15 years
Access baseline demographic data about the city
Execute simulation of traffic system and population
Visualise simulation outputs
Augment with new forms of data
Run dynamic models to assess future patterns (congestion, health, social inequality)

Case Study: NeISS Photo: NASA/GSFC, MODIS Rapid Response

Case Study: NeISS

One last question How do we get credit for sustaining software?

This is where YOU come in
Research credit is based on publication output
Data citations and credit for reuse are still not commonplace
Software credit is the next stage
Otherwise how can we persuade people to contribute back ?
YOU are the researchers, developers, providers and funders… YOU can make it happen!

In conclusion
Software Sustainability is not easy, but it is possible
Build it into the process and build the right teams
Three key things to remember:
ANALYSE CUSTOMERS
BUILD COMMUNITY
CULTIVATE CONTRIBUTORS
We need to give credit so people contribute back

The Software Sustainability Institute
A national facility for research software
Providing services for research software users and developers
Developing research community interactions and capacity
Promoting research software best practice and capability
Sustaining software by helping to negotiate the stages of the software maturity cycle

A National Facility for Research Software
Become our next collaborators!
Email: [email_address]
Blog: http://www.software.ac.uk/blog
Twitter: twitter.com/SoftwareSaved
SlideShare: slideshare.net/SoftwareSaved
YouTube: youtube.com/user/SoftwareSaved
Telephone: +44 (0) 131 650 5030
Pilot collaborators:
Fusion Energy
Climate Policy
Geospatial Linked Data
Crystal Structure
Brain Imaging
Scholarly Journals

Case Study: Crystal Energy Landscapes
Understanding polymorphism in drugs
E.g. Dosage profile
Chemists
Computational
Experimental
Developers
Domain
S/W Engineers
Integrators
Research Computing Services
Facilitator
http://www.youtube.com/watch?v=bkbRwOWmiwo

Software Sustainability: preservation vs sustainability Image courtesy of RGB Kew – not for reuse Image courtesy of London Permaculture under CC-by-nc-sa license Preservation? Sustainability?

Why are you considering software sustainability? Achieve legal compliance Create heritage value Enable continued access to data Encourage software reuse Manage systems and services Purpose

How are you going to choose the right approach? Preservation (techno-centric) Emulation (data-centric) Migration (functionality-centric) Transition (process-centric) Hibernation (knowledge-centric) Approach

Tradeoffs and Balancing Acts

Software Sustainability in e-Research: Dying for a Change

by Neil Chue Hong on Nov 11, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 72

Statistics

Software Sustainability in e-Research: Dying for a Change — Presentation Transcript

http://olaiainforarch.wordpress.com	70
http://drupals.humanities.manchester.ac.uk	2