Software Sustainabilitypreserving the future of research software6 December 2010University of Tsukuba – University of Edinburgh Joint SymposiumNeil ChueHong, DirectorN.ChueHong@software.ac.uk
AgendaWhy software sustainability is important for the UKWhat the Software Sustainability Institute does, and its collaborative projectsThe research challenges for software sustainability for the future
3UK investment in e-ScienceOver 100 projects, worth more than £250 millionFacilities, services and community supporte-Science InstituteNational and Regional Centres, NCeSS, Environmental e-Science Centres, …NGS & Grid-PP, OMII-UK, Digital Curation Centre, …Many ongoing activities and services unperturbedEuropean Bioinformatics Unit, HPCx & HECToR, …Velocity magnitude flow field pertaining to a cerebral patient specific vascular tree, simulated with HemeLB, an example of urgent computingImage courtesy Peter Coveney, UCLSlide courtesy Malcolm Atkinson, UK e-Science Envoy
Just the Nature of the problem?No one starts writing software intending to abandon it completely and yet sustainability –  developing and maintaining over a long period – is a problem for scientific software Statistics courtesy of Greg Wilson, Software Carpentry, from Nature articlePublished online 13 October 2010 | Nature 467, 775-777 (2010) doi:10.1038/467775a
What is neededEffort, support and guidance to ensure that researchers can continue to use their chosen software as a cornerstone of their researchAnd beyond the lifetime of its original funding cycleThe lifetime of a software code is now greater than that of hardwareSoftware is the capital investment, hardware is the consumable?UK funding organisations want to ensure investment in software is not wastedSimilar efforts in the USA: NSF SI2 initiative and AustraliaSituation in Japan?
The Software Sustainability InstituteA national facility for research software Providing services for research software users and developersDeveloping research community interactions and capacityPromoting research software best practice and capabilitySustaining software by helping to negotiate the stages of the software maturity cycle
What the SSI bringsProvides specialist skills to drive the continued improvement and impact of research softwareDrawn from a large and varied pool of expertise
PALs programme funds researcher championsLed by University of Edinburgh with Universities of Manchester and SouthamptonDirector: Neil Chue Hong
Funded by EPSRC for 5 years, 9.5 FTE, £4.2 million, !st June 2010 – 31st May 2015
Builds on existing collaborations and experience from OMII-UK and EPCCWhat the SSI doesWork in collaboration with research groups in many subjects within the UK to improve key research software online materials (tutorials, guides)
consultative advice  (software evaluation ,  development process, community engagement, dissemination, workshops+surgeries)
collaborative partnerships (usability, quality, maintainability)Engagement with international community, doctoral training centres and funding programmes to change policy towards software sustainability
SSI GuidesSoftware developmentSoftware development: general best practice Developing maintainable softwareTesting your softwareRepositoriesChoosing a repository for your software project Migrating project resources: what to rememberCreating and managing SourceForge projectsRetrieving project resources from NeSCForgeOpen sourceAdopting an open-source licenceSupporting open-source software Community buildingRecruiting champions for your projectRecruiting student developers
Software Preservation
SSI WorkshopsCollaborations Workshop3-4 March 2011, EdinburghSoftware Sustainability SurgeriesRun on behalf of fundersCan also provide to universitiesSoftware Sustainability TrainingPotential Summer School for Doctoral Training CentresTraining on development practices for research teams
SSI Pilot ProjectsPilot collaborators:Fusion Energy
Climate Policy
GeospatialLinked Data
CrystalStructure
Brain Imaging
Scholarly JournalsCase Study: TavernaWorkbenchInitially funded through e-Science myGrid project (2001-2005)Directly funded through OMII-UK (2006-2010)Plus marketing, outreach, legal and networkingPlatform funding (2009-2014)caBIG subcontractEli Lilly development40,000+ downloads of Taverna 1.xTake up in other domains,e.g. astronomy
Case Study: NERC Data Grid SecurityProvides single sign-on to federated data infrastructure NDGS software now installed at major NERC data centres in the UKNow used across multiple projectsFilter based approach and OpenID work used by US Earth System Grid for access to CMIP5 archiveNCAR deployed worldwide to provide interoperable access control mechanism for next IPCC report in 2013METAFOR QUESTIONNAIRECOWS/NCEOContributions back to Python communityndg_saml, ndg_xacml, MyproxyClient
Case Study: Fusion ResearchGS2 used to study low-frequency turbulence in magnetized plasmaNo common visualisation across different groupsDeliver mutually agreeable framework that can be extended easily and can be maintained by the small fusion communityInitially looking at Paraview
Case Study: ECIASEnhanced Community Integrated Assessment SystemUnderstand impact of climate policyBring together disparate numerical models and datasets from community into common frameworkProvide public CLIMASCOPE portalFigure: Modelled sea level rise from 1765 to 2100 using the simple models in the MAGICC package.
Case Study: GeoTod-IITransform legacy geospatial data sources to a ‘linked-data’ representationExtend OGSA-DAI for transformation of both relational and file data sources to linked-data, and integrating geospatial data sources (e.g. ESRI ShapeFiles, Oracle Spatial)Initial demonstration of exposing INSPIRE hydrography datasets as linked-dataFigure: Data from the National Hydrography Dataset (NHD) accessed by Gaia 3 geospatial viewer
Case Study: NeISSEvaluate impact of traffic control measures over next 5/10/15 yearsAccess baseline demographic data about the cityExecute simulation of traffic  system and populationVisualise simulation outputsAugment with new forms of dataRun dynamic models to assess future patterns (congestion, health, social inequality)

Software Sustainability Institute

  • 1.
    Software Sustainabilitypreserving thefuture of research software6 December 2010University of Tsukuba – University of Edinburgh Joint SymposiumNeil ChueHong, DirectorN.ChueHong@software.ac.uk
  • 2.
    AgendaWhy software sustainabilityis important for the UKWhat the Software Sustainability Institute does, and its collaborative projectsThe research challenges for software sustainability for the future
  • 3.
    3UK investment ine-ScienceOver 100 projects, worth more than £250 millionFacilities, services and community supporte-Science InstituteNational and Regional Centres, NCeSS, Environmental e-Science Centres, …NGS & Grid-PP, OMII-UK, Digital Curation Centre, …Many ongoing activities and services unperturbedEuropean Bioinformatics Unit, HPCx & HECToR, …Velocity magnitude flow field pertaining to a cerebral patient specific vascular tree, simulated with HemeLB, an example of urgent computingImage courtesy Peter Coveney, UCLSlide courtesy Malcolm Atkinson, UK e-Science Envoy
  • 4.
    Just the Natureof the problem?No one starts writing software intending to abandon it completely and yet sustainability – developing and maintaining over a long period – is a problem for scientific software Statistics courtesy of Greg Wilson, Software Carpentry, from Nature articlePublished online 13 October 2010 | Nature 467, 775-777 (2010) doi:10.1038/467775a
  • 5.
    What is neededEffort,support and guidance to ensure that researchers can continue to use their chosen software as a cornerstone of their researchAnd beyond the lifetime of its original funding cycleThe lifetime of a software code is now greater than that of hardwareSoftware is the capital investment, hardware is the consumable?UK funding organisations want to ensure investment in software is not wastedSimilar efforts in the USA: NSF SI2 initiative and AustraliaSituation in Japan?
  • 6.
    The Software SustainabilityInstituteA national facility for research software Providing services for research software users and developersDeveloping research community interactions and capacityPromoting research software best practice and capabilitySustaining software by helping to negotiate the stages of the software maturity cycle
  • 7.
    What the SSIbringsProvides specialist skills to drive the continued improvement and impact of research softwareDrawn from a large and varied pool of expertise
  • 8.
    PALs programme fundsresearcher championsLed by University of Edinburgh with Universities of Manchester and SouthamptonDirector: Neil Chue Hong
  • 9.
    Funded by EPSRCfor 5 years, 9.5 FTE, £4.2 million, !st June 2010 – 31st May 2015
  • 10.
    Builds on existingcollaborations and experience from OMII-UK and EPCCWhat the SSI doesWork in collaboration with research groups in many subjects within the UK to improve key research software online materials (tutorials, guides)
  • 11.
    consultative advice (software evaluation , development process, community engagement, dissemination, workshops+surgeries)
  • 12.
    collaborative partnerships (usability,quality, maintainability)Engagement with international community, doctoral training centres and funding programmes to change policy towards software sustainability
  • 13.
    SSI GuidesSoftware developmentSoftwaredevelopment: general best practice Developing maintainable softwareTesting your softwareRepositoriesChoosing a repository for your software project Migrating project resources: what to rememberCreating and managing SourceForge projectsRetrieving project resources from NeSCForgeOpen sourceAdopting an open-source licenceSupporting open-source software Community buildingRecruiting champions for your projectRecruiting student developers
  • 14.
  • 15.
    SSI WorkshopsCollaborations Workshop3-4March 2011, EdinburghSoftware Sustainability SurgeriesRun on behalf of fundersCan also provide to universitiesSoftware Sustainability TrainingPotential Summer School for Doctoral Training CentresTraining on development practices for research teams
  • 16.
    SSI Pilot ProjectsPilotcollaborators:Fusion Energy
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
    Scholarly JournalsCase Study:TavernaWorkbenchInitially funded through e-Science myGrid project (2001-2005)Directly funded through OMII-UK (2006-2010)Plus marketing, outreach, legal and networkingPlatform funding (2009-2014)caBIG subcontractEli Lilly development40,000+ downloads of Taverna 1.xTake up in other domains,e.g. astronomy
  • 22.
    Case Study: NERCData Grid SecurityProvides single sign-on to federated data infrastructure NDGS software now installed at major NERC data centres in the UKNow used across multiple projectsFilter based approach and OpenID work used by US Earth System Grid for access to CMIP5 archiveNCAR deployed worldwide to provide interoperable access control mechanism for next IPCC report in 2013METAFOR QUESTIONNAIRECOWS/NCEOContributions back to Python communityndg_saml, ndg_xacml, MyproxyClient
  • 23.
    Case Study: FusionResearchGS2 used to study low-frequency turbulence in magnetized plasmaNo common visualisation across different groupsDeliver mutually agreeable framework that can be extended easily and can be maintained by the small fusion communityInitially looking at Paraview
  • 24.
    Case Study: ECIASEnhancedCommunity Integrated Assessment SystemUnderstand impact of climate policyBring together disparate numerical models and datasets from community into common frameworkProvide public CLIMASCOPE portalFigure: Modelled sea level rise from 1765 to 2100 using the simple models in the MAGICC package.
  • 25.
    Case Study: GeoTod-IITransformlegacy geospatial data sources to a ‘linked-data’ representationExtend OGSA-DAI for transformation of both relational and file data sources to linked-data, and integrating geospatial data sources (e.g. ESRI ShapeFiles, Oracle Spatial)Initial demonstration of exposing INSPIRE hydrography datasets as linked-dataFigure: Data from the National Hydrography Dataset (NHD) accessed by Gaia 3 geospatial viewer
  • 26.
    Case Study: NeISSEvaluateimpact of traffic control measures over next 5/10/15 yearsAccess baseline demographic data about the cityExecute simulation of traffic system and populationVisualise simulation outputsAugment with new forms of dataRun dynamic models to assess future patterns (congestion, health, social inequality)
  • 27.
  • 28.
  • 29.
    Case Study:NanoCMOS21The simulationParadigmnowVout2 [V]A 22 nm MOSFETIn production 2008Vout1 [V]A 4.2 nm MOSFETIn production 2023
  • 30.
    Case Study: CrystalEnergy LandscapesUnderstanding polymorphism in drugsE.g. Dosage profileChemistsComputationalExperimentalDevelopersDomainS/W EngineersIntegratorsResearch Computing ServicesFacilitatorhttp://www.youtube.com/watch?v=bkbRwOWmiwo
  • 31.
    Case Study: VIC+ RATMedia backbone tools for audio and video maintained by UCL since early 90sUsed as the basis for Access Grid, VRVSOMII-UK funding when other sources cutAllowed continued maintenance and bug fixesEnabled projects from Australia, Korea to contributeHowever difficulties in sustainingRapid changes in hardware / softwareToo low profileOther projects not contributing back
  • 32.
    Software Testing /Cloud ComputingHelping groups develop software tests appropriate to the situationLooking at Cloud Computing / Virtualisation to provide infrastructure for UK groupsWe would be very interested to collaborate on better frameworks for software testing, cloud computing and modelling of costs for CC 24
  • 33.
    Exploiting software forsustainabilityModelsGrant MosaicInstitutional supportFully Costed ServiceExternal Enterprise / ConsultancyRoyalties and FeesDonationsAdvertisingT-shirt (spinoff merchandising)VehiclesUniversity basedSpin out companyConsultancy and CustomisationIndustrial knowledge transferContractsLicensingCertificationSupport services / trainingSoftware as a ServiceSoftware FoundationMost common but what happens when PI retires?
  • 34.
    Sustainability in ContextSupport/ContributionsSoftwareSustainabilityCommunityEngagementSoftwareEngineeringProductManagementMarketDevelopmentFunding/Effort
  • 35.
    Software sustainability ispart of the processComparable to risk managementNo one right “solution” but many examples of best practice and processPlan from before the start if possibleBut must be reviewed regularlyNo longer considering timescales bounded by a project, but considering the product
  • 36.
    The Software MaturityCurvePortalsQuantumchemistryCloudComputingRDBMSSocialSimulationWorkflowsSpatio-TemporalvizMolecularDynamicsGeospatialvizDigitisedDoc AnalysisDigitalrepositoriesSoftware proliferationInnovationConsolidationCustomisationTime
  • 37.
    Enabling InnovationSupporting emergentdisciplinesNeeds recognition of innovative software development as part of funding Breaking down barriersWe cannot assume that the way people interact with resources will conform to expectationse.g. researchers will use/store files outside of universitiesResearchers will do whatever they can to get an edge – they will not always conform
  • 38.
    Supporting Consolidation“e-Science isan organic, emergent process requiring ongoing, coordinated investment from multiple funders and coordinated action by multiple research and infrastructure communities. It is both an enabler of research and an object of research” – RCUK Review of e-ScienceBridging the expectation gaps between participantsMaintenance vs. researchDifferent timescales for “exciting” workWell supported open platforms are the key in the age of the research mashupPlatforms to enable bottom-up innovationPlatforms to enable citizen participationCompetition/innovation built on top c.f. industry
  • 39.
    Sustaining Customisation“The timeconstants for real transformative impact and significant competitive advantage is decades” – RCUK Review of e-ScienceSustain software infrastructure in the long termDiffering models: through centres; within institutions; distributedNeed to change perceptions so that software is seen as valuable! (and not just invaluable)Lower barriers to community growth and participationIncrease value of providing servicesVirtually merge + map small amounts of effort / funding
  • 40.
    Invest in peoplePeopleare the most important investmentAdaptability, ability to recognise transferable skills, not strict career pathsSoftware developers come from many backgrounds If e-Science is multi-disciplinary, multi-institution, multi-scale then make it easier to recognise peoples efforts as they moveUniversity structures do not make it easyThese people are key to effective e-Science as they bridge the gap between other participants
  • 41.
    The credit questionHowdo we get credit for reusing, extending and sustaining software?Research credit is based on publication outputData citations and credit for reuse are still not commonplaceSoftware credit is the next stageOtherwise how can we persuade people to contribute back?This is a question which must be answered at an international level
  • 42.
    A National Facilityfor Research SoftwarePilot collaborators:Fusion Energy
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
    Scholarly JournalsBecome ournext collaborators!Email: info@software.ac.ukBlog: http://www.software.ac.uk/blogTwitter: twitter.com/SoftwareSavedSlideShare: slideshare.net/SoftwareSavedYouTube: youtube.com/user/SoftwareSavedTelephone: +44 (0) 131 650 5030

Editor's Notes

  • #5 Statistics from Greg WilsonAre academics software developers?Can research consortia manage production?Are timing constraints different?What is the role of the PI in software development management?Are the skills for software and research the same?
  • #7 Drawing on pool of specialists to drive the continued improvement and impact of research software developed by and for researchers
  • #13 JournalTOCS largest collection of TOCs from major publication
  • #17 Economic, hydrology, earth systems, coastal models etc.
  • #20 Update slide for surveymapper?
  • #21 Update slide for surveymapper?
  • #22 Update slide for surveymapper?
  • #25 Update slide for surveymapper?
  • #27 How does software sustainability fit within context of software engineering, community engagement, project management, fundingWhat are the external factors like change in effort, timelines and deadlines, licensing, step changes in product development
  • #33 Allowing people to move makes it easier to bridge gaps as you have a chance of creating common communication structures
  • #35 Become our next collaborator – email info@software.ac.uk