NIC Linked Data: the OHIO project


Published on

This paper describes the NHS National Innovation Centre's Linked Data initiative. A discussion on the OHIO (Open Health Innovation Ontology) is also provided.

Published in: Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

NIC Linked Data: the OHIO project

  1. 1. NIC <br />Linked Data <br />The OHIO (Open Health Innovation Ontology) Project<br />Michael Wilkinson<br />NIC <br />Executive Summary<br />The emergence of the semantic web provides a significant opportunity to improve the speed and quality of decision-making in the area of health technology innovations. <br />This Business Case documents how the NIC will continue to develop its semantic web capability in order to deliver significant benefits to key stakeholder groups. This document starts with a detailed analysis of the current market conditions. It then outlines NIC progress to date as well as projected deliverables for June and October. This document then considers potential benefits to the NHS, and the approach to evaluation as a function of ongoing improvement. This document also presents a risk analysis, including a discussion on how such risk will be mitigated – thus ensuring that the potential benefits are likely to be realized.<br />The projected benefits arising from this investment, which will be measured, are expected to exceed the cost of this initial investment by a significant margin.<br />The NIC Linked Data Initiative will contribute to a number of key deliverables that the DH is expecting from the NIC, including contribution to improved: <br /><ul><li>decision-making by clinicians, managers and commissioners in the NHS,
  2. 2. support for healthcare industries,
  3. 3. joined-up working with other government departments.</li></ul>2. Market Analysis<br />The NIC is in a prime position to take the leading role in delivering compelling semantically-enhanced applications to deliver value in the health innovation space. The challenge for the NIC is to assure delivery and real benefits in a timely fashion. This paper describes, at a high level, the business case.<br /><ul><li>Needs addressed by the Semantic Web
  4. 4. Emerging Opportunities in Healthcare
  5. 5. NIC Progress to Date
  6. 6. Next Steps for the NIC, with specified time frame, and cost
  7. 7. The NIC Website (</li></ul>The Opportunity<br />The CBI (Feb, 2010) recently reported that the ‘NHS is data rich and information poor’, and noted that there is a pressing need to improve access to high-quality, utilisation-focused information in order to improve efficiency, effectiveness and quality in the NHS <br />The data-rich, information-poor challenge is not restricted to the NHS. As reported by The Economist (Feb, 2010), the world contains an unimaginably vast amount of digital information which is getting ever vaster ever more rapidly. This makes it possible to do many things that previously could not be done: spot business trends, prevent diseases, combat crime and so on. Managed well – and this is a pressing challenge - data can be used to unlock new sources of economic value, provide fresh insights into science and hold governments to account. <br />Governments around the world are freeing-up data to promote open innovation and open government. The US has taken the lead in increasing public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government through the creation of, which provides a limited portion of the rich variety of Federal datasets presently available. It has also invited people to actively participate in shaping the future of by suggesting additional datasets and site enhancements to provide seamless access and use of Federal data. This gives people a single place to go for all anonymous Government datasets to make it easier to reuse and innovate with Government data.<br />Informed from the US experience, the British Government recently launched This site seeks to give a way into the wealth of government data in a way that is easy to find, license and re-use by publishing data as RDF. Within months of its launch, the site contains nearly 3,000 freely available datasets from across government, 364 of which are from the Department of Health.<br />Making sense of all this data<br />Information has become super-abundant. <br />With so much information on the internet already, access to quality, context-specific information delivered to people when, where and how they want it is a challenge for all sectors.  To meet this challenge, the business of information management has steadily grown in recent years. Estimated to be worth more than $100 billion now, this sector is growing at almost 10% a year, roughly twice as fast as the software business as a whole.<br />A key enabler for this development is the Internet itself, which is currently at the cusp of a transformation, moving from Web2 - social networking sites - to Web3  - semantic applications <br />The Semantic web enables data to be linked to and expressed in meaningful ways, delivering a wide-range of benefits to end-users. The critical path to success is to understand end-user needs, then to develop applications that meet such needs. The global race to build semantic applications started in earnest at the WWW Conference in April 2009, where the full potential of linked data was demonstrated.   <br />In April 2009, the NIC joined this race.<br />The prize to those who succeed at building the first set of compelling semantic applications is significant, given how important speed and access to quality information is to clinical decision-making, investment, and performance in the global economy. <br /> <br />Having produced compelling applications to empower key NHS Stakeholder groups, the NIC will be well-positioned in its delivery on a number of key DH policy initiatives, including:<br /><ul><li>DH Innovation Procurement Plan (DH: 2009);
  8. 8. Office of Live Science: Delivering the Blueprint (DBIS & DH, January 2010), and 
  9. 9. Operational Efficiency Programme: Final Report (HM Treasury: April, 2009). 
  10. 10. Components of the Linked Data System
  11. 11. There are a number of key components of the Linked Data system that are important to ensuring that the system functions optimally. Described in more detail, below, these components include: ontology, triple-store, data, and applications.
  12. 12. Ontology
  13. 13. An ontology is is the philosophical study of the nature of being, existence or reality in general, as well as the basic categories of being and their relations. In Computer science an ontology is a formal representation of the knowledge by a set of concepts within a domain and the relationships between those concepts. It is used to reason about the properties of that domain, and may be used to describe the domain. An ontology provides a shared vocabulary, which can be used to model a domain – that is, the type of objects and/or concepts that exist, and their properties and relations.
  14. 14.
  15. 15. Since April 2009, the NIC has been building an Innovation bridging ontology, which is known as ‘OHIO’ (Open Health Innovation Ontology). By that we mean that the ontology covers the full life-cycle of an innovation, starting with need, then leading to commissioning, then R&D, and then market entry. This bridging ontology has been crafted through the creation of bespoke innovation ontology components, which have been enhanced with ontologies that are currently widely-used, such as FoF (Friend-of-a-Friend, as used in Linked-In and Facebook) and Dublin Core (used in document classifications).</li></ul>Triple-Store <br />In the Semantic Web. an ontology is often expressed computationally as either OWL (Web Ontology Language) or SKOS (Simple Knowledge Organization System). Data, which sits in a triple-store (which is a type of database for the semantic web), is rendered against OWL or SKOS. Following this rendering, the semantically-enhanced data can be disseminated via the tipple-store’s ‘sparkle-endpoint. This semantically-enhanced data can then be expressed via a wide range of applications, many of which are freely available via open-source, such as MIT’s SIMILE and Google. The end result of this semantic pipeline is that the end-user is much more likely to get information that is highly context-specific and meaningful than what is currently the case. The inter-relationships between these components is illustrated below.<br />Data<br />Note, from the illustration above, that in order for the system to work optimally so that it delivers benefits to end-users, there is a need for high-quality data, ontologies, and applications. Data may come from a wide-range of sources. In order to deliver a quick-win, the NIC has commissioned the curation of semantically enhanced data from: <br /><ul><li>Pera (general descriptive data),
  16. 16. TTP (European networks),
  17. 17. QinetiQ (future technologies) , and
  18. 18. Innovation Exchange (technology market assessments. </li></ul>Each supplier is working to their own unique strengths and are providing data that will contribute to meaningful information via the applications.<br />It should be noted that the curation of data can be expensive and time-consuming. As part of a global trend, data is becoming increasingly freely available from government as well as private-sector sources. In order to benefit from these developments, this initiative will link to high-value data sources that are already freely available from other trusted sources (e.g., PubMed, NHS Network, NHS Information Centre, to name a few). Such data will normally get to the NIC’s triple store via direct feeds. Once in the triple-store the data will be rendered via OWL and SKOS, as described above. <br />Another way that the NIC will be able to receive data is to have RDFa code (which contains the NIC ontology), which can be imbedded into other sites. The benefit of this is that the data that the NIC’s triple-store receives will have a high degree of specificity. To help propagate RDFa uptake on other sites, the NIC will provide tutorials and free downloads of the RDFa code. In addition to this, all applications that the NIC develops will include RDFa and management tracking code. What this means, practically, is that when an application is downloaded from the NIC site and uploaded onto another site, both the RDFa code and the management tracking code are also downloaded. Note that widgets will be designed to ensure informed consent from end-users – ie the user will explicitly grant or not grant use of RDFa code and management tracking code in the Terms and Conditions of use. The benefit of this will be that the NIC will be able to extend its access to both data and networks, and also produce evaluation metrics to inform the NIC business.<br />Applications<br />Recently, there have been big advances in displaying massive amounts of data to make them easily accessible. This is emerging as a vibrant and creative field, melding the skills of computer science, statistics, artistic design and storytelling. Gapminder, for example, enables people to load data into applications, which generate new understanding and innovative ways to communicate.<br />Example of Gapminder display of data<br />Having entered data and produced a graph, end-users are also able to share their graph with others. This entire offer is provided via a simple, easy-to-use interface, as illustrated below.<br />Example of Gapminder Data-Sharing Tool<br />One of the consequences of the Gapfinder initiative is that a number of people have started to produce open source libraries that can enact some of these kinds of visualisations. This provides an important opportunity for the NIC to become a Go-To place, where the NIC will:<br /><ul><li>build and host compelling semantically-enhanced applications
  19. 19. provide space for others to create and post their own applications
  20. 20. enable stakeholders to use the applications to inform their decision-making
  21. 21. enable stakeholders to share their data outputs with others. This would contribute to the NIC ontology, increase the amount of data in the NIC’s data hopper, and encourage a community of users.
  22. 22. provide added-value widgets, which end-users can upload onto their sites. These widgets will contain RDFa code and management tracking code, which the end-user can choose to enable or not. For the end-user, benefits include having their data (e.g. innovations) contribute to the larger data base and the applications arising from these (e.g. mapping of innovations). For the NIC, benefits include further reach to networks and datasets, and the ability to track and evaluate the impact of the NIC’s offering.</li></ul>Progress to Date<br />From April 2009, the NIC assembled a group of leading experts operating in the semantic web space. The NIC’s goal is to exploit the potential of the semantic web to support innovation development in the NHS. The NIC has:  <br /> <br />NIC Server on the Cloud<br />The idea of this was to establish a triple-store in the cloud, running 4Store, and then import the Virtuoso triples into it. Once there, the aim was to allow these triples to be queries using SPARQL. All of this was to be reproducible by running scripts to launch further servers.<br />The server is running and has been demoed.<br />There is one remaining deliverable which I'd like to do, which is to move all of the scripts into the new Linked Data Platform project. I'd hoped to do that this week, but it looks like it will have to wait until the end of next week, now.<br />Application Demonstrators<br />In order to test the Linked-Data system, the NIC commissioned the development of two demonstrators are able to prove data being obtained from the triple store, via the SPARQL end-point, could be displayed in a widget of some kind.<br />These tests have proved successful. Both of these applications, described in more detail, below, will become part of the widget platform/iStation.<br />Smaller Demo<br />In the end we went further and used this demo to (a) illustrate that we could convert the data into other formats, and (b) provide a second visualisation.<br />The first format that we were able to convert to was the Google Visualization data format, and by supporting that we opened up a whole library of widgets that can be used with the data. The second format that we can convert to is MIT's Exhibit format, which gave us access to a basic faceted browser and a timeline component.<br />Larger Demo<br />The larger demo is about linking a number of different data visualisations together.<br />For this we chose information that could be represented on a map, in a bar chart, and in a stacked chart. As the selection criteria is changed (data range, for example), all of the components on the page change at the same time.<br />This is more ambitious than the simple demo, because it involves binding components together. But it builds on the first demo, because the basic unit that we want to build the page from is a simple widget.<br />This means that once the framework is complete we can add any widgets we like to a page, and create quite complex layouts.<br />This work is pretty much complete. The framework part has been in place for about a week now, but what has held up the final stage is the lack of grouping support in 4Store, which makes it impossible to count things such as 'the number of innovations set to enter the market in 2020, grouped by the setting in which they would be used'.<br />For now we're working around this by adding that counting mechanism to the JavaScript code, which should be ready shortly.<br />NLP Crawler<br />The crawler work is underway. The aim in this project is to install the GATE framework on the same box as the triple-store, and to do it in such a way that it can be split out onto a different server if necessary (NLP processing can be quite resource-hungry, so we need to retain flexibility). Once GATE is installed then some of the GATE plug-ins [1] can be added and configured.<br />The first step is taking a little longer than anticipated in terms of the configuration scripts, but it should be resolved at the end of next week. Once that is done, the GATE modules I'm proposing to install are:<br />Reuters' OpenCalais;<br />KEA's SKOS processing. This would be configured to do two things:<br />look for MeSH terms;<br />look for terms from our own SKOS vocabularies (such as SHAs, scenario locations, and so on).<br />The reason for picking this collection is as follows:<br />OpenCalais provides a good spread of news-related terms, that are by their nature non-specialist, but give a good sweep. This means that we'll pick up a lot of basic information such as countries and cities, high-profile companies and people, universities, and so on;<br />KEA can process any SKOS document -- you probably remember that SKOS is a vocabulary language -- and use that to provide terms against which to run the NLP. This is ideal, because when our developer created the OWL ontologies for tmg, he adopted a strategy of using SKOS for any lists of items, such as SHAs, innovation status, types of innovation, and so on. All of these can be fed in to the NLP engine. Also, someone has produced a SKOS version of MeSH, which would be interesting to feed in.<br />Once the NLP framework is in place, our plan is to 'point' the crawler at the current NIC web-site (competitions, needs, forums, etc.), and one other location, such as New Scientist, PubMed, or something along those lines.<br />Adding more targets for crawling will be an ongoing task, of course, so ongoing management of the service will need to be in place after the build.<br />Data Curation<br />The NIC has commissioned the semantic curation of publically available files. Over 5,500 files have been loaded onto the NIC’s triple-store as of 20 April 2010. <br />June Deliverable<br />Linked Data<br /> <br />This the new public face of the Linked Data Platform (LDP), and would have the following features:<br /><ul><li>built with Drupal ( is also built with Drupal). This makes it very quick to add modules that provide features such as comments, forums, mailing-lists, ratings, questionnaires, and probably just about anything else one could think of. This would also make it very easy to manage user registration, access to different areas (even to different widgets), and so on;
  23. 23. instructions on how to embed RDFa. This work is being carried out separately, but will be incorporated into this area.
  24. 24. a widget directory, showing a list of widgets, with ratings, comments, categorisation, etc.;
  25. 25. a widget embedding page, where users can configure a widget for use on their site, by choosing a widget type along with a selection of the data that is available in the platform. The result will be a piece of mark-up that the user can then embed into their organisation's web-site, their personal blog, etc. This is a common technique, used in sites such as YouTube;
  26. 26. the widget delivery mechanism. Once the code for a widget is embedded in an external web-site, then every time a user accesses the web-site, a request will be made by the end-user's browser to the widget platform, to obtain the widget.</li></ul>RDFa Tagging of NIC Site<br />The Needs, Competition, Projects and Showcase sections of the NIC website will be given RDFa tags to enable data generated from these sections to be semantically enhanced and added to the triple-store. <br />October Deliverable<br /> <br /><ul><li>The NIC will focus squarely on building and sharing compelling semantically-enhanced applications to support innovation-related decision-making. The NICs work continues to develop at pace and is advancing very well.  Taking a rapid, user-centred design approach, the NIC has kicked-off the next phase of build. 
  27. 27. Network Collaboration
  28. 28.  In time for the DH Expo event in October, the NIC intends to work collaboratively with national NHS organisations to add-value to their user-communities.
  29. 29. NHS Networks. Build an application to aggregate need, as expressed by networks of clinicians and managers.
  30. 30. NHS Information Centre. Build an application to support commissioning of high-impact technologies.
  31. 31. NHS Scientists. Build a semantically-enhanced social networking platform to support NHS scientists.
  32. 32. NHS Choices, Deliver semantically-enhanced feeds of NIC Showcase content to this channel. Establish patient-centred needs suggestion form.
  33. 33. NHS Supply Chain. Semantically-enhance Scorecard tools, enabling data to contribute to the triple-store.</li></ul>New Needs Section<br />The NIC will work with the TSB and KTN to deliver the next generation of the NEEDS section on the NIC website. This section will benefit from multiple sources of data, including those listed above. These sources will be combined with the exiting resource, which targets individual clinical statements.<br /><ul><li>User-Engagement
  34. 34. To reach these goals, the NIC will deploy an Agile development approach in building its ontology and applications build. The aim is this approach is to release both ontologies and applications early and often, and then to engage end-users to identify what is most compelling and what could be done to improve the proposition.
  35. 35.
  36. 36. The most compelling applications will receive the most attention. Developers will seek to sprint such applications along the development pathway, delivering benefits to end-users as quickly as possible.
  37. 37. In order to reduce risk, save money, and benefit from the earlier work of others, the NIC will leverage word previously produced for the COI In this context, note the use of Bundles, Discussion, and Issue-Tracking and Source Control </li></ul>Bundles<br />At the bottom of the opening page you'll see that there is a list of what I called 'bundles'. A bundle is a collection of terms that tend to be used together, but they don't have to come from the same ontology or taxonomy.<br />For example, the vacancy bundle uses terms from FOAF and Dublin Core, and groups them together to allow the expression of the concept 'vacancy'.<br />Discussion<br />At the bottom of this page you'll see that each page can have a discussion about it:<br /><><br />This is a very useful way to get input on the ontologies as they are being developed.<br />Issue-Tracking and Source Control<br />Note that the argot-hub project has source control (where the ontologies can be stored) and an issue-tracking system (which allows people to flag up problems with the ontologies, if they spot them).<br />Moving forward, the NIC’s OWL files will move to the argot-hub source control system, and then a wiki page to be created for each area. This will make changes easier to track, and also make it easier to discuss each section of the ontology. The development of the Innovation Ontology is modular and bridging. To explain, the core innovation ontology will be enhanced with ontologies that already exist and in wide use (FoF, Dublin Core, MESH). Where ontologies do not exist and are needed, the NIC will commission new core elements, such as Need, Competitions, and RoI.<br /><ul><li>April 2011 Deliverable
  38. 38. From mid-Oct to end-Oct, the NIC will conduct a process and impact evaluation of the Linked Data Initiative. The evaluation strategy, which is discussed in some detail, below, is designed with the specific purpose of providing information to improve NIC performance against the NIC’s Goals and Objectives. </li></ul>3. Assessment of benefits<br />The National Innovation Centre (NIC) has assembled a group of leading experts operating in the semantic web space, with the goal of exploiting the potential of the semantic web to support innovation development in the NHS. <br />The aim of the initiative is to find and demonstrate possible solutions to improving how publically available data can be better used to inform decision-making for key stakeholder groups. To achieve this, the programme team is building and sharing semantically enhanced applications to support innovation-related decision-making. <br />By October 2010, the NIC intends to deliver the following:<br />a set of compelling semantically-enhanced innovation tools to support clinicians with commissioning responsibilities;<br />an evaluation of the efficiency, effectiveness and quality impacts of these tools; and<br />the establishment of a user-community, which both informs and is informed by these DH assets.<br />Aims and Objectives<br />The main aim is to deliver an evaluation of the efficiency, effectiveness and quality impacts of semantically-enhanced innovation tools, with the following objectives:<br />To undertake an assessment of the NIC Linked Data Initiative Programme, and its component work streams, to identify:<br />success of delivery against programme aims and objectives <br />level and success of stakeholder engagement with the programme<br />positive and negative elements of the programme and its component work streams<br />To capture learning from the programme and to disseminate this to the NIC team for discussion<br />To provide input into the development of the initiative against intelligence collected, to deliver increased impact<br />To assess and understand the sustainability challenges<br />To assess whether the programme has demonstrated an impact<br />To assess and understand the cost implications of the programme and identify whether it is considered to be value for money<br />Overall Approach<br />The evaluation will take a formative, before and after process and impact approach, linking closely with the application development, communication plan and the end user engagement phase.<br />Broadly, the aims of the process evaluation will be to:<br />Identify what is what is working well;<br />Identify what is not working well; and <br />Inform the desired end state.<br />The aims of the impact evaluation will be to:<br />Identify what the impact been and is likely to be; and<br />Identify potentially what the value of the impact is compared with not doing anything different/investing resources.<br />The formative approach will build in critical development cycles at key points during the Linked Data Initiative, in line with the agile application development approach (as shown in Figure 2), which will enable the programme team to review and reflect on the programme activities, to learn from the outcomes, and consequently to refocus activities, in order to better implement the initiative. Each development cycle will build on the other to:<br />facilitate the process of feedback to NIC about key findings, learning outcomes and challenges that have arisen at critical points in the evaluation;<br />provide commentary on the process of change throughout the lifecycle of the Linked Data Initiative;<br />provide interim recommendations about how the Linked Data Initiative can increase its impact; <br />provide an assessment of the prospective impacts of the Linked Data Initiative, considering whether these are likely to be sustained; and<br />drive the focus of the research questions, data collection and analysis methods.<br />Critical development cycles in the agile development of applications<br />793751270<br />379984080645180467080645-13652580645<br />The evaluation will use a logic model approach to provide the framework upon which to develop the research questions. The logic model will be useful for linking the outcomes with programme activities/processes and the theoretical assumptions/principles of the programme. It will provide a systematic and visual way of presenting and sharing understanding of the relationships among the resources operating the programme, the planned activities, and the anticipated changes or result. <br />It will also be useful for facilitating thinking, planning, and communicating about objectives and actual accomplishments, thus informing the NIC’s ISO9001 processes. In line with the critical development cycle approach, the logic model will be used at key points in the evaluation (e.g. post Expo), to validate the key assumptions with the programme and project teams about the overall programme and the component work streams’ inputs, processes, outputs, outcomes.<br />NIC Linked Data Initiative logic model<br />-3238506350<br />Process evaluation<br />The process evaluation will document and analyse the development and implementation of the NIC Linked Data Initiative, assessing whether this has been implemented as planned and whether the expected outputs have actually been produced. <br />It will consider the quality of the implementation of the programme and its component work streams, and if it has supported innovation-related decision-making in the short and long term. It will also assess the logic and implementation of the programme’s objectives, inputs and processes, and whether the programme has delivered its expected outputs.<br />The research questions for the process evaluation, which will be refined and agreed with the programme team, are summarised in the table below with the range of methods we propose employing in order to address these:<br />Evaluation focusResearch QuestionMethodologyProgramme design / objectivesWere the programme’s objectives realistic and achievable?Were new objectives added and why? How effectively was the programme designed to meet its objectives?Qualitative interviews with programme delivery team to review programme logic model and programme design (April 2010)Programme inputsWere the right people involved (programme team, designers, developers)?What changes occurred in leadership or personnel? What effect did these changes have? What costs were incurred? Did they exceed initial projections?Survey with programme delivery team to monitor the implementation of the programme (monthly)Attendance at Heartbeat meetings (monthly)Programme implementationHave the planned businesses processes been implemented?Do all elements of the project work coherently?What problems were encountered in implementing objectives? How were they resolved? Has delivery been in line with the programme plan? Have all planned activities been implemented? If not, what remains to be done? Were they accomplished on schedule? If objectives, plans, or timetables were revised, why was this necessary? Were the methodologies appropriate for developing possible solutions to improving how publically available data can be better used to inform decision-making for key stakeholder groups?How effectively have the project streams contributed to the delivery of the programme?Survey with programme delivery team to monitor the implementation of the programme (monthly)Review of programme / project plans in line with agreed route map (monthly)Attendance at Heartbeat meetings to identify what is / is not working well and what needs to be changed (monthly)Programme managementHas the programme management stream been effective?Has the NIC assured delivery of the programme?Has the communication work stream been effective in engaging and informing user communities?Qualitative interviews with the programme, quality assurance, and communications leads (monthly)Programme approachHow effective has the rapid, user-centred design approach been?How effective has the establishment of the user-community been?How effective has the 3 stage Agile approach been?Did the initiative identify users’ needs?Did the initiative effectively develop solutions to meeting need?Has the initiative been successful in sustaining user value/interest?Matrix input into user testing (monthly)Qualitative interviews with clinicians (August / September)Delivery of programme outputsHave outputs been delivered within agreed timescales (i.e. by Expo in October 2010)?Has the programme delivered the following outputs:To have started development of 9 application (minimum) at beta level and the establish of a user community for each via Open LabsRelease of 3 applications past beta, with an average of at least 100 regular users per day from an influential marketThree influential champions who are motivated to act on behalf of the initiativeTo have a business process which is explicit and reproducibleA business case for further investment (including ROI calculation in terms of numbers of adoptions of new technologies which are likely to happen earlier that would not have happened otherwise)Is the quality of outputs of a high standard?Review of programme / project plans in line with agreed route map (September / October)Assessment of the extent to which the programme outputs, as specified, have been delivered (September / October)<br />Impact Evaluation<br />The aims of the impact evaluation will be to identify what the impacts of the programme have been and are likely to be and to identify potentially what the value of the impacts are compared with not doing anything different/investing resources. It will assess whether the programme has been successful in meeting its stated aim of creating the “Go To” trusted source of technology innovation information which is routinely used in decision making in the NHS”.<br />The first step in the impact evaluation is mapping qualitatively what the outcomes and likely impacts might be. This will be achieved through discussions with the programme team, in line with the development of the programme’s business case.<br />The research questions for the impact evaluation, which will be refined and agreed with the programme team, are summarised in the table below with the range of methods we propose employing in order to address these:<br /><ul><li>Evaluation focusResearch questionMethodologyOutcomes: NICWhat are the benefits that are being delivered Vs a do nothing or other options (i.e. what value is the investment creating?)Has the NIC been successful in developing tools that will find and link relevant data to give more meaningful results?Has the NIC exploited the potential of the semantic web to support innovation development in the NHS?Have the semantically-enhanced applications delivered value in the health innovation space?Has the initiative accelerated adoption of vfm and low risk technologies?Has the take up/usage been what was proposed?Qualitative interviews with programme delivery team to review programme logic model (September /October)Programme team assessment of programme outcomes (Final Heartbeat meeting)Website hits using Google Analytics (ongoing)Outcomes: user communityDid the initiative meet users’ needs?Has the initiative helped innovators and clinicians find the answers to all their questions about innovation related to people, places and technologies in their respective fields?Has it helped clinicians discover more meaningful information about Healthcare technology innovation when using the web?Has the programme assisted with behaviour change – are clinicians doing anything differently? Has this initiative been successful in supporting innovation-related decision-making? Has it increased rationality in decision making processes?Has the initiative been successful in supporting clinicians with commissioning responsibilities?Qualitative interviews with clinicians (August / September)ImpactWhat are the likely impacts of the programme?Will the initiative improve performance in healthcare?Is the programme likely to have the following impacts?Improved outcomesMore delivery of care at homeImproved patient experienceImproved staff satisfactionReduced hospital utilisationImproved use of resourcesProgramme team assessment of potential programme impacts (Final Heartbeat meeting)</li></ul> <br />Methodology<br />The evaluation will employ a variety of methodologies, both quantitative and qualitative, as outlined below:<br />Interviews with the programme team<br />The qualitative interviews with members of the internal delivery team will form an integral component of both formative and summative elements of the evaluation. The interviews will be used to monitor objectively how well the programme is being implemented and will therefore be conducted by a member of the Matrix team who has not been involved in the initiative to date.<br />We will conduct a set of in-depth qualitative interviews with all members of the delivery team in April, to review the programme’s logic model and the design of the programme. These will be either be face-to-face or telephone, whichever is most convenient for participants.<br />We will then conduct a set of in-depth qualitative interviews in September / October to assess the outcomes of the programme, the benefits it has delivered, the success of the tools that have been developed, and how effective the NIC has been in exploiting the potential of the semantic web to support innovation development in the NHS.<br />Additionally we will conduct monthly telephone interviews with the programme, quality assurance and communications lead to monitor whether the programme management stream been effective, whether the NIC has assured delivery of the programme, and whether the communication work stream been effective in engaging and informing user communities.<br />Survey with programme team<br />In addition to the qualitative interviews we propose administering an online survey to the programme team on a monthly basis to monitor the implementation of the programme.<br />The findings from the survey will be analysed for presentation at the monthly Heartbeat meetings, to assess the extent to which implementation has occurred as planned, problems that have been encountered in implementing objectives, whether delivery is in line with the programme plan, and how effective each component work stream has been in contributing to the delivery of the entire programme.<br />Interviews with the user community<br />The implementation team will have contact with the clinicians on a weekly basis. Therefore to reduce potential burden on the user community, we would propose suggesting a set of additional questions for the delivery team to ask to monitor the effectiveness of the agile approach and the stakeholder engagement stream. The findings from these interviews will feed into the formative element of the project to address how effective the user-centred design approach has been, whether the initiative has identified their needs and developed solutions to meet their needs, and whether the initiative has been successful in sustaining their interest and value.<br />As part of the outcome evaluation, in August / September we would propose conducting a set of interviews with a sample of the user community, either by telephone or face-to-face, to assess the extent to which the initiative has met their needs, whether the programme has assisted with behaviour change, and whether it has assisted with the decision-making process.<br />Website hits<br />We will use Google Analytics to track visitors to the websites. Google Analytics is the enterprise-class web analytics solution that gives rich insights into website traffic and marketing effectiveness: This will enable us to find out how many people visit, how long they stay on the website, which pages they visit, where they came from (Google search, a link from another site etc), which country they are based in, and numbers of new and returning visitors.<br />-161703557767Timeline<br />Cost / Benefits Assessment<br />This will clearly identify the cost benefit against the investment that is being made. This should consist of a simplified presentation of the financial cost/benefit analysis - presented in tables or graphs;This should include an assessment of:Investment value – Revenue and CapitalCost Saving first 12 months (£)Cost Saving after first year (£)Depreciation costsTotal cost (£)Cash outlay (£)Additionally, an assessment of unquantified Benefits and Costs, and Strategic Contribution should be considered:Try very hard to quantify all costs and benefitsAll direct costs should be includedIndirect costs should be includedIf a non-financial benefit is significant, then define itMake the impact of the benefit tangible – describe all likely effects and implication<br />Dissemination and reporting<br />A monthly evaluation report will be generated by the Programme Team, and published on PMPT. These reports will use a standard format and will present the key findings, learning outcomes and challenges that have arisen at critical points in the programme, provide commentary on the process of change throughout the lifecycle of the programme, and provide interim recommendations about how the Linked Data Initiative can increase its impact. These will be presented using Power Point and will be uploaded onto the Linked Data website for consideration by the programme team.<br />The final phase of the initiative will be completed on completion of the Expo in October. At this time, a report will be produced to document process and impact evaluations. This will identify what the impact been and is likely to be; and what the value of the impact is compared with not doing anything different/investing resources.<br />