Terms and Definitions Document
Upcoming SlideShare
Loading in...5
×
 

Terms and Definitions Document

on

  • 384 views

 

Statistics

Views

Total Views
384
Views on SlideShare
384
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft Word

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Terms and Definitions Document Terms and Definitions Document Document Transcript

  • Terms and Definitions Term Definition Source Assessment is an ongoing process aimed at understanding and improving learning. It involves making expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and (Angelo, interpreting evidence to determine how well performance matches those expectations American and standards; and using the resulting information to document, explain, and improve Association of Assessment performance. When it is embedded effectively within larger institutional systems, Higher Education assessment can help us focus our collective attention, examine our assumptions, and Bulletin, November create a shared academic culture dedicated to assuring and improving the quality of 1995, p. 7) higher education. A measurement-based strategic management system, originated by Robert Kaplan and David Norton, which provides a method of aligning business activities to the strategy, and monitoring performance of strategic goals over time. There are four http://www.army.m Balanced linked perspectives in the framework: Customer, Internal Processes, Learning and il/armybtkc/rc/glos Scorecard Growth and Financial. The Balanced Scorecard acts as a strategic management sary.htm system, a measurement system and a communication tool. The current level at which an organization, process, or function is performing. A http://www.army.m Baseline company currently producing 25 units per week has a current baseline performance il/armybtkc/rc/glos Performance of 25 units per week. sary.htm A measurement or standard that serves as a point of reference by which process Benchmark performance is measured GAO, Glossary The comparison of similar processes across organizations and industries to identify best practices, set improvement targets, and measure progress. Benchmarking results http://www.army.m Benchmarking can serve as potential targets for measures. The process of comparing one set of il/armybtkc/rc/glos measurements to another. sary.htm Continuous Process Improvement is a strategic approach for developing a culture of Continuous continuous improvement in the areas of reliability, process cycle times, costs in terms http://www.army.m Process of less total resource consumption, quality, and productivity. il/armybtkc/rc/glos Improvement sary.htm Comparison of a program's outputs or outcomes with the costs (resources expended) to produce them. Cost-effectiveness Evaluation analysis assesses the cost of meeting Cost-Benefit and a single goal or objective, and can be used to identify the least costly alternative to meet that goal. Cost-benefit analysis aims to identify and compare all relevant costs GAO Glossary, Cost- and benefits, usually expressed in dollar terms. The two terms are often interchanged 1998 Effectiveness in evaluation discussions. 1
  • Evaluations are systematic, objective studies conducted periodically or on an ad hoc basis to assess how well a program is working. They help managers determine if timely adjustments are needed in program design to improve the rate, or quality, of achievement relative to the committed resources. • Impact Evaluation: The application of scientific research methods to estimate how much of the observed results, intended or not, are caused by program activities and how much might have been observed in the absence of the program. This form of evaluation is employed when external factors are known to influence the program's outcomes in order to isolate the program's contribution to achievement of its objectives. • Needs/Market Assessment Evaluation: An evaluation that assesses market baselines, customer needs, target markets, and how best to address these issues GAO, Definitions by the program in question. Findings help managers decide who constitutes the 1998, p. 3. OMB Evaluation program's key markets and clients and how to best serve the intended customers. Circular No. A-11 When performed at the beginning of a program, needs/market assessment 2003, Section evaluations also establish baselines against which to compare future progress. 200-2 • Outcome Evaluation: Measurement of the extent to which a program achieves its outcome-oriented objectives. It measures outputs and outcomes (including unintended effects) to judge program effectiveness but may also assess program process to understand how outcomes are produced. • Cost-Benefit or Cost-Effectiveness Evaluation: These evaluations show the relationship of benefits achieved by program activities and outputs to the costs incurred to achieve those benefits. The findings allow program managers to judge the cost-effectiveness of entire programs and component activities and, based on that, to decide whether to retain, revise, or eliminate the efforts in question. Cost-benefit evaluations may be performed using benefits to society or to individual customers of the program. Benefits usually consist of economic benefits expressed in dollars; however, benefits such as increased comfort, reduced environmental emissions, reduced waste products, and increased national security can also be used. In these cases, interpreting the results requires qualitative judgment. A particular characteristic used to measure outputs or outcomes; a quantifiable Indicator (also expression used to observe and track the status of a process. An indicator constitutes OMB, Circular Performance the observable evidence of accomplishments, changes made, or progress achieved. A-11 2003, Section Indicator) 200-2 The specific programs, activities, projects, or actions an organization will undertake http://www.army.m Initiatives in an effort to meet performance targets. il/armybtkc/rc/glos sary.htm These measures track resources used to drive organizational results. Typical input http://www.army.m Input Measures measures include staff time or financial resources il/armybtkc/rc/glos sary.htm Key A short list of metrics that a company's managers have identified as the most http://www.army.m Performance important variables reflecting mission success or organizational performance. il/armybtkc/rc/glos Indicators sary.htm Knowledge “spillover”: Category of research spillovers in which the profit or ATP and the US incentive produced is not sufficient for the private sector to undertake the work. Innovation System: A Methodology for Knowledge Idenfying Enabling Spillover R&D Spillover Networks, NIST GCR 06-985, 2006 A plausible and sensible diagram of the sequence of causes (resources, activities, and Logic Model outputs) that produce the effects (outcomes) sought by the program. GAO Glossary Observations collected over a period of time. The sample (instances or cases) may or Longitudinal may not be the same each time but the population remains constant. Longitudinal data are sometimes called "time series data." GAO Glossary Data 2
  • Often used interchangeably with measurements. However, it is helpful to separate these definitions. Metrics are the various parameters or ways of looking at a process that is to be measured. Metrics define what is to be measured. Some metrics are http://www.army.m Metric specialized, so they can't be directly benchmarked or interpreted outside a mission- il/armybtkc/rc/glos specific business unit. Other measures will be generic, and they can be aggregated sary.htm across business units, e.g. cycle time, customer satisfaction, and financial results Changes or benefits resulting from activities and outputs. Programs typically have multiple, sequential outcomes, sometimes called the program's outcome structure. First, there are "short term outcomes", those changes or benefits that are most closely associated with or "caused" by the program's outputs. Second, there are "intermediate http://www.army. Outcome outcomes," those changes that result from an application of the short-term outcomes. mil/armybtkc/rc/gl "Longer term outcomes" or program impacts, follow from the benefits accrued ossary.htm though the intermediate outcomes. This type of measure refers to an assessment of the results of a program activity compared to its intended purpose. These measures track the number of people served, http://www.army.m Outcome services provided, or units produced by a program or service. For example the il/armybtkc/rc/glos Measures number of inoculations provided. sary.htm The product, good, or service offered to a program's direct customers. http://www.army. Output mil/armybtkc/rc/gl ossary.htm A description of the level of activity or effort that will be produced or provided over a period of time or by a specified date, including a description of the characteristics and attributes (e.g., timeliness) established as standards in the course of conducting http://www.army.m Output Measure the activity or effort. A tactical or short-term quality or efficiency indicator for a il/armybtkc/rc/glos business process. sary.htm This is a targeted level of accomplishment expressed as a tangible and measurable objective against which actual achievement is compared. [GPRA] For example, a http://www.army.m Performance performance goal may be to process, on average, 60 procurement requisitions per il/armybtkc/rc/glos Goal day. sary.htm A particular value or characteristic used to measure output or outcome. [GPRA] This is a comparative performance metric used to answer the question, “How are we http://www.army.m Performance doing?” for a specific issue. The average number of requisitions processed per day is il/armybtkc/rc/glos Indicator an example of a performance indicator. sary.htm A standard used to evaluate and communicate performance against expected results. Performance measures are designed to gauge progress toward effective http://www.army.m Performance implementation of the organization’s strategy and track achievement of il/armybtkc/rc/glos Measurement organizational objectives, which are aligned with the strategy. sary.htm • Program may be any activity, project, function, or policy that has an identifiable purpose or set of objectives. GAO, Performance • Portfolio is a collection of programs that focus on a common strategic goal, Performance Measurement and objective, or science area. Measurement Evaluation, May and Evaluation • Organization an organization is a group of people intentionally organized to 2005, accomplish an overall, common goal or set of goals. GAO-05-739SP: • System includes links and interactions forming the skeleton of the R&D system: structured links between firms and between them and other organizations (universities, industry associations, federal laboratories); presence of scientific and technological infrastructure etc.). Performance A performance measurement process involves the collection, synthesis, analysis and http://www.army.m Measurement reporting of the accomplishments or outputs, as well as performance indicators that il/armybtkc/rc/glos Process affect work output and accomplishments. sary.htm 3
  • The Performance Reference Model (PRM) is a standardized framework to measure the performance of major IT investments and their contribution to program Performance performance. The PRM is intended to help produce enhanced information to improve http://www.army.m Reference daily and strategic decision-making, improve alignment and articulate contributions il/armybtkc/rc/glos Model and facilitate the identification of improvement opportunities. sary.htm A collection of projects. A single individual or organization can have multiple R&D Portfolio (GAO) portfolios. (DOE, R&D Portfolio Management) GAO Glossary A collection of investments that aims to maximize value while constraining risk. http://www.army.m Portfolio (US il/armybtkc/rc/glos Army) sary.htm The management of a selected grouping of IT investments using integrated strategic planning, integrated architectures, measures of performance, risk management techniques, transition plans, and portfolio investment strategies. The core activities associated with portfolio management are analysis, selection, control, and evaluation. 1. Defining goals and objectives — clearly articulate what the portfolio is expected to achieve. Questions to consider: What is the mission of the organization and how does IT support and achieve that mission? 2. Understanding, accepting, and making tradeoffs — determine what to invest in and how much to invest. Questions to consider: Which initiatives contribute the Portfolio most to the mission? http://www.army.m Management 3. Identifying, eliminating, minimizing, and diversifying risk — select a mix of il/armybtkc/rc/glos (US Army) investments that will avoid undue risk, will not exceed acceptable risk tolerance sary.htm levels, and will spread risks across projects and initiatives to minimize adverse impacts. Questions to consider: When and how do you terminate a project? At what point do you cancel a project that is still behind schedule and over budget? 4. Monitoring portfolio performance — understanding the progress a portfolio is making towards achieving of the goals and objectives of the organization. Question to consider: In whole, is the portfolio’s progress meeting the goals of the mission? 5. Achieving a desired objective —Which combination of investments best supports the desired outcome? Assessment of the extent to which a program is operating as intended. Process Process (or evaluation examines the efficiency and effectiveness of program implementation processes. It assesses program activities' conformance to program design, to (GAO, Definitions Implementation professional standards or customer expectations, and to statutory and regulatory 1998, p. 5) Process): requirements. Automated simulation of the processes that capture, manipulate, store, and distribute http://www.army.m Process data between a system and its environment and among components within a system. il/armybtkc/rc/glos Modeling sary.htm Automated simulation of the processes that capture, manipulate, store, and distribute http://www.army.m Process data between a system and its environment and among components within a system. il/armybtkc/rc/glos Modeling sary.htm "Program" refers to a collection of activities that are unified with respect to Program management structure and overall goal. GAO Glossary HHS regulations define research at 45 CFR 46.102(d) as follows: Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge. Activities which meet this definition constitute research for purposes of this policy, whether or not they are Research (HHS) conducted or supported under a program which is considered research for other HHS purposes. For example, some demonstration and service programs may include research activities. 4
  • NIH classifies its funding according to whether it is basic research, applied research, or development work, using definitions supplied by the federal Office of Management and Budget (OMB) for government-wide reporting purposes • Basic research is defined as "systematic study directed toward greater knowledge or understanding of the fundamental aspects of phenomena and of observable facts without specific applications towards processes or products in mind" (Office of Management and Budget, 1997). Office of Research (NIH) • Applied research is "systematic study to gain knowledge or understanding Management and necessary for determining the means by which a recognized and specific need Budget (OMB) may be met." • Development is the "systematic application of knowledge toward the production of useful material, devices, and systems or methods, including design, development, and improvement of prototypes and new processes to meet specific requirements.'' Research “spillover”: Excess of public benefits of research over the private benefits. ATP and the US Innovation System: A Methodology for Research Idenfying Enabling Spillover R&D Spillover Networks, NIST GCR 06-985, 2006 Human and financial inputs as well as other inputs required to support the program's Office of Resources activities. Management and Budget (OMB) The benefit of a program intervention that accrues to individuals or organizations that Office of Spillover are not direct recipients of the program's outputs. Management and Budget (OMB) Represents the broad priorities adopted by an organization in recognition of its http://www.army.m Strategy operating environment and in pursuit of its mission. All performance measures il/armybtkc/rc/glos should align with the organization’s strategy. sary.htm A term used to describe the overall defensibility of the evaluation as assessed by use of scientific practice, asking appropriate evaluation questions, documenting GAO, Designing Strength/ assumptions, making accurate measurements, and ruling out competing evidence of Evaluations 1991, Defensability causation. p. 16-18 Presented or formulated as a coherent body of ideas or principles, methodical in http://aolsvc.merria procedure or plan, marked by thoroughness and regularity. m-webster.aol.com/ Systematic dictionary/systemat ic The idea that people and the process of improvement are connected and related. Webster's Revised Systemic Continuous improvement requires a balance of both systematic actions and systemic Unabridged thinking. Dictionary Any assemblage or set of related science activities that form a complex unitary Science Systems whole. Systems The systems analysis process is an iterative one that cycles repeatedly through the http://www.answer Analysis Process following interrelated and somewhat indistinct phases: s.com/topic/system (1) problem statement, in which the system is defined in terms of its environment, sanalysis? goals, objectives, constraints, criteria, actors (decision makers, participants in the cat=technology system, impacted constituency), and other objects and their attributes; (2) alternative designs, in which solutions are identified; (3) mathematical formulation, in which a mathematical description of the system is developed, tested, and validated; (4) evaluation of alternatives, in which the mathematical model is used to evaluate and rank the possible alternative designs by means of the criteria; and (5) selection and implementation of the most preferred solution. The process includes feedback loops in which the outcomes of each phase are reconsidered based on the 5
  • analyses and outcomes of the other phases. The analysis of problems and processes in a logical manner, particularly through the Systems use of mathematical models and formulas and with the aid of computers and other http://www.answer Analysis Process data processing equipment. s.com/topic/system (US Military s-analysis? Definition) cat=technology A quantitative measurement of a performance metric that is to be achieved by a given time. Both the metric and the schedule need to be specified for targets. A stretch target is the same thing, but its quantitative value is much higher, demanding http://www.army.m Target breakthrough performance to achieve. [BSI] Targets make the results (metrics) il/armybtkc/rc/glos derived from the measurement process more meaningful and provide the sary.htm organization feedback regarding performance. • Salience relates to a user’s ability to communicate with the users whose decision Analysis of Global it seeks to inform and whether the information is relevant. Change Assessments: Three Essential • Credibility addresses the technical quality of the information, as perceived by Lessons Learned, the relevant scientific or other expert community. Properties of Committee on Assessment • Legitimacy concerns the fairness and impartiality of the assessment process, as National Global Process judged by its users and stakeholders. Change Assessment, National Research Council • Internal Validity: The extent to which the causes of an effect are established by an inquiry. Validity • External Validity: The extent to which a finding applies (or can be generalized) GAO Glossary to persons, objects, settings, or times other than those that were the subject of study. Generalizability is used interchangeably with "external validity." 6