Methods in Open ECB-Check


Published on

Published in: Business, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Methods in Open ECB-Check

  1. 1. Open ECBCheck Methods for Quality Development Rafael García Rodríguez University of Augsburg, 2010
  2. 2. Introduction <ul><li>Open ECBCheck label makes use of five central methods for quality evaluation and validation that all have distinct characteristics and potential advantages and disadvantages. </li></ul>Benchmarking Qualitative Weighting and Summation Benchlearning Community of Practice Quality Assessment Self-assessment Peer-review
  3. 3. Benchmarking <ul><li>Definition </li></ul><ul><li>Benchmarking is a systematic and continuous process to evaluate products, services or processes of an organization with the recognized market leaders (“best in class&quot;). </li></ul><ul><li>The target, in addition to improve himself, would be to get in long term market leadership or excellence. </li></ul>
  4. 4. Benchmarking <ul><li>B enchmarking is divided into four main stages with a concluding fifth phase maturity: </li></ul>Maturity Implementation Planning I ntegration A nalysis Identification of object, organisations and methods Identification of gaps and performance levels Communication of results and setting of targets Development, execution and check Leading position and integration of benchmarking Self-assessment
  5. 5. Benchmarking <ul><li>There are, however, a number of restrictions associated with benchmarking: </li></ul><ul><li>First, learning from benchmarking is mostly concerned with the past as the competitor has already achieved this level before. </li></ul><ul><li>There is a high uncertainty in identifying the “best in class” organisation or best practice. It's necessary a step by step approach. </li></ul><ul><li>Benchmarking only focuses on current best practices and is not a source for innovation and possible future best practice. </li></ul><ul><li>The best practice cases are not specific, they need to be generalisable to be transferable to one organisation. </li></ul>
  6. 6. Benchlearning <ul><li>Definition </li></ul><ul><li>Benchlearning is the process of learning from the &quot;best in class&quot; with the purpose of integrating these best practices in all organizational levels of the company. </li></ul>
  7. 7. Benchlearning <ul><li>Benchlearning is connected to benchmarking. </li></ul>These two processes search the way of how the best practices can be transferred to the own organisation and how skills and processes can be improved, and they distinguish benchaction as the actual implementation of all changes that have been set as targets. (Foundation) (Learning process) Benchlearning Benchmarking = =
  8. 8. S elf-assessment <ul><li>Definition </li></ul><ul><li>Self-assessment is a regular and systematic analysis of strengths and weaknesses of a company or organisation to determine one position, to identify areas for improvement and to transfer these insights into implementation. </li></ul><ul><li>The initiative is supposed to come from the organisation itself and the organisation that conducts the assessment is also responsible for the process. Usually, the self-assessment is conducted against a set of criteria. </li></ul>
  9. 9. S elf-assessment <ul><li>The E uropean F oundation F or Q uality M anagement (EFQM ) offers the EFQM Excellence Model as a reference point for a self-assessment . </li></ul>EFQM Excellence Model
  10. 10. S elf-assessment The EFQM manual discusses data collection methods that could be used for self-assessments that differ in results, time or resources required. Questionnaires Assessment workshop “ Award Simulation” Easy to use Don’t consume many resources Used as foundation for other methods Data collection Presentation and assessment Reach a consensus Full documentation for “European Quality Award” Results from organisations itself or outside
  11. 11. <ul><li>Definition </li></ul><ul><li>Peer-review is a external evaluation of an organisation or individual by experts that belong to a different organisation or may also be colleagues within the same organisation for a Quality Assurance. </li></ul><ul><li>P eer-review does n’ t focus on data collection but on a competent review of existing data to derive core issues, however, peer-reviewers may collect additional data by a peer-review visit or interviews. </li></ul>Peer-review
  12. 12. Peer-review <ul><li>Design review as the subject of interest: </li></ul><ul><li>R eview of the self-assessment report/results . </li></ul><ul><li>Review of the conclusions in the self-assessment report/ site visits and interviews with stakeholders . </li></ul><ul><li>Review of the subject of interest itself </li></ul>Basic design of the Peer-review process Self-evaluation Final review report External evaluation (+ on-site visit) Even so, there are numerous possible designs for peer-reviews
  13. 13. Peer-reviews Advantages Disadvantages <ul><li>SRCIVEN (1991, p. 255) </li></ul><ul><li>- Halo-effects </li></ul><ul><li>- Secret-contract bias </li></ul><ul><li>- Fear of retaliatory action </li></ul><ul><li>Gutknecht Gmeiner (2008, p. 23) </li></ul><ul><li>- Lack of qualification </li></ul><ul><li>- Lack of objectivity </li></ul><ul><li>(refers to the reviewers) </li></ul><ul><li>In a formative way: </li></ul><ul><li>- Mutual learning possibility </li></ul><ul><li>Relatively good cost-benefit ratio </li></ul><ul><li>More acceptable perception of colleagues or experts in the same field of study than external evaluators. </li></ul>
  14. 14. Qualitative Weighting and Summation <ul><li>Definition </li></ul><ul><li>Qualitative Weighting and Summation is a method for evaluation based on criteria catalogues. Is described in the context of the evaluation of learning management systems. This method is proposed as an alternative to the Numerical Weigh t ing and Summation method to avoid some disadvantages of this latter. </li></ul><ul><li>In Numerical Weigh t ing and Summation method: </li></ul><ul><li>1. Each criterion is weighted according to importance. </li></ul><ul><li>2. The evaluand is rated in all criteria </li></ul><ul><li>3. The products of all ratings with the corresponding weighing are summed up to </li></ul>one final score that indicates the relative rank of product, organisation or service)
  15. 15. Qualitative Weighting and Summation Qualitative Weighting and Summation Process <ul><li>All criteria are weighted based on non numeric characters (Essential / Very Valuable / Valuable / Marginally Valuable / Zero). All criteria that are rated with “Zero” are excluded. </li></ul><ul><li>Only criteria that have been weighted as essential are assessed. </li></ul><ul><li>All remaining products are rated only with help of the remaining criteria (“Very Valuable” to “Marginally Valuable”) </li></ul><ul><li>The ratings in the three different importance levels are then summarised resulting in three aggregated values for each product. </li></ul>
  16. 16. Qualitative Weighting and Summation <ul><li>Disadvantages </li></ul>Qualitative Weighting and Summation Numerical Weighting and Summation <ul><li>Method relatively complex . </li></ul><ul><li>It doesn ’ t offer a definite decision algorithm . </li></ul><ul><li>S ometimes needs to be applied several times in an iterative proces s to final conclusion. </li></ul><ul><li>Existence of minimum criteria. </li></ul><ul><li>Existence of interconnections between criteria. </li></ul><ul><li>The assumption of linear scales. </li></ul><ul><li>How to weigh and consider the relevance of the criteria. </li></ul><ul><li>The procedure of weighting is unreproducible. </li></ul>
  17. 17. ECBCheck . University Duisburg-Essen, Germany . <ul><li>Ehlers , Ulf-Daniel (2009). Selected Methods for Quality Development in Open </li></ul><ul><li>http :// </li></ul><ul><li> </li></ul><ul><li>More information about Open ECB-Check in: </li></ul>References