Use of case pairs can potentially improve the efficiency and effectiveness of radiology residency

2,679 views

Published on

My masters thesis in education (2012), part of the 2 year Master in Health Professions Education program at Maastricht University. The focus of this experimental study was to determine whether compare and contrast practice with similar and contrasting case pairs improved the efficiency and effectiveness of radiology training. The results of this study were promising, and have resulted in continuing work to systematically build up and utilise case based repositories of radiology images in undergraduate and postgraduate training for competency, proficiency and mastery using the principles of deliberate practice These knowledge repositories currently contain over 3000 cases, which demonstrate the full spectrum of any given radiological/clinical presentation in the major diagnostic themes within neuroradiology, not just exemplar or outlier examples. We are using neuroradiology training as an case study to build a comprehensive knowledge repository to support training in other specialties within radiology, and as a template for a recently launched pilot project to build an institution wide repository to support clinical training in the different specialties at our academic medical centre. This knowledge repository will be linked to our undergraduate and residency curricular maps.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,679
On SlideShare
0
From Embeds
0
Number of Embeds
1,725
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Use of case pairs can potentially improve the efficiency and effectiveness of radiology residency

  1. 1. MHPE 2009 - 2011 Poh-Sun Goh, i665606 1 Master of Health Professions Education 2010 – 2012 Maastricht University Unit 11 Master Thesis Title: Use of case pairs can potentially improve the efficiency and effectiveness of radiology residency Date 12 April 2012 Name of Student – ID number Poh-Sun Goh – i665606 Supervisor: Ellen Kok Co-supervisor: Anique de Bruin Acknowledgements: Grateful thanks to Ellen Kok for much excellent advice and commitment to this project; Prof Jan van Dalen for very helpful feedback including advice on the experimental design during Unit 7 in Maastricht, and as part of Unit 10; and to fellow student Nynke de Jong for helpful feedback during Unit 10 assignments.
  2. 2. MHPE 2009 - 2011 Poh-Sun Goh, i665606 2 Table of Contents Summary Page 3 Preface Page 4 A. Introduction Page 5 Establishing significance Previous research and contributions Gap/problem/questions/prediction Research question B. Methods Page 8 Setting Participants Materials Apparatus Procedure Data Analysis and Statistical Analysis C. Results Page 11 D. Discussion Page 12 E. Conclusion Page 14 F. References Page 15 G. Appendices Page 18 Figures 1 to 10 (on sequential separate pages)
  3. 3. MHPE 2009 - 2011 Poh-Sun Goh, i665606 3 Summary In this Master Thesis, the topic to be investigated is a method to improve the efficiency and effectiveness of residency training in radiology in order to potentially shorten residency training while maintaining effectiveness, in order to increase the numbers of trained radiologists. The research question to be investigated is whether compare and contrast case review with pairs of cases is more time efficient and effective compared with traditional sequential case review with single cases in radiology problem solving. The research question was investigated by the use of a quantitative experimental design – an independent groups (or between groups) design. 10 residents participated with a within subjects design allowing the 10 residents to take part in both the control and experimental conditions with crossover between the groups. Resident performance with 30 CXRs and 30 CT head scans was compared between the experimental and control group, in a pretest, practice and posttest phase each with 10 different cases, with all 30 cases examples from a single key clinical diagnostic category for each type of examination. There was a consistent trend in time efficiency for the experimental group compared with the control group, with a medium effect size comparing pretest times (d = 0.48), and a relatively smaller effect size comparing times taken for the posttest phase (d = 0.33). A strong effect size, with statistically significant difference in performance was also found in the pretest phase of the study favoring the experimental group (d = 1.06). The control group comparing pretest with posttest also demonstrated a learning gain. Though a learning effect was not demonstrated for the experimental group, this group was found to have reached a high (good pass) level of performance in the pretest, with no significant change in performance for the posttest. The results of this Master Thesis project provide some support for the experimental hypothesis, that compare and contrast case review of pairs of cases, is more time efficient than, and at least as effective as sequential single case review, in interpreting radiology cases, and potentially in learning radiology for residency training. There is also some support for the effect of deliberate practice. The control group showing learning gain between the pretest and posttest phases is consistent with the expected effect of traditional radiology residency training, with one case at a time case presentation and review. The experimental group showing not only more efficient, but also more effective problem solving, with moderate and strong effect respectively, in the pretest phase, raises the interesting possibility that a much fewer number of cases might be required for training for any one diagnostic category if paired presentation and review is systematically used. Key words: Deliberate practice, paired and mixed practice, effectiveness, efficiency, and radiology residency.
  4. 4. MHPE 2009 - 2011 Poh-Sun Goh, i665606 4 Preface How do you learn to recognize the appearance of a dog, or a cat? A common method is to look at an example of a dog, then an example of cat, together with a description of the key features of each animal, followed by looking at an illustration of a dog and cat side by side. This is an example of sequential case review, followed by contrasting case review of a pair of cases. I would like to begin with a personal story, to explain my decision to undertake this topic of investigation for the Master Thesis. In 2008, at the AMEE conference in Prague, I had the opportunity to sit in the audience and listen to a plenary lecture by Professor Geoff Norman on the topic of teaching for transfer. In his presentation, Professor Norman reviewed the evidence for deliberate practice, by quoting the work of Anders Ericsson; and he also mentioned examples from his own work, including an early study on the role of compare and contrast practice in teaching Chest radiograph interpretation (Anders- Ericsson, 2004; Hatala, Brooks & Norman, 2003; Norman, 2008; Norman, Young & Brooks, 2007). The implications of this work resonated strongly with me, and I nearly leapt out of my chair with excitement. It was clear to me that many educators in radiology were unaware of these ideas, and certainly have not taken these ideas further into educational practice, and I sensed a great opportunity, not only as a budding researcher, but also as a teacher practitioner in radiology. Using similar and contrasting case examples to highlight a point is a common practice by radiology teachers whenever we try to explain difficult concepts to residents. This practice is also commonly used whenever a radiologist encounters a difficult case. We immediately search for a comparable example to confirm that a feature is a variation of normal, or the manifestation of an abnormality. We do this by either searching other areas on the current image, retrieving a previous radiograph or scan to compare, or search the literature for typical examples of a feature, and attempt a side-by-side comparison. It is clear from empirical practice that compare and contrast case review facilitates diagnosis. The question is whether this practice aids learning? Despite the evidence in the educational literature, the systematic use of a compare and contrast strategy, under conditions of deliberate practice, has not been implemented routinely for radiology teaching. This has probably been due to both a lack of awareness of the potential of these ideas, and to the limitation up till recently, of the lack of material for individual radiology teachers, encompassing sufficient examples of the whole range or spectrum of the manifestations of each disease, to use this teaching method as a core teaching strategy. With the advent of digital repositories, and the increasing practice of radiology teachers archiving teaching material in digital format, which can be shared, the limitation of availability of case material is now potentially no longer the case. The challenge is to validate these educational ideas, and to assemble or make accessible collections of case material. After AMEE 2008, I proceeded to first systematically build up and organize a collection to teaching material, which demonstrates the spectrum of disease presentation in two areas of radiology that I teach and practice in (the Chest radiograph and CT scan of the brain). I then completed several pilot studies with small groups of medical students, and radiology residents, in both an undergraduate, and postgraduate setting to test the hypothesis, that deliberate practice with paired cases (allowing compare and contrast review) is more effective and efficient than sequential practice with single cases in teaching and learning radiology (Goh, 2009, 2010 and 2011). The positive results of these empirical qualitative studies led to the proposal, and design of this formal quantitative study, as a Master Thesis project. I will now present the formal introduction to the Master thesis, which will elaborate on the background to this Master Thesis project.
  5. 5. MHPE 2009 - 2011 Poh-Sun Goh, i665606 5 A. Introduction Training of radiologists is a long and expensive process, taking an additional 6 to 8 years after medical school. The bulk of this time is spent building up clinical experience and diagnostic expertise by sequential exposure to case material with supervision and feedback. There is a pressing need to increase numbers of radiologists due to massive growth in the demand for radiological imaging, resulting in a worldwide shortage of trained radiologists (Royal College of Radiologists, 2012). Measures used to address this manpower shortage have been to increase the number of training places in radiology training programs, systematic use of innovative eLearning teaching methods, the development of more structured shorter training programs, and by reducing the period of specialist training to 48 months or 4 years (Specialist Accreditation Board, Singapore, 2012). Implementation of recommendations similar to the European Time Directive furthers reduces the time available for specialty training (Roedling, Robinson, Lough-White, & Miller, 2008). One key requirement in the quality specifications for these programs is the requirement for the exposure of the radiology resident to a minimum number of radiological examinations – 7,000 per year; with the implicit idea that clinical experience by case exposure is required for developing expertise (ACGME-I, 2010). The key assumption is that the more cases a resident is exposed to, the greater is their experience, which should lead to greater expertise. However, with the increasing numbers of residents, there is consequently reduced clinical case exposure per resident. The reason is that more residents share the same case volume in each training centre, with reduced opportunities for involvement by each resident in direct reporting or radiological interpretation, assuming that each clinical case is reviewed by one resident. The challenge in providing training experiences for these residents is improving the effectiveness and efficiency of the training and learning process, to mitigate the reduction in training duration and reduction in clinical exposure. Traditional teaching of radiology with exposure to clinical teaching material in mentored diagnostic sessions with an experienced teacher is usually opportunistic and haphazard in nature, as it is based on clinical case material that is encountered prospectively on a daily basis. To address this problem, systematic thematic teaching sessions and self study with published materials, film libraries and recently online repositories of case material have been used by radiology trainees to broaden their clinical experience and develop a comprehensive diagnostic ability. Clinical case review, and self-study is usually done with sequential case review, i.e. one case at a time. Due to reduced training time, any method to improve the effectiveness and efficiency of this process will be welcome by both residents, and teaching faculty. This is where the idea of the use of case pairs, to improve effectiveness and efficiency of training, can potentially be applied. The purpose of this Master Thesis project is to investigate a method of reducing training time, and making the time used in training more effective, by the systematic use of deliberate practice of sets of radiological cases; and use of compare and contrast practice with pairs of cases, rather than traditional sequential case presentation and review. This idea takes advantage of a common diagnostic strategy in day to day radiology problem solving to compare a suspected abnormal area with an equivalent normal area on the same radiological image, or another comparable scan.
  6. 6. MHPE 2009 - 2011 Poh-Sun Goh, i665606 6 Radiology Training and Review of Literature Training of radiologists has historically followed an apprentice progressive accumulation of experience model. Skill as a radiologist requires development of the ability to recognize and differentiate between visual representations of different diseases as displayed on plain radiographs and cross sectional imaging (Wood, 1999). Recognition of key abnormalities on radiology images is a core skill in clinical radiologists. Traditional teaching involves presentation of a concept followed by one or more examples, followed by the next concept and more examples (Avrahami et al., 1997). Educational literature contains evidence that suggests this is not the most effective or efficient way of teaching (Norman, 2005). The literature suggests that deliberate practice with paired and mixed examples is a more effective and efficient method for teaching and learning image recognition and discrimination (Anders-Ericsson, 2004; Hatala, Brooks & Norman, 2003; Norman, 2008; Norman, Young & Brooks, 2007). Deliberate practice refers to a consciously planned and executed program of repeated focused exercise of the skills required for expert performance accompanied by appropriate feedback (Anders Ericsson, Krampe, & Tesch-Romer, 1993). Paired and mixed practice involves the use of a series of similar and contrasting examples of imaging abnormalities, with an attempt to describe and reflect on similarities, and differences between the similar and contrasting examples (Norman, 2008). There have only been a few published research studies to date on the application of compare and contrast practice, with dermatology problems, ECG strips, and paediatric chest radiographs (Hatala, Brooks & Norman, 2003; Norman, 2008; Norman, Young & Brooks, 2007). It has been suggested that paired practice promotes the ability to recognize key diagnostic features, and the ability to elaborate on these features (Markman & Gentner, 1993). Mixed practice with contrasting cases may promote schema acquisition and analogous transfer (Gick & Patterson, 1992). Combination of paired and mixed practice, or use of similar and contrasting examples, helps in developing and refining categorical knowledge (Namy & Clepper, 2010). Radiology diagnostic practice requires the radiologist to see or detect, recognize and interpret (Krupinski, 2010; Nodine & Mello-Thomas, 2010). Two separate cognitive tasks are required as a clinical radiologist - image or pattern recognition, which has been described as non-analytical thinking or exemplar thinking; and the ability of list key distinguishing and discriminating features, which has been described as analytical thinking or rule-based thinking (Norman, 2005). Professionally, a radiologist has to state a diagnosis (image recognition), and then describe and justify the diagnosis, by listing key features as a radiology opinion or written report. Learning radiology can be therefore seen to involve a process of image or pattern recognition and discrimination (Koontz & Gunderman, 2008). There is a progressive improvement in performance with increasing experience in daily practice as a trainee and practicing radiologist involving exposure to clinical material (radiology images) with feedback (Helberg Engel, 2008). A combination of deliberate practice and increasing clinical experience accelerates the visual perception and recognition process, and this decreases the amount of purposeless visual searching of an image for diagnostic information (Kundel & Nodine, 1975). Building up schemas in long term memory accelerates image or pattern recognition, and facilitates the global impression and holistic recognition that is the predominant visual and thinking process of experienced radiologists (Nodine & Mello- Thomas, 2010, pp. 144-145). It is common practice to use a compare and contrast strategy in solving individual day-to-day clinical problems, and this strategy facilitates diagnosis (Berbaum, Franken & Smith, 1985). Use of pairs of
  7. 7. MHPE 2009 - 2011 Poh-Sun Goh, i665606 7 cases would be expected to result in solving individual radiology case problems in less time, and with equal or better accuracy when compared with single case at a time problem solving; or solving problems with greater accuracy, in the same amount of time. The availability of pairs of cases would be expected to therefore improve performance in radiological problem solving or diagnosis, and thereby potentially improve the efficiency and effectiveness of radiology training. Efficiency in the context of this study refers to the achievement of learning objectives in the least amount of time and/or achievement of significantly increased learning compared with traditional learning methods in the same amount of time, and effectiveness refers to the successful achievement of learning objectives (Rumble, 1997). For the purpose of this study, this refers to the ability to recognize an abnormality, and to discriminate between different clinical conditions on imaging and make an argument for one or other diagnosis by describing positive and significant negative findings. The gap in the educational and radiology literature is a systematic attempt to evaluate the potential of deliberate practice using compare and contrast paired cases compared with traditional sequential case review in radiology training. It is the intention of this Master Thesis project to attempt to investigate this topic. Research Hypothesis The research hypothesis of this Master Thesis study is that deliberate practice using pairs of similar and dissimilar contrasting examples of imaging (radiographic) abnormalities is a more efficient and effective method of radiology training compared with traditional sequential single case review, with exposure and review of concept followed by examples one case at a time. The research question to be investigated is whether compare and contrast case review with pairs of cases is more time efficient and effective compared with traditional sequential case review with single cases in radiology problem solving. Experimental Design The research question was investigated by the use of a quantitative experimental design – an independent groups (or between groups) design (Mulhern & Greer, 2011, pp. 109, and pp. 174 -175). Numerical data for diagnostic performance in interpreting CXR and CT scans (the dependent variable – effectiveness and efficiency of learning radiology measured by accuracy of diagnosis and time taken during deliberate practice sessions) are collected for two separate groups – the sequential practice group or control group, and the compare and contrast paired practice group, the experimental group (the independent variable).
  8. 8. MHPE 2009 - 2011 Poh-Sun Goh, i665606 8 B. Methods Setting of study The setting of this study is the Department of Diagnostic Radiology at the National University Hospital in Singapore. During any 6-month rotation period, approximately 30 residents in training will be present in our department. These residents undergo a national residency rotation system to broaden their clinical exposure amongst the three main teaching hospitals. Residents in the training program rotate every 6 months between the 3 teaching hospitals. We perform over 300,000 radiology examinations a year and provide subspecialty training within a structured residency program characterized by guided supervised practice by experienced radiologists. Participants The participants in this research project were 10 residents comprising a matched group (with equal length of residency and experience) of 6 senior residents (who are in their 3rd year of residency), and 4 junior residents (in their 1st year of residency). There were an equal number of male and female residents, with a similar background having all attended medical school and an additional 2 years of clinical practice experience after graduation including interpreting basic Chest radiographs and CT scans of the brain before entering radiology residency training, with an age range between 26 and 30 years of age. Their participation in the “experiment” took place over one month in January 2012 during their weekly scheduled academic learning sessions (scheduled personal learning time consisting of one afternoon per week off clinical duties), in our departmental library, using computers to present teaching material, with one computer terminal per resident. The involvement of the residents in the study was voluntary, and residents were free to discontinue their participation at any stage if they felt their participation did not meet their individual learning needs, which did not occur. The residents were informed that participation in the program provided additional opportunity for structured practice and learning, and can be considered as part of their learning activities, but they were not informed of the underlying structure or hypothesis of the research study, so that there would not be any performance expectation or bias. They were reassured that the results of the study were anonymised, and the residents were provided with individual feedback on their performance, which was given to each resident at the end of each session. The department chairman and postgraduate training committee approved this study. Materials The 30 CT and 30 CXR cases used were selected from an online repository containing over 1000 examples of Chest Radiographs (CXRs) and 1000 CT scans of the brain (CT brain) reflecting day-to-day practice in our academic medical center, and are of moderate difficulty. A conscious decision was made to select cases on the more difficult end of the spectrum for the posttest compared with easier cases in the pretest phase of the study. The learning objective of the CT cases was the diagnosis of acute ischaemic strokes and its mimics, and the CXR cases pneumothoraxes and free abdominal air beneath the diaphragm. CXRs and CT scans are 2 of the commonest radiological examinations, and the conditions chosen are 2 categories of the most important to accurately diagnose in clinical practice.
  9. 9. MHPE 2009 - 2011 Poh-Sun Goh, i665606 9 How to ensure credibility, reliability and validity To ensure credibility and replicability, the assessment items as well as all teaching and practice cases are available for peer review online. By making the case material content as well as the methodology available not only for local but external peer review, this allows the possibility of reproducing the research study in other sites, promoting reliability, replicability and credibility of this study (Bryman, 2009, pp. 31- 32). To further ensure reliability of the testing process, and validity of the case items, for difficulty level and appropriateness for radiology residents, these case items were tested on several cohorts of novice residents, and senior residents to provide a performance benchmark, during the prestudy preparation phase on all pretest and posttest cases in November and December 2011. As these residents rotate through the department at 6 monthly intervals, a different group of residents was used for the study phase in January 2012. The assessment instrument (pre-test and post-test items), and learning process (learning and practice case selection) was validated (measurement validity) by agreement with three members of the departmental postgraduate training committee (all senior radiology instructors), to enhance the content validity or direct validity of the experimental design (Schuwirth, & van der Vleuten, 2011). Apparatus A dell PC workstation with a single widescreen 22 inch LCD monitor (Dell U2211 22 inch Widescreen IPS Panel LCD Monitor), keyboard and infrared mouse was used for the experiment. Microsoft Powerpoint version 2010 software was used to present the cases, and the “rehearse Powerpoint show” function with onscreen digital timer which resets as each slide is advanced was used to display the time taken with each case or pair of comparison / contrasting cases. The experiment took place in our department library (Figure 1). Each resident entered answers for the cases in a hardcopy workbook with separate sheets for each case (Figure 2). Procedure Experimental Design The study followed a “classical” quantitative experimental design using 2 groups with “crossover” between the 2 groups. 10 residents participated with a within subjects design allowing the 10 residents to participate in both the control and experimental conditions (Bryman, A. pp.37) (Figure 3). In the first 2 weeks, a “control group” of 5 residents (group A) used sequential practice (Figure 4), and an “experimental group” of 5 residents (group B) used case pairs (Figure 5 and 6). In second half of the month, a “cross-over” was undertaken with a change in type of examination from CT scans of the head cases to CXR cases, with the difference where group A now used case pairs, and group B used sequential cases (Figure 7). The scheme for the crossover is outlined in Figure 10. The 10 residents were randomly assigned to the experimental and control group based on the alphabetical order of their first names, in order for each group to have 2 junior, and 3 senior residents. In addition to the crossover, the order of case items was randomized for each resident to minimize ordering effects. A total of 30 cases were presented, with a single radiological image for each case. Each test item was presented without further clinical information to avoid any additional contextual learning – recall effects. This is because the experimental hypothesis focused on visual perceptual learning – recall – and the deliberate practice of this.
  10. 10. MHPE 2009 - 2011 Poh-Sun Goh, i665606 10 The case presentation sequence of pretest (10 cases), learning phase (answers for pretest), practice phase (10 cases followed by answers) and posttest (10 cases) differed for the control and experimental group, with the control group undergoing sequential case presentation (Figure 4), while experimental group undergoes paired and mixed presentation with twin sets of radiological cases presented side by side, with a total of 5 pairs of similar and 5 pairs of contrasting cases in the learning and practice phases (Figure 5 and 6, 8 and 9). Compare and contrast paired case presentation of similar and contrasting cases offers pairs of cases (out of a total of 20 cases) similar to the total number of cases for sequential case presentation. Of the 20 cases in the learning and practice phase, there are 5 similar pairs (10 cases), and 5 contrasting pairs (10 cases). Each resident participated individually. For example, the steps taken by each participant are described in sequence as follows: Each session took place in the departmental library, one resident at a time, with departmental administrative staff in the adjoining room to maintain “privacy” of resident (there were no interruptions during the course of each training session). Each session was self-paced by the resident advancing the PowerPoint slides, with the time for each case recorded by the resident in the workbook. At the end of the session, the answers for the posttest phase were given to each resident during a feedback session with the principal investigator. For each pair of cases the time taken for the second case entered in workbook included the time taken for first case of the paired set (as the timer resets only when the slide is advanced). The actual time taken for every second case was obtained by subtracting the entered time from the time taken for the first case for each pair of cases by the investigator. In all phases, for both groups, residents were challenged for each case to write down a single diagnosis in the workbook, followed by the reasons for the diagnosis (by listing key discriminating and differentiating features) (Figure 2). For paired or mixed cases, a diagnosis had to be given for each one of the cases separately. The pre-test and post-test cases follow a similar format and were scored with 0.5 for a correct diagnosis, and 0.5 for correct key features, with maximum of 1 for each item, and a maximum score of 10 (for each pretest or posttest phase). The principal investigator using a marking template, blinded to the resident cohort during marking, performed the marking. Data Analysis and Statistical Analysis After collection of data, a comparison between the 2 groups was made by comparing averages of scores for each resident, and times for each resident (means) (Mulhern & Greer, 2011, pp. 56-59). For example, the difference in the mean of the time taken to complete the pretest or posttest phases, comparing the experimental group with the control group, is a measure of efficiency. The difference between the pretest scores, comparing the experimental with control groups, is a measure of diagnostic performance. Statistical analysis of the data used a student’s t-test comparing the time taken, and performance between the experimental and control groups. The t-test was initially performed using an online calculator (http://www.physics.csbsju.edu/stats/t-test_bulk_form.html) (Kirkman, 1996) with final calculations made with the SPSS 17 software package. The effect size was also calculated, by using an online calculator (http://www.uccs.edu/~faculty/lbecker/) (Becker, 1999). For interpreting Cohen's d, “an effect size of 0.2 to 0.3 might be considered a "small" effect, around 0.5 a "medium" effect and above 0.8, a "large" effect” (Cohen, 1988; Walker, 2008).
  11. 11. MHPE 2009 - 2011 Poh-Sun Goh, i665606 11 C. Results A strong effect, with significant difference in performance was found in the pretest phase of the study, with the experimental group (M = 7.5, SD = 0.85) performing better than the control group (M = 5.9, SD = 1.60), t (9) = 2.89, p = .019. The effect size comparing pretest performance was large with d = 1.06. The control group demonstrated a higher learning gain than the experimental group comparing pretest scores with posttest scores (with mean gain of M = 0.90, SD = 2.16). The experimental group did not show a learning effect with a negative gain (M = - 0.4, SD = 1.07), however the experimental group reached a relatively high (good pass) level of performance in the pretest with no significant deterioration in performance in the posttest. There was no significant difference in the learning gain between the 2 conditions with t (9) = 1.62, and p = 0.14. The experimental group (M = 780.9 seconds, SD = 333) also appeared to be more time efficient, taking less time than the control group (M = 939.5 seconds, SD = 329.8) comparing the mean of the time taken for the pretest for the CT and CXR cases. Though this difference was not statistically significant with t (9) = 1.11, p = .30, there was a medium effect size seen comparing total time taken with d = 0.48. For the practice phase, the experimental group also took a shorter time (M = 777.9 seconds, SD = 311.1) compared with control group (M = 905.2 seconds, SD = 413), t (9) = 0.66, p = 0.53. This consistent trend of the experimental group taking less time was again seen in the posttest (M = 928.9 seconds, SD = 364.5 seconds), compared with the control group (M = 1060.7 seconds, SD = 431.1 seconds). Though this difference was not statistically significant with t (9) = 0.79, p = .45, there was a small effect size also demonstrated with d = 0.33.
  12. 12. MHPE 2009 - 2011 Poh-Sun Goh, i665606 12 D. Discussion What might explain the results of the study? There is a consistent trend in time efficiency for the experimental group compared with the control group in the pretest, practice and posttest phases, with the greatest difference seen in the pretest phase with medium effect size. Having the comparison case appears to speed up decision-making, hence the more rapid completion of cases by the experimental compared with the control group (Gick & Patterson, 1992; Hatala, Brooks & Norman, 2003; Markman & Gentner, 1993; Namy & Clepper, 2010; Norman, 2008; Norman, Young & Brooks, 2007). In my professional practice, having a comparison case improves my own diagnostic certainty, and I can appreciate why this would reduce decision making and problem solving time. A strong effect size, with statistically significant difference in performance was found in the pretest phase of the study. The performance advantage for the experimental group, particularly in the pretest phase of the study, comparing the performance of the experimental with the control group suggests that compare and contrast review of pairs of cases facilitates diagnosis, and can potentially also have a learning effect (Berbaum, Franken & Smith, 1985). The control group comparing pretest with posttest also demonstrated a learning gain. Though a learning gain was not demonstrated for the experimental group, this group was found to have reached a high (good pass) level of performance in the pretest, with no significant change in performance for the posttest. It could be argued that “a ceiling effect” in performance (potentially learning) occurs with the CXR and CT cases relatively quickly, with the paired practice group achieving this quicker due to the availability of a comparison similar or contrasting case (Markman & Gentner, 1993; Namy & Clepper, 2010). This again shows the greater performance efficiency of the experimental group. What are the limitations of the study? There are several potential limitations to the generalizability of the results of this study, and potential confounding factors in this experiment. While the crossover design between the CT and CXR problems reduces the problem of potential unequal ability between the control and experimental groups, as the control group becomes the experimental group; and vice versa between the CT and CXR problem experimental observations; the use of CT and CXR problems which have not been matched for difficulty is a weakness in the experimental design. It may also be pointed out that the CT scan and CXR are different types of image, and are not equivalent types of examination. The purpose of this study however is to train residents to pick up abnormalities on radiological images, with a selected single relevant section of the CT scan used for each case, and also single CXR for each case. As the task accurately reflects actual diagnostic clinical practice, it could be argued that there is equivalency in the use of single CT brain images and single CXRs. Another limitation is the relatively small number of residents, and residents with variable clinical experience and clinical skill. This limitation was partly compensated for by choosing a spectrum of residents of differing experience, from novice to relatively experienced, who were randomly and equally matched. A further limitation to the generalizability of the results to broader radiology training is the limited range of conditions evaluated. Though the range of problems tested was limited, it can be argued that having both Chest radiograph and cross sectional CT cases, and showing variations and look alike examples of key clinical diagnoses provides enough variation of visual problem examples, and helps increase the
  13. 13. MHPE 2009 - 2011 Poh-Sun Goh, i665606 13 generalizability of this study to diagnostic problems in radiology. Having the residents perform multiple cases also increases the statistical power of the analysis (20 cases combining both the Pretest and Posttest phases per resident). While this study shows some support for the research question, and the research hypothesis grounded in educational theory, we have not evaluated how the training effect has occurred. While we have shown that the experimental group has a consistent trend with medium effect size being more time efficient while performing as well as the control group, more needs to be done to discover why this effect has occurred. Future qualitative and more detailed quantitative studies can be designed to explore this question. What are some of the implications of this study for future research, and on teaching practice? This study has shown some support for the use of deliberate practice in radiology training. There was great willingness by the residents to participate in the study, and to complete both limbs of the study, which was completely voluntary, due to the perceived learning value that the residents obtained from participation. Half of the residents went on to perform up to 4 more sessions with additional sets of CXR and CT cases under similar conditions. Ad hoc qualitative feedback from one resident is quite revealing about the perceived benefits of deliberate practice in residency training – “60 minutes of deliberate practice felt like, and has the teaching and learning value of one week of clinical experience, or even two weeks!!” There are many implications for radiology training, and presentation of teaching materials, if the trends in this paper are validated with larger groups of radiology residents, with a wider range of plain radiographic, and cross sectional imaging. The findings in this paper suggest that one method for improving learning or teaching radiology is for a resident to engage in deliberate practice with pairs of similar and contrasting cases, after initial presentation of exemplar case examples with descriptions of key features. This practice could reduce training time, to an extent dependent on residency program design, and resident motivation factors, while maintaining or improving learning and performance. The requirements for implementing the principles of deliberate practice, with review of a wide enough spectrum of case pairs, is a comprehensive collection of radiology cases, ideally in electronic format, covering the whole spectrum of presentation of each radiology disease condition. This collection would include look alike conditions. This would require at least 30 to 50 examples of each condition to allow the full spectrum of each disease to be covered, and facilitate deliberate practice with not only exemplar cases, but also more subtle cases at the tails of the normal distribution (Pusic, Pecaric & Boutis, 2011). This effort would require the coordinated effort of an institutional team of radiology educators, or a multi-institutional coordinated effort. The rewards of this initiative in reduced residency time, and potentially improved residency learning, would be potentially significant, and could justify the time, effort and expense of this process. The presentation of teaching materials, in both print, and online formats, can also be improved to take advantage of these ideas. For example, after presentation of a series of typical cases, and description of their key diagnostic features, authors can then present a series of similar and contrasting pairs of cases, to amplify the key diagnostic features of each disease entity. The use of online case repositories which provide a wider range of the spectrum of diagnostic findings of each disease entity can be used to supplement print formats, again with the material available for compare and contrast deliberate practice.
  14. 14. MHPE 2009 - 2011 Poh-Sun Goh, i665606 14 E. Conclusion The results of this Master Thesis project provide some support for the experimental hypothesis, that compare and contrast case review of pairs of cases, is more time efficient than, and at least as effective as sequential single case review, in interpreting radiology cases, and potentially in learning radiology for residency training. There is also some support for the effect of deliberate practice. The control group showing learning gain between the pretest and posttest phases is consistent with the expected effect of traditional radiology residency training, with one case at a time case presentation and review. The experimental group showing not only more efficient, but also more effective problem solving, with moderate and strong effect respectively, in the pretest phase, raises the interesting possibility that a much fewer number of cases might be required for training for any one diagnostic category if paired presentation and review is systematically used. There are implications for structuring radiology residency programs, if the trends in this paper are validated with larger groups of radiology residents, and a wider range of plain radiographic, and cross sectional imaging. In addition, the application of these ideas in continuing medical education and professional development, to maintain radiology expertise, and in teaching non-radiologists radiology interpretation are exciting possibilities, and need to be explored further.
  15. 15. MHPE 2009 - 2011 Poh-Sun Goh, i665606 15 F. References ACGME-I (2010). ACGME-International Advanced Specialty Program Requirements for Graduate Medical Education in Diagnostic Radiology. Retrieved 30 October 2011, from Accredication Council for Graduate Medical Education International website, http://www.acgme- i.org/web/requirements/specialtypr.html. Anders Ericsson, K. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79, S70-81. Anders Ericsson, K., Krampe, R.T., & Tesch-Romer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406. Avrahami, J., Kareev, Y. et al. (1997). Teaching by examples: implications for the process of category acquisition. Quarterly Journal of Experimental Psychology, 50A, 586–606. Becker, L.A. (1999). Effect size calculators. Retrieved 7 April 2012 from University of Colorado Colorado Springs website, http://www.uccs.edu/~faculty/lbecker/. Berbaum, K., Franken, A., & Smith, W.L. (1985). The effect of comparison films on resident interpretation of paediatric chest radiographs. Investigative Radiology, 20(2), 124-128. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (second ed.). Lawrence Erlbaum Associates. Goh, P.S., & Tam, K.C.J. (2009). Deliberate distributed practice with paired and mixed examples is an efficient and effective method to learn key abnormalities on the CXR. AMEE conference, Malaga. Goh, P.S., & Tam, K.C.J. (2009). Application of Educational Theory Regarding Teaching for Skill Transference for Chest Radiograph Interpretation in Undergraduate Education and Training. 6th APMEC conference, Singapore. Goh, P.S., & Tam, K.C.J. (2010). Applying deliberate practice to chest radiology teaching – further work with undergraduates and experience with postgraduates. 7th APMEC conference, Singapore. Goh, P.S. (2011). Deliberate Practice Using Paired and Mixed Examples of Imaging Abnormalities on the CXR is more effective and efficient for postgraduate training compared with traditional methods. 8th APMEC conference, Singapore. Gick, M. L., & Paterson, K. (1992). Do contrasting examples facilitate schema acquisition and analogical transfer. Canadian Journal of Psychology-Revue Canadienne De Psychologie, 46(4), 539-550. Hatala, R.M., Brooks, L.R., & Norman, G.R. (2003). Practice makes perfect: the critical role of mixed practice in the acquisition of ECG interpretation skills. Adv Health Sci Educ, 8, 17–26. Helberg Engel, P.J. (2008). Tacit knowledge and visual expertise in medical diagnostic reasoning: Implications for medical education. Med Teach, 30(7), 184-188. Kirkman, T.W. (1996) Statistics to Use. Retrieved 15 March 2012, from College of Saint Benedict and Saint John's University website, http://www.physics.csbsju.edu/stats/.
  16. 16. MHPE 2009 - 2011 Poh-Sun Goh, i665606 16 Koontz, N.A., Gunderman, R.B. (2008). Gestalt theory: Implications for radiology education. American Journal of Roentgenology, 190, 1156-1160. Krupinski, E. (2010). Perceptual factors in reading medical images. In Samei, E., & Krupinski, E (Ed.), The handbook of medical image perception and techniques. Cambridge: Cambridge University Press. Kundel, H.L., & Nodine, C.F. (1975). Interpreting chest radiographs without visual search. Radiology, 116, 527-532. Markman, A. B., & Gentner, D. (1993). Structural alignment during similarity comparisons. Cognitive Psychology, 25(4), 431-467. Mulhern, G., & Greer, B. (2011). Making Sense of Data and Statistics in Psychology. 2nd edition. Palgrave Macmillian, Great Britain. Namy, L. L., & Clepper, L.E. (2010). The differing roles of comparison and contrast in children’s categorization. Journal of Experimental Child Psychology, 107, 291–305. Nodine, C. & Mello-Thomas, M. (2010). The role of expertise in radiologic image interpretation. In Samei, E., & Krupinski, E (Ed.), The handbook of medical image perception and techniques. Cambridge: Cambridge University Press. Norman, G.R. (2005). Research in clinical reasoning: past history and current trends. Medical Education, 39, 418–427. Norman, G.R. (2005). From Theory to Application and Back Again: Implications of Research on Medical Expertise for Psychological Theory. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 59(1), 35-40. Norman, G.R. (2008). Making basic science relevant: Teaching for transfer. Plenary lecture. Association of Medical Educators Europe (AMEE) Annual Scientific Meeting, Prague. Norman, G.R., Young, M., Brooks, L. (2007). Non-analytical models of clinical reasoning: the role of experience. Medical Education, 41, 1140-1145. Pusic, M., Pecaric, M., & Boutis, K. (2011). How much practice is enough? Using leaning curves to assess the deliberate practice of radiograph interpretation. Academic Medicine, 86(6), 731-736. Roedling, S., Robinson, A., Lough-White, C., & Miller, R. (2008). Impact of a European working time directive-compliant working pattern on delivery of medical specialty teaching for senior house officers in a teaching hospital. Clin Med, 8, 116-7. Royal College of Radiologists (2012). Retrieved 9 November 2011, from Royal College of Radiologists website, http://www.rcr.ac.uk/content.aspx?PageID=164 Rumble, G. (1997). The costs and economics of open and distance learning. Cogan page Limited. Specialist Accreditation Board, Ministry of Health, Singapore (2012). Retrieved 1 November 2011, from Ministry of Health Singapore website, https://www.cms.moh.gov.sg/servlet/Satellite?c=Content_C&cid=1303267836998&pagename=SAB% 2FContent_C%2FSAB_ArticleDetailPage&rendermode=preview-dawn_aio-1119926675547.
  17. 17. MHPE 2009 - 2011 Poh-Sun Goh, i665606 17 Walker, I. (2008). Null hypothesis testing and effect sizes. Retrieved 7 April 2012, from University of Bath website, http://staff.bath.ac.uk/pssiw/stats2/page2/page14/page14.html. Wood, B.P. (1999). Visual expertise. Radiology, 211, 1-3.
  18. 18. MHPE 2009 - 2011 Poh-Sun Goh, i665606 18 G. Appendices Figure 1. Department Library (clockwise from top left – 2 views of computer workstations for resident use; library can be sectioned off for privacy; answer sheet booklets laid out for the “experiment”).
  19. 19. MHPE 2009 - 2011 Poh-Sun Goh, i665606 19 Figure 2: Example of answer sheet. Answers are handwritten on workbooks with individual sheets for each case divided into sections – Pretest, Learning and Practice Phase, and Posttest. The answer sheet for each case has sections for 1) a statement of the likely diagnosis; 2) concise justification for the diagnosis offered; 3) entry of time taken for each case.
  20. 20. MHPE 2009 - 2011 Poh-Sun Goh, i665606 20 Figure 3. Experimental methodology.
  21. 21. MHPE 2009 - 2011 Poh-Sun Goh, i665606 21 Figure 4. Example of sequential practice for CT scans of the brain.
  22. 22. MHPE 2009 - 2011 Poh-Sun Goh, i665606 22 Figure 5. Example of paired practice with dissimilar or contrasting cases for CT brain.
  23. 23. MHPE 2009 - 2011 Poh-Sun Goh, i665606 23 Figure 6. Example of paired practice with similar pairs for CT brain.
  24. 24. MHPE 2009 - 2011 Poh-Sun Goh, i665606 24 Figure 7. Illustration of paired practice for Chest Radiographs (CXRs).
  25. 25. MHPE 2009 - 2011 Poh-Sun Goh, i665606 25 Figure 8. Illustration of paired practice with dissimilar or contrasting case pairs for CXR.
  26. 26. MHPE 2009 - 2011 Poh-Sun Goh, i665606 26 Figure 9. Illustration of paired practice with similar case pairs for CXR.
  27. 27. MHPE 2009 - 2011 Poh-Sun Goh, i665606 27 Figure 10: The scheme for the experimental crossover.

×