Download to read offline
Uploaded byeAssessment in Practice Symposium
Caa2013 9 10-july2013
The document summarizes a pilot study analyzing 115 question items from an online export controls training. Various techniques were used to analyze the data, including exploratory data analysis, principle component analysis, and clustering analysis. The analysis identified questions with deficiencies or anomalies, indicating issues with question design. It was found that exploratory data analysis helped highlight both strong and weak question items, identifying areas for improvement. Recommendations include developing a visual dashboard to monitor question performance, applying validation rules to the authoring tool, and taking a more systematic approach to question design and alignment with learning objectives.
More Related Content
Caa2013 9 10-july2013
- 1. in compliance training and educationfor UK Export Controls Venkat Sastry Technology Enhanced Learning Laboratory (TELL) v.v.s.s.sastry@cranfield.a c.uk
- 2.
- 3.
- 4. Word cloud – structured Background Whatwas done? Data Analysis Results and Conclusions
- 5. Word cloud – structured Background Whatwas done? Data Analysis Results and Conclusions Strategic Export Controls focus through to 4 online supported by and Initial pilot and evaluation of current assessment Various techniques used including ing analysis and principle analysis and Questions with clear deficiencies or anomalies identified indicating issues with question item
- 6. What was done? • Whywas the pilot question item analysis undertaken? • Description of the data analysed - 115 question items - 9740 question responses - 974 users • Expected results – what did we hope to find
- 7. Data analysis • Exploratory dataanalysis consisted of: - 4 attributes observed: 1) Exposure count 2) % of correct answers 3) ratio of question stem length to distractor length 4) presence of a negative term - Principle Component Analysis - Clustering Analysis EDA = direction
- 8. Results – Observation of attributes Question IDs 69,71, 78, 86, 176, 84, 81,105, 106, 96, 104 100% 50%/50% Majority incorrect
- 9. Results – Clustering Analysis Perfect clustershape Items that may not belong in the cluster Cluster 1 – Low exposure counts Cluster 2 – High exposure counts. High % correct Cluster 3 – High ratio for question stem to distractor Cluster 5 – Negative present Cluster 6 – % correct is 100% Cluster 4 – High ratio for q stem to distractor. High % correct Question item
- 10. Results – Chernoff faces Key Each colourrepresents a cluster. Attributes mapping Exposure count = size % correct = elongation Ratio = forehead shape Negative = jaw shape
- 11. Conclusion s • Exploratory dataanalysis has acted as a guide and highlighted both healthy and weak question items • Identifies weaker question items • Gives direction for improvement • Item design and consistency issues identified (+ve and –ve) • Results broadly inline with expectations
- 12. Recommend ations • Development ofvisual dashboard based on the data analysis algorithms • Validation rules should be applied to the question item authoring tool • Aligning of learning objectives with question items at topic level to improve validity and reliability • Adoption of a systems approach e.g. extended ADDIE process model • Identify roles and activities required to manage current Level 1 design
- 13.
- 14. Contact details Peter Jolliffe -p.m.jolliffe@cranfield.ac.uk Piers MacLean - p.j.maclean@cranfield.ac.uk Venkat Sastry - v.v.s.s.sastry@cranfield.ac.uk SEC project website: http://www.strategicexportcontrols.org
Editor's Notes
- #2 Good morning Ladies and Gentlemen. My name is Venkat Sastry and I am from Cranfield University. Together with my colleagues Peter Jolliffe and Piers MacLean we have produced a paper called ‘UK Export Controls: a case-study in compliance training and education’ I am very pleased to be here at CAA to present our paper.
- #3 I thought I would start with a nice graphic. This is a word cloud of words from our paper. These graphics are also known as tag clouds or weighted lists. It makes a pretty and visually simulating summary of our paper, however, I thought I would simplify it in order to concentrate on the essence of the paper.
- #4 So we can see the key messages contain in our paper, however, even this focussed version is still not easy to digest. What is missing? Some form of structure.
- #5 So I’ve tried to align the focussed word cloud against the main sections of our paper as an aid to what I would like to talk about this morning at this conference. You will see how these themes can be grouped against the following structure; background to the project, what was undertaken in the paper, the data analysis techniques and results and conclusions. So let’s start to organise these themes.
- #6 Ok, so we now have a grouping of key themes against the areas I would like to talk about today. But we are still missing some detail to be able to make sense of these words. I will now run through each section in order to summarise the paper before looking, in depth, at elements of the paper. Background – SEC is the StrategicExportControls project run by Cranfield University and EGAD (Export Group for Aerospace and Defence) designed to provide learning and knowledge, with a UK focus, on the export of strategic items. In this context, strategic items can be defined as items relating to security, defence and foreign policy. Effectively the project delivery compliance training all the way through to post-graduate education, via 4 levels of structured courses. Levels 1 and 2 are assessed online by e-assessment. The project is supported by both industry and government (UK). What did we do in the paper? Well we undertook an initial pilot investigation into the relevance and validity of the assessment question items. Data analysis techniques So how was this analysis achieved. Various data analysis techniques were adopted including clustering analysis and principle component analysis. Results and Conclusions? So what was the outcome of the analysis that was undertaken for the paper? The results lead us to identify a number of significant weaknesses with the question item design. The results highlighted specific deficiencies within a question item or group of items and this has produced a number of conclusions and recommendations for e-assessments. Now that I have summarised the paper let’s look in more detail at certain aspects.
- #7 Why did we undertake the pilot exploratory data analysis? The project has experienced rapid organic growth. Processes for the analysis and review of question item results, in order to inform the design and development of the course content, were not agreed and implemented from the outset This, in combination, with the fact the that question items have been created by multiple authors, therefore leading to the possibility of potential inconsistency, resulted in the need to formally assess the question bank items. Also the project is about to scale up significantly with the user population increasing to over 17,000. Therefore, the importance of a formal analysis is much greater. Finally we need to test the robustness of the assessments in order to ensure they cover the required competencies and skills of the courses Description of the dataset 115 questions included in the analysis. 9740 question responses from 974 users. One thing to note is that although the question bank is small the exposure rate is significant What did we expect to find? Well with the earlier points in mind, we expected to find issues with some of the questions items. Multiple authors and indeed multiple editors may have lead to some of these issues. However, without the initial analysis to guide the ongoing development of the question items, the validity and reliability of the assessments could not be quantified.
- #8 Our exploratory data analysis consisted of: Observation of four attributes from each question item. These were; the exposure count, % of correct answers, ratio of question stem length to distractor length, and the presence of a negative term in either the question stem or a distractor. What is the purpose of the EDA? The purpose of the data analysis is to provide a guide as to the direction we should concentrate on to identify issues and for further development and analysis.
- #9 The graphic shows the whole date set sorted by the percentage correct by question item. As you can see a number of questions have been answered entirely correct (100%) Clearly these need to be examined in more detail. Also a group of questions have been answered 50% correct and 50% incorrect. Why is this? Could it be some form of ambiguity in the question stem or item distractors?
- #10 We started by looking at a formal 2 cluster component analysis. In a perfect world we would expect to see each cluster represented as a near rectangle around the 0.8 value i.e. all questions items within that cluster contain very similar attributes with a 80% confidence. In the 2 cluster analysis the form is almost a rectangle but for these areas. This indicated that further segmentation of the items was required so we repeated the process with 3 and then 4 clusters with the same result, indicating most clusters were required to represent all of the potential grouping of attributed for the data set. Finally we settle on 6 clusters as broadly these presented closer to the ideal rectangle shape and the 0.8 confidence value. However, within each cluster a number of anomalies presented that would indicate it would be worth examining these to find out more. To run you through the general observations against each cluster we found the following….
- #11 Chernoff faces allow the display of complex data to be visually represented in a manner that allows easy pattern and trend recognition. Chernoff faces were also used to make the clusters and map the item attributes to the various features of the faces. As you can see each item id is shown under the faces and they are ranked by percentage correct. Broadly we can say the following Each cluster of the 6 cluster analysis is represented by a different colour. In an ideal world we would see the majority of faces in the same colour as identical or very similar in shape and features. But as shown by the clustering analysis (where we did not have the percent rectangle shape) some of these faces may not belong to that particular section Describe the mapped attributes - Exposure count is shown by the size of the face, so a small sized face represents question items with a low exposure and a large face those with high exposure counts. - % correct is represented by the elongation of the face; question items with a high percentage correct are shown on a face with a long elongated look. - The ratio question stem to item distractor length is shown by the forehead shape. Question items with a larger question stems compared to the distractor length are shown with large foreheads. Finally the presence of a negative term in either the question stem or distractors is represented by the jaw shape of the face. Items with negation have a wide jaw. Those without have a narrow jaw. Highlight some particular examples and look at the actual data from the PDFs – could be 119 & 131 and or 56 and 61 Yellow faces map to cluster 1 which you might recall represents question items with a low exposure count. These might be questions that are under development or that have just recently been released into the question bank. If we look at question 62 we can interpret this item had a medium exposure count with a high percentage correct (shown by the elongation). The ratio of the question stem to the distractos was high, as shown by the wide forehead. The wide jaw indicates the presence of a negative term in either the distractors question stem Some interesting areas have been shown in question ids 119 and 131. The exposure count is fairly low as these questions are only exposed to users from a particular organisation. 60% of respondents answered correctly and the item contained a negative term. Another example is shown in question id 6: the face shows an item with a low exposure count, answered 100% correct and the small forehead shows the length of the distractors is longer than the question stem. Is it the fact the distractors are long and very clear that has resulted in the high percentage correct.
- #12 So in conclusion the exploratory data analysis has acted as a guide for further analysis and highlighted both strong and weaker areas We are now able to target our resource on the weaker items in order to improve the health of the question bank over time. This is particularly important as the project and question bank grows We also identified issues with item design and lack of consistency across items. These conclusions have resulted in a number of recommendations for the e-assessment, I will now summarise these.
- #13 The paper resulted in a number of recommendations. 1) It would be desirable to develop a web-based application to provide a visual dashboard, based on the data analysis algorithms used in the paper. This would allow question item performance to be monitored in real-time and therefore any remedial action could be taken quickly to improve the validity and health of the question bank. 2) If validation rules are applied to the question item authoring tool, subject matter experts and reviewers could improve item design and consistency as many of the common design mistakes could be designed out of the process. For example, authors can be notified if a particular part of the proposed question might lead to a weak or poor performing question item. 3) Learning objectives could be set at topic rather than course level and therefore allow greater alignment of question items to content. This would also allow the implementation of targeted feedback. 4) Adoption of a systems approach. The ADDIE ( Analysis, Design, Development, Implementation, and Evaluation) process model could be extended to capture