Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Defining criteria for measuring impact of discipline specific information skills training for PhD students. Ryan

43 views

Published on

Presented at LILAC 2010

Published in: Education
  • Be the first to comment

  • Be the first to like this

Defining criteria for measuring impact of discipline specific information skills training for PhD students. Ryan

  1. 1. James Hardiman Library Defining criteria for measuring the impact of discipline specific information skills training for Humanities & Social Science PhD students A postcard from the journeyA postcard from the journey
  2. 2. Gwen Ryan James Hardiman Library What is Impact? Success criteria .. are based on positive changes in people Markless & Streatfield (2006) Changes can be Affective, ie. increased confidence Behavioural, ie increased use of library resources Knowledge-based, ie. knowing about key sources for relevant information Competence-based, ie. using better search techniques
  3. 3. Gwen Ryan James Hardiman Library Methods of ascertaining impact? • Setting measurable impact criteria • Pre & post skills audits/tests • Setting a graded assignment reflecting the learning objectives • Conducting a post-event course evaluation • Comparing intervention outcomes for sample group vs. control group/wider population
  4. 4. Gwen Ryan James Hardiman Library What I did • Module was assignment based – assignment was directly relevant to their own research and would position them where they should be after the first semester • Didn’t elicit feedback after each individual class • Post module course evaluation when semester had finished to ascertain whether the module had any lasting impact
  5. 5. Gwen Ryan James Hardiman Library Why did I think anyone else would be interested? • I started from close to scratch • I have made mistakes that I would like others to learn from • I have also done some things right – A variation closely modelled on this module currently under consideration by Dean of Graduate Studies to be offered to PhD students across all Colleges from next year
  6. 6. Gwen Ryan James Hardiman Library Background to the module • Commenced my post in November 2008 • Inherited a credit-based Library Information & Services module • Joined the existing STM research support librarian who was already offering a comprehensive workshop series tailored to the needs of STM researchers • National graduate information skills online tutorial being piloted locally • Two previous Researchers’ Needs Surveys – feedback on training needs available
  7. 7. Gwen Ryan James Hardiman Library The first year – module on Certificate in Postgraduate Development • Used much the same course outline as was taught previously • All students required to complete a pre course skills self-audit so pretty confident that the classes were pitched at a suitable level • End of course evaluation feedback OK but sensed that some students were bored/perceived some classes as irrelevant to their own research endeavours • Gnawing doubt, how do you REALLY know that your class has made a difference and that students aren’t just taking it for easy credit?
  8. 8. Gwen Ryan James Hardiman Library Description of the revised module • Curriculum based on results of previous local Researchers’ Needs Survey and general reading in the area of PG information skills • Students who elected the module took 3 core units plus 3 units from a choice of 13 other units; & recorded progress in a private blog • All face-to-face classes supported by VLE presence • Students offered choice from range of AHSS Researchers Workshops open to all HSS researchers • End of module assignment: Literature review plan and summary of research proposal supported by list of references
  9. 9. Gwen Ryan James Hardiman Library Impact indicators derived from learning outcomes 1. Proportion of students who are able to map their research plan and have devised an efficient search strategy (competency) 2. Proportion of students who have identified and evaluated the most relevant library databases for their research (competency) 3. Proportion of students who have set up a working EndNote library and submitted a document with citations and references list formatted using EndNote’s CWYW function (competency) 4. Proportion of students who have identified key journals and have set up ToC alerts or RSS feeds for same/set up a citation alert for a key journal citation/set up a topic alert; or have included as part of lit review plan (competency) 5. Proportion of students who state their confidence in their skills to conduct a literature review has increased (affective)
  10. 10. Gwen Ryan James Hardiman Library Example reflective Blog postsPost on December 9 2009 (permalink) -edithistorydelete Created on Wednesday, 12/09/2009 11:24 AM by XXXX XXXXXX Updated on Friday, 12/11/2009 2:26 PM by XXXX XXXXXX Since previous post, I've taken three more sessions in the Researcher Workshop Series. On 25/11, I attended a workshop on Subject Specific E Resources. This gave me an overview of the wealth of resources that are available on JH Library's E-knowledge. At this stage, I am familiar with quite a few of these databases, archives etc. but it was particularly good to get an overview of how the E-knowlege page works as this has changed considerably since I last attended any training in this area, This also introduced me to two new multidisciplinary databases, Web of Knowledge and Informaworld, both of which I will be using to locate information and sources during that particular stage of my literature review. On 27/11, I went to a session on Sourcing Primary Materials. Having attended this, I now know how to apply for membership of the ALCID and SCONUL schemes, which will prove very useful for the research trips I will need to make in subsequent years of my PhD. In addition, I also discovered COPAC during this workshop- this will be invaluable when I begin searching library catelogues during the location of information sources stage of my literature review as it will allow me to search multiple university library catelogues simultaneously. I also acquired some hints and tips on locating references from partial citations, this was especially good, as in semester two, I have several resources that I need to locate for which I do not have the full details. This workshop also introduced me to the British
  11. 11. Gwen Ryan James Hardiman Library Getting the Most from the Web Class (permalink) -edithistorydelete Created on Friday, 12/18/2009 1:18 PM by XXXXXX XXXX Thoroughly enjoyed this class. I use Google Scholar all the time, and it was great to get a bit of advanced information about it. I had never even tried the advanced search option, and am so glad now that I know it's there! I also really liked the part about evaluating the information we find on the web. The spoof sites are great, and just make me stop and think about what I'm looking at before I take the information I find out there for greanted. Thanks Gwen, I've really enjoyed the classes! Happy Christmas! Bonnie
  12. 12. Gwen Ryan James Hardiman Library Assignment • Directly relevant to own research • Designed with intention of keeping all students on course with managing their literature review • Submitted to supervisor as well as course instructor to ensure academic content OK • Pass or Fail • 5 ECTS ~ 100 student hours
  13. 13. Gwen Ryan James Hardiman Library Analysis of assignments 1. Proportion of students who mapped their research plan and devised an efficient search strategy (competency) 4/14 2. Proportion of students who identified and evaluated the most relevant library databases for their research (competency) 8/14 3. Proportion of students who set up a working EndNote library and submitted a document with citations and references list formatted using EndNote’s CWYW function (competency) 13/14 4. Proportion of students who identified key journals and set up ToC alerts or RSS feeds for same/set up a citation alert for a key journal citation/set up a topic alert; or have included this as part of lit review plan (competency) 4/14
  14. 14. Gwen Ryan James Hardiman Library Example of generic literature review plan Example of ideal literature review plan Example assignments
  15. 15. Gwen Ryan James Hardiman Library Post module evaluation results • 17 students registered for the module but only 14 attended classes and submitted the assignment • Module evaluations from those 14 reported here • Overall the workshops fully met the requirements and expectations of 71.4 % (10) of students and partially met the requirements of the other 28.6% (4) • Proportion of students who report both their confidence in conducting their literature review and proficiency in using resources demonstrated has increased - 57.1% (8) and 42.9% (6) reported a partial increase – mapped to Indicator 5
  16. 16. Gwen Ryan James Hardiman Library Recollection of which sessions booked/attended in comparison with actual attendance recorded Student recollection of attendance Students unsure of whether attended Nos. who actually attended (from class records) Preparing your literature review 100.0% (14) 0.0% (0) 100.0% (14) Subject Specific E Resources for your discipline 64.3% (9) 7.1% (1) 71.4% (10) Introductory EndNote for AHSS researchers 100.0% (14) 0.0% (0) 100.0% (14) Sourcing primary materials 42.9% (6) 7.1% (1) 57.2% (8) Using electronic journals effectively 71.4% (10) 0.0% (0) 78.5% (11) Getting the most from the Web 35.7% (5) 0.0% (0) 28.6% (4) Web of Knowledge 28.6% (4) 0.0% (0) 28.6% (4) Rare Books 7.1% (1) 0.0% (0) 7.1% (1) Theses 21.4% (3) 0.0% (0) 14.2% (2) Maps 0.0% (0) 0.0% (0) 0.0% (0)
  17. 17. Gwen Ryan James Hardiman Library Feedback on which unit perceived to be most useful
  18. 18. Gwen Ryan James Hardiman Library Feedback on which session perceived to be least useful
  19. 19. Gwen Ryan James Hardiman Library Other survey results • 92.9%(13) reported level sessions pitched at was just right • General pace was just right for 71.4% (10) but too fast for 21.4%(3) • 92.9% (13) felt the duration of sessions was appropriate • Handouts were not distributed at every session and 9 responded that they would have valued supporting documentation • All responded that they had since referred to documentation they had received (EndNote guide)
  20. 20. Gwen Ryan James Hardiman Library If there is anything that you feel would improve future sessions, or any topics that you would like to see offered in future workshops, please give details here: • “Just slow down a bit. Make sure we're with you before moving on to next slide.” • “Perhaps, the PPT slides used in each workshop could be placed on Blackboard” • “more supporting documents or handouts” • “During application/registration for attendance, it could be helpful if we are asked our expections to ensure that participarts's needs are addresses. This could also help to put classify students according to their expections.” “I wondered at some points if the order in which the sessions are taken would improve the outcome - this however would take from the flexibility of the whole Series, which would be a shame. I named one as being the least useful, which is misleading - it was the least useful only in that I learned less from this one than the others, as much had already been covered. That said, it was still of benefit. The assignment proved to be a real test of learning - I was forced to put the learning into practice very early on and that has already shown up for me areas I will need to re-visit. An assignment like this one undoubtedly contributed to the value of the Series to me.” “No, I genuinely think that it's a really good course“
  21. 21. Gwen Ryan James Hardiman Library Conclusions from students’ module evaluations • Obvious missing data on whether students felt the learning outcomes had been met • Suggestions for improvement/aspects reported as unsatisfactory: – Handouts/Slides added to VLE as matter of course – Tutor’s style too fast – Surveying participants’ expectations before the module
  22. 22. Gwen Ryan James Hardiman Library Changes on foot of what has been learned • Completion of a self audit of information skills on registration will be mandatory • Will reiterate learning outcomes in post-module evaluation survey for students to rate extent to which they were met and will also incorporate post-module skills audit into this feedback tool • Clarify requirements for the assessment - More details and guidance on drawing up literature review plan • Future for this cohort: see if % completions on time greater than in the general cohort of PhD candidates
  23. 23. Gwen Ryan James Hardiman Library Tips of developing your own meaningful criteria for measuring the impact of training modules • Use the assignment to test the programme as well as the students • Translating intended learning outcomes into tangible competencies is relatively easy to do • Where evidence is gathered through assignments and end of course evaluations, students will not be unduly inconvenienced • Performance measurement becomes part of the process

×