all started with archie - tried to reduce the amount of duplication of effort by physicians by creating a library of systematic reviews of the literature which could give definitive statement about best evidence (+ therefore practice) in particular situation. but these things take time, and when there isn&apos;t a systematic review available, you still have to go out and see if anyone&apos;s solved the problem that you have already.... evidence based practice..... sackett..... evidence based practice - 5 steps Step 1 - Convert your information need into an answerable question Step 2 - Find the best evidence Step 3 - Appraise search results for validity and usefulness Step 4 - Apply the findings to your clinical practice and step 5 - evaluate your professional performance (step 6 - disseminate findings....)
skills required have since become embedded in professional competencies - royal colleges, &quot;modernising medical careers&quot;, &quot;skills for health&quot; etc etc.....
- even with the european working time directive, staff are time poor
- while at least there are regular intakes of new jnr doctors in august and feb (note to self - avoid hospital at these times!) (though not of other staff) they can&apos;t be considered a cohort in the way a new intake of students can.
- skill levels vary enormously (predictable)
- reasons for attending vary enormously (cpd, direct patient care, publishing article, professional exams, &quot;on list of things to do and they&apos;ve got a free afternoon - can I fit them in??&quot; ) and sessions are likely yto be a one-off, so few/limited opportunities for follow-up - adult learners who often just want to know what they need to know right now, but who are also self-selecting (have recognised a rumsfeld moment - they know that they don&apos;t know something....)
562 pre – training surveys completed across the region 163 matched post-training surveys Response rate of 29%
First of all, To try and ascertain whether trainees had retained learning we asked the same question on the pre and post surveys about awareness of electronic resources. For each resource we asked trainees to rate whether they had Never Heard of it Heard of it, but not used it Used it occaisonally Used it frequently Bearing in mind that training content would have varied across the region, and not all trainees would have been trained on all resources, the results were certainly encouraging. This slide shows the main resources and the numbers reporting they had never heard of it before and then 6 weeks after training. In all cases the numbers decreased, indicating less people had never heard of the resources after the training than before, Even if individuals did not then access resources after training at least there is some recall of fact, and we can hope that they would be aware of where to go for information if the need arises. The 2 there that still have around 50 people after training are EEL and Proquest. EEL is the Eastern Electronic Library for Health and Social Care, which is promoted as a starting point for accessing resources by some, but not all of the participating Trainers, and Proquest a journal collection that though many trainees may have accessed it may have been part of wider access to electronic journals.
We asked a similar question for search techniques. We listed the techniques: Keyword Searching Subject Headings MeSH Boolean Truncation Major / Explode And asked trainees to rate whether they had Never Heard of it Heard of it but not used Used during a search Again numbers reporting to have never heard of a technique decreased after training, showing trainees are recalling facts six weeks after a training session.
However, we wanted to show more than just our trainees could remember and recall resources and techniques, we wanted to show they were actually using what they had learned. As afterall, we know that for learning to be properly retained is has to be used.
The same is true for frequent use of search techniques. Were techniques used after training? This graph shows that yes, they were. Numbers self reporting to have used a technique frequently after training increased. Useage and the 2/3 things remembered
So far then, we have shown trainees remember what they have learned, abd have shown they are using techniques and resources. But what about specific usage. We know that learning is most effectviely retained through reinforcement. Much of the training would have been around doing a literture search in a database, such as Medline or Cinahl. How many trainees had actually done their own search after training. We found that three quarters had done a search, 95% either fully or partially planned the search first and 89% of those had actually used subject headings in that search, either fully or partially.. Bell paper – need to reinforce learning through doing. Have they been doing? Yes - % done a literature search and used subject headings
Another aspect of the training we were particularly interested in was the impact of it on trainees work activities or studies. We found that most reported that it did have an impact, but in very general terms. Such as “Increased confidence and knowledge to use data bases and search for evidence” and “It has taught me how to do literature searches and look for journals which have been key to my work” However, some examples given were more specific as shown here. Some respodents could demonstrate that the training had a direct impact on their work, with a consultant being able to do his oen literautre searcj to inform an expert report, the junior doctor undertaking a systematic review, the GP doing a specific search to help with patient care, and the occupational therapist who was able to find the evidence to update treatment guides. We feel we hsve the baisis here for critical incident technique Critical incident technique How are they using it after training - impact
All the graphs are going in the right direction, and we’re making a valuable, measurable difference to the skill set of these professionals
Link to Methods to increase response to postal and electronic questionnaires. Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Cochrane Database Syst Rev. 2009 Jul 8;(3):MR000008. Review. PMID: 19588449 [PubMed - indexed for MEDLINE]
Evaluating your training - a pragmatic approach. The East of England experience. Lepley, Hockley & Kuhn
Evaluating your training:
a pragmatic approach
the East of England experience
Herding cats… us!
31 libraries across 6 counties
autonomous but collaborative
style and delivery of training sessions
content of session might vary depending
on group vs 1-2-1, trainer, IT….
A core set of searching skills / techniques:
Subject Headings / MeSH
Explode / Major Descriptors
Using a variety of electronic resources:
National Library for Health (now Health Information
Resources / NHS Evidence)
Databases (Medline, CINAHL, Embase, etc)
What we wanted to find out…
Use resources after training
Use search techniques after training
Do they retain learning?
What we did
Built on our previous Essex study
Developed pre- and post-training evaluations
Pilot using paper surveys
Transfer to Surveymonkey
Region wide roll out
Completed by all trainees prior to training
Assessed baseline knowledge and resource /
technique use & awareness
Link emailed 6 weeks after training
Follow up depended on individual trainers
Same questions on resource & technique use
Additional questions on literature searching & impact
Have they used their new
Learning is retained through reinforcement
75% did a literature search after training
89% used Subject Headings
What was the impact of
Had to provide an expert report on one of
my patients – I used the search tools and
techniques learned on the course
updated treatment guides
used the training
most recently to
look at evidence
I was able to complete a
systematic review on
interval in MTOP
Would you recommend the
99% said YES
“In fact I already have recommended it!
I found it opened up facilities that
I didn't know existed.”
“Finding what evidence is available in an efficient way is
such an important skill.
Being able to be confident that if evidence is not found it is
not because skills are lacking but that the evidence is not
available is also important.
This training helps start on that journey.”
Junior Doctor A
Junior Doctor B
we’re making a valuable contribution to the
daily work and CPD of these professionals!
So what now?
Improve/update the pre- & post-training
MCQs or other tests?
Personalised post-training questions?
Use benefits of http://bit.ly/butwYz
Review training sessions in light of changing
resources and skills of learners
Get more sessions translated into online
versions and evaluate further