Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Building Motivation, Achievement
and Progression Online
Evaluating the Brightside Online-Mentoring Approach
Siobhan Neary
...
“Without this scheme, I doubt I
would be as capable as I am
now in achieving the goals I set,
and the goals set would be m...
Overview of presentation
 Background to project
 Methodology
 Engaging and supporting partners
 Engaging and sustainin...
Background to project
 International Centre for Guidance Studies (iCeGS) was
commissioned in late 2012 to undertake an ev...
Methodology
Adopted a mixed methods approach including;
 A literature review
 Interviews with Brightside staff
 Intervi...
Methodology (Continued)
 The sample for the online survey was 2211 based on age,
engagement level and availability of ema...
Engaging and supporting partners
 Organisations selected by Brightside to represent long term and recent
relationships an...
Engaging mentees
www.derby.ac.uk/icegs
Profile of participation in online mentoring
www.derby.ac.uk/icegs
Note: This power law
distribution is typical of
online ...
Usability issues for participants
 Most users were happy with their experience.
 Some raised technical issues. Generally...
Models of delivery: 4 key approaches
1. As a repository of reliable online information, with little or no use of the
mento...
Example from blended delivery
“It was really nice meeting you this week and I hope you had
a really good time at the summe...
Quality assessment
 A key aim of the evaluation was to make an assessment of the quality
of mentoring available
 Assessm...
Quality identifiers for online mentoring
1. establish an appropriate relationship
2. establish the purpose of the conversa...
Some examples of quality mentoring
 “I hope to be able to be an effective mentor for you while you apply to
university, o...
More examples of quality mentoring
 “As you may or may not know the programme will be ending shortly
which means you’ve o...
Quality identifiers in evidence?
Quality identifier Total
relevant
Yes (%) Partially
Yes (%)
No (%)
establish an appropria...
Average likelihood of any one quality
identifier being present
Where quality was most likely to be in
evidence
1. establish an appropriate relationship (4% not in evidence)
2. establish...
What determines bad practice?
 The projects: Average number of “Nos” ranges from 0.17
– 2.67
 Gender
www.derby.ac.uk/ice...
The content of e-mentoring conversations
www.derby.ac.uk/icegs
Choosing a University course 180
Choosing a University 164
...
Areas covered in mentoring conversations
by category
www.derby.ac.uk/icegs
What participants liked about e-mentoring
www.derby.ac.uk/icegs
Additional sources of support
While you were engaged in the e-mentoring programme did you seek help and support in any oth...
Impacts on mentees
“I have been more ambitious in having a variety of different experiences
and skills. It has made me wan...
www.derby.ac.uk/icegs
Did online mentoring help you to make
decisions?
“It helped me decide university was an option and helped me rule out
some...
Help making decisions
"The programme that you suggested was really interesting. It has really
had me thinking and consider...
www.derby.ac.uk/icegs
Did online mentoring help you to do things
differently?
“I became more aware of the work I needed to put in at Sixth Form ...
Doing things differently
“Thanks for all the tips + support on Personal statement- I think the
points you raised were very...
Proportions who ‘strongly agree’ before
and after online mentoring
www.derby.ac.uk/icegs
What were mentees most likely to report an
increase in?
I am aware of a range of different options for my future 51%
I hav...
In summary
 Brightside is well liked by its partners
 The initial engagement of mentees is the most
challenging issue to...
In summary 2
 The quality of mentoring is generally high, but there is room for
improvement, particularly in the areas of...
Questions:
How do these findings reflect your
experience?
What might you use these findings?
www.derby.ac.uk/icegs
Upcoming SlideShare
Loading in …5
×

Evaluating the Brightside Online Mentoring Approach - ICEGS

589 views

Published on

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Evaluating the Brightside Online Mentoring Approach - ICEGS

  1. 1. Building Motivation, Achievement and Progression Online Evaluating the Brightside Online-Mentoring Approach Siobhan Neary www.derby.ac.uk/icegs
  2. 2. “Without this scheme, I doubt I would be as capable as I am now in achieving the goals I set, and the goals set would be much more vague.” www.derby.ac.uk/icegs
  3. 3. Overview of presentation  Background to project  Methodology  Engaging and supporting partners  Engaging and sustaining participation in online mentoring  Models of delivery  Quality of provision  Content of online mentoring conversations  Measuring the impact  Conclusions www.derby.ac.uk/icegs
  4. 4. Background to project  International Centre for Guidance Studies (iCeGS) was commissioned in late 2012 to undertake an evaluation of the Brightside online mentoring programme. This is the first systematic evaluation of the effectiveness of the Brightside approach  iCeGS is a research centre with expertise in career and career development. The centre conducts research and provides consultancy and professional development for the careers sector. www.derby.ac.uk/icegs
  5. 5. Methodology Adopted a mixed methods approach including;  A literature review  Interviews with Brightside staff  Interviews with Brightside partners ( both individuals and groups of staff)  An online survey (n=555)  Analysis of online discussions between mentors and mentees (n=366)  The research relates to interactions taking place in academic year 2011- 2012  The criteria required a minimum of 2 messages had been exchanged between mentor and mentee  All participants were over 16 years of age  The empirical work was built on analysis of Brightside’s existing statistics www.derby.ac.uk/icegs
  6. 6. Methodology (Continued)  The sample for the online survey was 2211 based on age, engagement level and availability of email addresses  65% female, 35% male  All were emailed with details of the on-line survey at least twice and telephone calls were used to ensure a representative sample  The base population for the analysis of online conversations was 3450. The sample was 10%  The analysis of the online conversations focused on the range of topics and the quality of the intervention www.derby.ac.uk/icegs
  7. 7. Engaging and supporting partners  Organisations selected by Brightside to represent long term and recent relationships and different types of schemes. Most were located within widening participation activities  Brightside is a good organisation to work with “We liked Brightside because of their passion for what they do and because of the online content for mentors and mentees.”  Partners see the organisation as responsive and helpful around technical and broader queries  They liked the training and support materials for mentors that are provided by Brightside  Some uncertainty as to exactly what the support covers but the team were always responsive to requests  However, they would like more support for mentors i.e. supporting sustainability  Technical elements could be improved including moderation processes www.derby.ac.uk/icegs
  8. 8. Engaging mentees www.derby.ac.uk/icegs
  9. 9. Profile of participation in online mentoring www.derby.ac.uk/icegs Note: This power law distribution is typical of online phenomenon n.b. The highest number of interactions were 124 and 137
  10. 10. Usability issues for participants  Most users were happy with their experience.  Some raised technical issues. Generally these were about the clunkiness of Brightside in comparison to mainstream tools that they were used to (Facebook, IM etc.)  The message system was perceived as inefficient at times  Creation and remembering of passwords and users names was an issue for some  Technical issues were only raised by a small minority and there was no evidence to support these contributed to lower levels of engagement www.derby.ac.uk/icegs
  11. 11. Models of delivery: 4 key approaches 1. As a repository of reliable online information, with little or no use of the mentoring function 2. As a communication tool to support face-to-face projects with little or no mentoring  Using the system as a message board. Mentors were usually matched with many mentees and acted as a facilitator of group discussions and feedback 3. As an e-mentoring system in which mentoring takes place entirely online  Most projects used this approach, approximately 55% were conducted wholly online 4. As a b-mentoring system in which mentoring is delivered through a mix of online and onsite activities  This blended (b) approach accounted for approximately 34%. Online mentoring is used to extend and sustain a relationship for example in summer schools www.derby.ac.uk/icegs
  12. 12. Example from blended delivery “It was really nice meeting you this week and I hope you had a really good time at the summer school and also a safe journey home. Just want to give you a quick reminder about the e-mentoring system. I will be emailing a few times a month with information regarding various aspects of the university application process and useful information for your last year at sixth form/college. However feel free to email me with any questions you may have at anytime as I am more than happy to respond and give you as much information I can. I will aim to reply as soon as possible and not leave you waiting ages for a response.” Mentor (online conversation) www.derby.ac.uk/icegs
  13. 13. Quality assessment  A key aim of the evaluation was to make an assessment of the quality of mentoring available  Assessment of complete sets of text-based mentoring conversations often occurring over a period of time  A framework was developed drawing on guidelines from the mentoring and befriending foundation, relavent literature exploring the issue of quality mentoring and quality assessment tools for web based careers advice developed by iCeGS  A 10 element quality assessment framework was developed which it is argued describe a quality mentoring experience  The conversations were reviewed and coded in relation to each of the quality identifiers www.derby.ac.uk/icegs
  14. 14. Quality identifiers for online mentoring 1. establish an appropriate relationship 2. establish the purpose of the conversation 3. provide information or links to resources 4. refer to appropriate services 5. prompt and relevant responses 6. encourage the mentee to reflect on their own strengths and weaknesses 7. encourage the mentee to explore their career goals 8. identify opportunities or explore ways to overcome barriers 9. move the mentee progressively towards their goals 10. bring the process to a mutually satisfactory close www.derby.ac.uk/icegs
  15. 15. Some examples of quality mentoring  “I hope to be able to be an effective mentor for you while you apply to university, offering advise and someone to talk to when, where and if you need it. Hopefully you will find the process helpful, and benefit from it as much (or more) than I hope I will. More detailed about the e-mentoring will be (or will have been) discussed at the e-mentoring introductory session.” Mentor (online conversation)  “Now for some more mentoring specific stuff. The first thing that would be great to do is to establish a weekly contact schedule so we know when to write to each other etc. Can I suggest that I'll write to you by Tuesday evening, and you by Friday? Also, I will try and answer the questions you have, and will ask you questions about things so we can get the most out of this.” Mentor (online conversation) www.derby.ac.uk/icegs
  16. 16. More examples of quality mentoring  “As you may or may not know the programme will be ending shortly which means you’ve only got me to use for a limited period of time. But don’t worry; you are an extremely confident, intelligent and capable individual. I’m sure you will have great success in whatever path you head down.” Mentor (online conversation)  “Thank you for the article about moving up to A level. It was really good to know that perhaps it isn't too scary moving up to A level! I also hadn't really thought about how I'd use my free time. It hadn't really occurred to me I'd need to sort out whether I was doing an enrichment activity one free period or studying English in another.” Mentee (online conversation) www.derby.ac.uk/icegs
  17. 17. Quality identifiers in evidence? Quality identifier Total relevant Yes (%) Partially Yes (%) No (%) establish an appropriate relationship 281 85 11 4 establish the purpose of the conversation 295 82 13 4 provide information or links to resources 287 67 14 20 encourage the mentee to explore their career goals 272 66 19 15 prompt and relevant responses 348 63 28 9 identify opportunities or explore ways to overcome barriers 256 63 18 20 refer to appropriate services 237 62 11 27 move the mentee progressively towards their goals 288 60 22 18 encourage the mentee to reflect on their own strengths and weaknesses 272 60 23 17 bring the process to a mutually satisfactory close 241 41 24 34
  18. 18. Average likelihood of any one quality identifier being present
  19. 19. Where quality was most likely to be in evidence 1. establish an appropriate relationship (4% not in evidence) 2. establish the purpose of the conversation (4%) 3. prompt and relevant responses (9%) 4. encourage the mentee to explore their career goals (15%) 5. encourage the mentee to reflect on their own strengths and weaknesses (17%) 6. move the mentee progressively towards their goals (18%) 7. provide information or links to resources (20%) 8. identify opportunities or explore ways to overcome barriers (20%) 9. refer to appropriate services (27%) 10. bring the process to a mutually satisfactory close (34%) www.derby.ac.uk/icegs
  20. 20. What determines bad practice?  The projects: Average number of “Nos” ranges from 0.17 – 2.67  Gender www.derby.ac.uk/icegs N Average Female mentor - Male mentee 68 0.54 Yes - both female 169 1.14 Male mentor - Female mentee 48 1.54 Yes - both male 64 2.19
  21. 21. The content of e-mentoring conversations www.derby.ac.uk/icegs Choosing a University course 180 Choosing a University 164 Subject choice 112 Work experience while at school 82 Entry requirements for courses 74 Whether to apply to University 58 Module choice 47 Placements while at University 46 Moving away from home 40 Personal Statement 40 Interview skills 36 Choosing a college course 36 Finding student accommodation 33 Revision/exam preparation 32 Project 30 Job search and CV 25 Logistics/Arrangements 24 Making new friends 21 Career choice 20 Caring responsibilities 19 Whether to go to college 15 Changing course 15 Assessment centres 10 Student finance 10 Work experience at university 9
  22. 22. Areas covered in mentoring conversations by category www.derby.ac.uk/icegs
  23. 23. What participants liked about e-mentoring www.derby.ac.uk/icegs
  24. 24. Additional sources of support While you were engaged in the e-mentoring programme did you seek help and support in any other ways? Tick all that apply I searched for information on the internet 481 94% I talked to my family about my future 423 82% I visited universities, colleges and/or employers 409 80% I talked to friends about my future 403 78% I talked to my teachers about my future 389 76% I talked to a careers professional about my future 203 39% www.derby.ac.uk/icegs
  25. 25. Impacts on mentees “I have been more ambitious in having a variety of different experiences and skills. It has made me want to be a more well-rounded individual so I was not solely about my chosen subject.” Mentee (survey) “My mentor has been a source of constant support. Being the first member of my family to attend university, I was not entirely sure what to expect with the application process. She has helped to keep me motivated and has given me invaluable advice and insight into my future career. My confidence has certainly grown as a result of this.” Mentee (survey) www.derby.ac.uk/icegs
  26. 26. www.derby.ac.uk/icegs
  27. 27. Did online mentoring help you to make decisions? “It helped me decide university was an option and helped me rule out some careers which was just as useful.” Mentee (survey) “It helped me further develop my understanding of the area of work I wanted to go into in the future.” Mentee (survey) “I explored my ambitions for the future and my e-mentor advised me and gave me information. My e-mentor helped me to write up and re-draft my personal statement for my university application.” Mentee (survey) “Realised I didn't want to go to university.” Mentee (survey) www.derby.ac.uk/icegs
  28. 28. Help making decisions "The programme that you suggested was really interesting. It has really had me thinking and considering all my options.” Mentee (online conversation) “I just wanted to say the message about Durham was really helpful! I am going to go along to some open days, I guess then I can really judge the distance from home and whether or not I will like that!” Mentee (online conversation) "I didn't even consider some of those things about choosing universities!” Mentee (online conversation) www.derby.ac.uk/icegs
  29. 29. www.derby.ac.uk/icegs
  30. 30. Did online mentoring help you to do things differently? “I became more aware of the work I needed to put in at Sixth Form to achieve and progress to University. I was motivated to work harder by having contact with someone who had been through A levels already”. (Mentee survey) “My e-mentor was good at helping identify my own skills and the skills I needed to develop and how they could help me in later life and my career”. (Mentee survey) “It made me believe in myself more and gave me confidence”. (Mentee survey) www.derby.ac.uk/icegs
  31. 31. Doing things differently “Thanks for all the tips + support on Personal statement- I think the points you raised were very true and I made the appropriate changes.” (Mentee online conversation) "I have certainly taken your advice and I have been reading a lot more than I used to and I find that it is really beneficial to me.“(Mentee online conversation) “I'd just like to say thank you for this help you are giving me Mark. You are seriously impacting my life. I feel like a much more driven determined individual.” (Mentee online conversation) www.derby.ac.uk/icegs
  32. 32. Proportions who ‘strongly agree’ before and after online mentoring www.derby.ac.uk/icegs
  33. 33. What were mentees most likely to report an increase in? I am aware of a range of different options for my future 51% I have an understanding of what I need to do to achieve my career goals 49% I have a clear idea of what I want to achieve from my career 46% I am confident in my ability to be successful 39% I am confident about my communication skills 38% I am confident about my ability to attain good qualifications 36% I am confident about my writing skills 33% I enjoy learning and am motivated to succeed 27% I am confident about my IT skills 19% www.derby.ac.uk/icegs
  34. 34. In summary  Brightside is well liked by its partners  The initial engagement of mentees is the most challenging issue to address  The online mentoring delivers high quality mentoring that has clear impacts for participants  91% were satisfied or very satisfied with their online mentoring experience  56% said online mentoring helped them to make decisions  49% said online mentoring helped them to do things differently  The online platform can be embedded in different ways to meet young people’s needs www.derby.ac.uk/icegs
  35. 35. In summary 2  The quality of mentoring is generally high, but there is room for improvement, particularly in the areas of referral to information and services and in the area of progressing the mentee on and ultimately closing the relationship  The mentoring conversations are strongly focused around the process of choosing and applying to higher education  Mentees enjoy the experience of being mentored and report a number of benefits  Some mentees report enhanced skills and knowledge – most notably in the area of career decision making and career management  It is effective in helping young people to transition to higher education by helping them to think where they want to go and what they need to do www.derby.ac.uk/icegs
  36. 36. Questions: How do these findings reflect your experience? What might you use these findings? www.derby.ac.uk/icegs

×