Some supervisors and examiners think they are great, they are actually convinced about it, but are they really as good as they think? How have they arrived at such conclusion and would this conclusion be valid in the presence of hard data? If you ask students, you might find a different answer, and hard data will show that. In general, supervisors and examiners are reluctant on having their performance evaluated, however, if they are to enhance their practice, a systematic approach needs to be used where they can obtain knowledge about their strengths and weaknesses which would allow them to take informed decisions on how to improve their practice. This project aims to derive strategies that will lead to a systematic enhancement of the quality of supervision and examination by using quantitative and qualitative methods. The author began assessing his performance from 2011-2013, and using the available data, a new method and instruments for quality enhancement will be derived in the future.

1. Quis custodiet ipsos custodes*?
Enhancement of Supervision and Examination Practices through
Qualitative and Quantitative Methods
Assoc. Prof. Dr. Ing. Luigi Vanfretti
KTH SmarTS Lab
luigiv@kth.se
LH219V
Stockholm, February 10th 2015.

2. Outline
• Motivation
• Observations, Challenges
and Assumptions, Hypothesis
• Goal of this study
• Previous methodology, survey
and sample data available
• What’s next?
• Revisiting Old Practices
• Proposed Methodology
• Enhanced Survey

3. Motivation: Observations
and Questions
• Some supervisors and examiners (S&Es) think
they are great, they are actually convinced about
it.
• Are they really as good as they think?
• How have they arrived at such conclusion?
• Would the S&Es self-assessment be valid if:
• If you ask students: might you find a different
answer?
• If you gather quantitative and qualitative data:
will you see consistent results across different
samples?

4. Motivation: Challenge and Assumptions
• Traditionally, S&Es have evaluated the quality of their teaching
through different methods, e.g. [1,2,3]
• Why is this not an approach also taken to evaluate the quality of
S&E?
• Challenge:
• S&Es are reluctant to evaluate the quality of their S&E
practices.
• Assumption seems to hold [4]
• Assumptions: self-development
• The S/E is a reflective supervisor or examiner.
• This S/E would be willing to use quantitative and qualitative
methods and tools to asses its practice
• This S/E would be willing to apply a systematic method for both
data acquisition, analysis and implementation of changes into
his own practice
Luigi does not, in any way, suggest that this should be even considered to be adopted for assessing
supervisors and examiners. I’m doing it for myself! Let me do my thing…

5. Motivation: Hypothesis
• Hypothesis:
• S&Es are “willing and able” to enhance their practice
• To do this, they will need to assess their performance to pin-point strengthens and weaknesses
• And, using their strengths, devise a plan for for continuous enhancement (and to fulfill their role
properly!)
• Therefore, a method and tools to achieve this are necessary.
• In general, these methods and tools should be able to aid in the following process:
• Reflection and Data Acquisition: Do I suck as a supervisor/examiner?
– I.E. What are my strengths and weakness as an S/E.
• Assessment: In what and how badly?
– How can I measure (quantitatively and qualitatively) the quality of my work?
• Action: Can I do something about it (systematically)?
– How can I change and how can I document these changes?
• Repeat: Closed-loop control loop for continuous enhancement.

6. Motivation: Goal
• Goal:
• To derive a new methodology for assessment and an enhanced survey that can be used in the
future by those supervisors and examiners that are interested in enhancing the quality of their
practice.
• To share the experience I’ve gained through a first iteration of applying such systematic
method.
• Background:
• I started this project a few years ago, when I was in the tenure track process at KTH.
• My skills as a supervisor and examiner where put into question by senior staff
• I needed evidence: to be left alone and do my thing
• Consider this specially when your thing is different and works better compared to ‘the way things have always been
done around here’.
• Critical when someone else has control over your own research funds
• I needed evidence to protect myself - so many things can go wrong in the tenure track
• If someone questions your practice, you can answer with facts
• But overall, I needed evidence to become better at what is supposed to be my real job!
• Of course, this is irrelevant for tenure or a full-professorship, nobody cares if you are a good
supervisor/examiner “as long as you show them the money”!

7. Previous Method
The previous method I used was the following:
1. Adaptation and implementation of a student survey (which
included quantitative and qualitative questions)
• Survey originally proposed in [5]. Changes made to adapt for own
needs.
• First implementation July 2011. Used Google Forms.
• Second July 2013. Changes to broaden breath (more student types (e.g.
BSc, Research Visitors, etc.) + clarify ambiguities.
2. Execution of the survey by the students
• At different stages of MSc or PhD or other.
3. Analysis of responses and open discussion during research
group meetings
• Not documented. Plan to hold one for the updated survey.
4. Implementation of changes.

8. Quantitative Data
Qualitative Data
In depth qualitative assessment.
Explanation and Instructions

9. Aspects Considered in the Survey
Clerical:
• Initial guidance and thesis contract/proposal
• Paperwork and bureaucracy
• Easiness in booking meetings with the supervisor/examiner
• Compliance with harassment policy
• Equal opportunities practices related to gender, race,
nationality, etc.
Quality of Supervision:
• Availability for continuous supervision and progress
feedback
• Time/value of the supervision meetings
• Provision on practical guidance, methods, etc., in early
stages
• Value and quality of feedback orally or written
Preparation of the
Supervisor/Examiner:
• Provision of literature and relative “currency” of the
supervisor/examiner knowledge on published work
• Level of ability/training of the supervisor/examiner
Social and Ethical Aspects:
• Social/psychological/emotional/motivational support
from supervisor/examiner
• Scientific standards and good practice of the
supervisor/examiner
• Ethical standards of the supervisor
• Level of advice, support and academic freedom at
each project stage
Dissemination:
• Opportunities provided for dissemination and
discussion of research results within technical settings
• Opportunities for exchange in research events,
seminars, lectures or others within the domain of the
project or general field

10. Execution of the Survey
• The survey was to be filled up at least 2 times a year by PhD students,
and at least 2 times by MSc students or others.
• For the MSc students, they would fill up the survey at around 50% of the
completion of their project, and a second time at the completion of their
project.
• Nevertheless, the implementation of the survey became more seldom in
2014, actually stopping. This was due to Luigi burning out…
• An additional execution of the survey will take place in February 2015,
which will be used in the preliminary data analysis.
• Towards the development of a new methodology and enhanced survey, a
“preliminary” version of the new survey will be executed following early
feedback.

11. Some Sample Quantitative Information
(2011-2013)
Project Progress Degree Information
Completed Projects

12. Some Sample Quantitative Information
(2011-2013)

13. Some Sample Qualitative Information
(2011-2013)
• From a MSc Student: Sometimes you don’t necessarily suck, even if everyone tells you that you suck.
Quantitative and qualitative data will show if there is consistency with other students,
or if this was an outlier case.
A good supervisor may change whole the future of his students.
Since now, I have attended many evaluation meetings, and I always criticized the style of supervision here. The Msc
students have a supervisor for their last semester and not the whole period of study, and no one care for their records
and improvements.
The supervisors are not that much intimate, and they usually don't support students in an adequate manner.
However, I found Luigi to be a different one. He is so motivated, fresh, and full of passion for the job. He works till late
in the night, and even during the weekends just to organize and manage everything in the best possible way. He is so
warm with his students, lets them to express themselves, and provide the necessary freedom for the students to
choose their own style. He asks a lot from the students, and gives them lots of opportunities and chances instead.
…
I love his academic style, and feel extremely comfortable in the research group. He is very intelligent, respectful, and
transfers the motivation to the students also. If it was not due to some limitations, he may perform even better.

14. Some Sample Qualitative Information
(2011-2013)
• From a Male MSc Student:
I experienced a big change during the Thesis. At the start I was not able to work alone and required help always.
I had not my own criteria in order to perform the work.
Luigi really gave me the opportunity to work by myself and taught me how to choose wisely and how to work in
an accurate manner.
So I would like to highlight that I grew personally and educationally while working in the project.
In addition, such an important commitment forced me to be constant and responsible for my work as I had to
report about it.
What I really think was a outstanding was the meetings with Luigi and Tetiana, it was incredible how they were
able to propose new solutions and they came up with ideas. At the beginning I was a little lost but while the time
was going on we improved the ideas and we achieved great results.
Finally, I would to say that having such a nice team as the SmartSLab provides a lot of support and always I
needed help I had it.
• From a Female MSc Student:
He seems to treat women kindlier than men, which is good for me (since I'm a girl) but not for the guys, I believe.
Perception might vary, while the quantitative and qualitative data will show if there is any consistency
in my practices.

15. Some Sample Qualitative Information
(2011-2013)
• From a Female PhD Student:
After having been under Dr. Luigi’s supervision for about one year, little by little, I have come to realize several important
things which have significantly influenced my view of the project I am currently conducting. With the guidance from Dr. Luigi,
I started to have a clearer picture and a better understanding of the topic; the important problems in my field, the relevant
issues, the directions I should be aiming at, and how to tackle the problems. By gradually being questioned, I have
developed a ‘self-questioning’ habit and a thinking process – train of thought – in which I have learnt to connect thoughts
systematically.
Additionally, having been working with Dr. Luigi, I have realized the different (academic) levels of standards, of which I have
now complied with the one thought to have high and international standard, and academic morality. By observing him, I
have realized the difference between a regular researcher and a very good researcher, in which he is the role model, and
what made him became one of those.
One challenge faced under Dr. Luigi’s supervision is that he always comes up with (many nice) new ideas, to catch up with
them and become more independent can be difficult. However, after discussing with him, he agreed to loosen it up. Up to
the present moment, I am satisfied with the situation and his supervision.
Two crucial attitudes a supervisor should have, in my opinion, are a commitment to the role and a will to improve him/herself
to become a better supervisor. I have seen these attitudes from Dr.Luigi; he often asks my opinion about his supervision
attentively listens to what I say and is open to comments as well as suggestions. He is willing to change his supervising
style if it is believed to improve the situations.
In summary, I value the experiences I have had with Dr. Luigi; I have learned many different aspects from him, even from
the mistakes, both by my own and by his own. Last but not least, I started to have ‘fun’ with the project which I did not
have it before, thanks to Dr. Luigi.
Accepting to make changes will improve your student’s feelings about the project, and they will even
start having fun!

16. 2014 - No data (from Feb.)
2015 – Some preliminary data
Quantitative:
Only 8 responses out of 22 expected. Too early to show figures that are meaningful.
Qualitative:
Students notice the difficulties you face (response from 2013)
• I am really sad because he is suffering a lot from the stup** hierarchy system here, and he is always forced to do things
that he doesn't want. He is working in a very stressful situation…
Some things are now worse – but the students want to give good data to help fixing things
• From this question onwards, the grades reflect my personal perception of the last year. I'm totally aware of your situation,
and the consequences it may involve on your supervision of the different students, therefore it is not in an attempt of
criticizing, but more for helping you determine areas of possible improvement, as I noticed you were these past month
making a strong effort in getting better and more present.
Some things are still good
• The freedom that you enjoy while working under the supervision of Dr. Luigi Vanfretti is extra-ordinary. He will give you
some basic ideas during discussions and will allow you to explore it. I was indeed blessed to have such a supervisor and i
am sure i wont be able to have such a Boss in my life again.

17. So… what’s next?
Revisiting Old Practices
• What’s next?
• Revisiting Old Practices
– Revisit the method and the survey.
– Perform data analysis – determine how the survey
has to be analyzed (which methods?) to extract
weaknesses and strengths.
– Check for ambiguities in survey.
– Carry out and document an open workshop with the
students that remain in the research group.
• Important to determine how the examiner’s
“health” affected the students, and what to do
about it.

18. So… what’s next?
QaQ Techniques and Methodology
• Methods
– Investigate which quantitative and qualitative analysis methods could
be best suited for the available data
– Make modifications to the survey in order to gather data more
accurately
• Proposed Methodology
– Do the survey again, with updates.
– Define a new method using the results of the survey and by applying
the methods.
– Assess the results from the methods through a set of repertory grid
interviews as a cross-validation test.
• Enhanced Survey
– Will make available, together with analysis, in a publication.

19. Thank you!
Questions? Constructive Comments?
Want to help? Or buy me a beer? Get me a date?
luigiv@kth.se
Good luck with the rest of your day!
(mine is already hopeless...)

20. References
[1] L. Vanfretti and M. Farrokhabadi, “Consensus-Based Course Design and Implementation of Constructive Alignment Theory in a Power
System Analysis Course”, European Journal of Engineering Education, 2014.
https://dx.doi.org/10.1080/03043797.2014.944101
[2] L. Vanfretti and M. Farrokhabadi, “Evaluating Constructive Alignment Theory Implementation in a Power System Analysis Course through
Repertory Grids”, IEEE Transactions on Education, vol. 56, no. 4, pp. 443 – 452, Nov. 2013.
DOI: http://dx.doi.org/10.1109/TE.2013.2255876
[3] L. Vanfretti and Federico Milano, “Facilitating Constructive Alignment in Power System Engineering Education using Free and Open
Source Software,” IEEE Transactions on Education, vol. 55, no.3, August 2012, pp. 309-318.
DOI: http://dx.doi.org/10.1109/TE.2011.2172211
[4] S. Taylor, “Managing postgraduate research degrees”. In H. Fry, S. Ketteridge and S. Marshall (eds) The Effective Academic: A Handbook for
Enhanced Academic Practice. London: Kogan Page.
[5] S. Taylor and N. Beasley. A handbook for doctoral supervisors. Routledge, New York, 2005.