Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QUALITY OF EDUCATION IN THE UNIVERSITIES OF UKRAINE: STUDENT ASSESSMENT

as part of the Project Implementation «Ukrainian quality assurance e.service for education»: 2019-2020
 
KYIV – 2020

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

QUALITY OF EDUCATION IN THE UNIVERSITIES OF UKRAINE: STUDENT ASSESSMENT

  1. 1. QUALITY OF EDUCATION IN THE UNIVERSITIES OF UKRAINE: STUDENT ASSESSMENT NON-GOVERNMENTAL ORGANIZATION ELIBUKR   IN COOPERATION WITH THE NATIONAL UNIVERSITY OF KYIV-MOHYLA ACADEMY AND NATIONAL AGENCY FOR HIGHER EDUCATION QUALITY ASSURANCE (UKRAINE)   WITH SUPPORT OF THE EMBASSY OF BULGARIA IN KYIV, UKRAINE AS PART OF THE PROJECT IMPLEMENTATION «UKRAINIAN QUALITY ASSURANCE E. SERVICE FOR EDUCATION»: 2019-2020         KYIV – 2020
  2. 2. ABOUT US Non-governmental organization ELIBUKR was founded in 2009 aiming to promote the integration of higher education and science of Ukraine in the world academic community. ELIBUKR projects intend to develop scholarly communication, Open Science, and Open Access to scholarly information, academic integrity, quality of education. Read more: http://elibukr.org 02
  3. 3. 02 The National University of Kyiv-Mohyla Academy (founded in 1615, rebuilt in 1991) is a classic research university, the initiator of most innovations in the field of higher education in Ukraine. Read more: http://www.ukma.edu.ua National Agency for Higher Education Quality Assurance (founded in 2015) – is an independent, permanent collegial body that implements the state policy in the field of quality assurance in higher education. Considers itself as a catalyst for change in higher education in Ukraine to form a culture of its quality. Read more: https://naqa.gov.ua/ Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 03 PROJECT PARTNERS
  4. 4. IMPORTANCE OF THE PROJECT Students are stakeholders and real participants in the quality assurance system of educational activities in universities. Their opinion is important for assessing the quality of teaching, as well as for the development of training courses and educational programs. One of the main tools for assessing the quality of Higher Education Institutions is regular student surveys conducting. Unfortunately, such surveys, particularly in Ukraine, have not yet become a daily practice for most Higher Education Institutions. Also, algorithms for the systematic application of their results have not yet been created. The project aims to change the situation for the better by developing a technical and methodological solution for conducting surveys of students (also, if necessary, surveys of any other stakeholders, including graduates and employers) and creating an Independent Nationwide Research Portal for Quality Assurance in Higher Education. Ensuring the quality of higher education is a requirement of modernity, an indisputable priority for the academic community and public education policy in countries of The European Higher Education Area (EHEA) and other developed countries. Education and science affect all structural elements of any society, are a factor in its modernization, social, technological, and innovative development. They are also included in the parameters of national security. Without exaggeration, the future of the country depends on the quality of higher education. How to assess the quality of education in Ukrainian Higher Education Institutions? ·        To make sure that the university develops and responds to the challenges of time, to the needs of the labor market? 1 2 Compare? 04
  5. 5. National University of Kyiv-Mohyla Academy (NaUKMA) – is the oldest and the most time-honored university in Ukraine (founded in 1615), today – a classic research university, the initiator of most innovations in the field of higher education in Ukraine since 1991. WHY WAS THE PROJECT LAUNCHED IN THE NATIONAL UNIVERSITY OF KYIV-MOHYLA ACADEMY? The quality of education has always been and remains a key priority for NaUKMA. Insistence on high standards, zero tolerance towards any academic dishonesty, deficiency of corruption, critical thinking, and an atmosphere of true academic freedom are the hallmarks of the internal culture of this university. Regular student surveys on the quality of courses and teaching have been among the components of the internal quality assurance system of education at NaUKMA since the 1990s. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 05
  6. 6. Upon words of the former NaUKMA President (2007-2014), the former Minister of Education of Ukraine (2014-2016), the Head of the National Agency for Higher Education Quality Assurance (since 2017), Professor Serhiy Kvit: “Initially, we had the task of resuming regular student surveys before each session regarding the quality of courses and teaching. Back in the 1990s, such surveys were conducted on paper in NaUKMA and were freely distributed throughout the university in consolidated files. In those days students could simply fill out questionnaires with a few questions - and immediately pass them over to volunteers. Now the situation has changed on a global scale, as the transition to online surveys in all universities around the world has led to a sharp decline in student participation. Now they need to find the time and inspiration to personally register and fill out lots of questionnaires for all the courses. first, in order not to perceive calls for surveys as spam, students attention should be drawn to the teachers, who should devote their time to filling in surveys during their courses. Second, students should feel that their participation in surveys affects the situation at the university. Answering my question about the best way to involve students in such surveys, one of the leaders of The European Students' Union (ESU), Lea Meister at a symposium in Brussels in June 2017, indicated that: 06
  7. 7. The National University of Kyiv-Mohyla Academy decided to follow this path - giving students time and opportunity to register and answer the questionnaire on the last pair before the session. Now, according to the "Regulations on the educational process of NaUKMA", the online survey has become a part of the training itself. It is considered that the student has completed the course when he/she had attended classes, assessed the course, and passed the exam or test. It is important to emphasize that there is no coercion. Students recognize the importance of regular surveys, understand that in this way they can affect the academic quality of the University. However, paradoxically, they do not have enough time. They also ask questions about anonymity. NaUKMA, for its part, emphasizes that such surveys should only be anonymous. It should become part of the academic culture of the Kyiv-Mohyla Academy to clarify the real circumstances, take into account students' perspectives (through their answers to open questions), disseminate positive teaching experience, promote the academic quality of the University.” Source: Serhiy Kvit. Quality Assurance of a Modern University (2017): https://bit.ly/2RLIMbt Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 07
  8. 8. The National University of Kyiv-Mohyla Academy (Kyiv) V.N. Karazin Kharkiv National University (Kharkiv) Uzhhorod National University (Uzhhorod) Vasyl Stus Donetsk National University (Vinnytsia) Donetsk National Technical University (Pokrovsk, Donetsk Region) Petro Mohyla Black Sea National University (Mykolaiv) Since NaUKMA has many years of experience in organizing, conducting, and summarizing the data of student surveys on the quality of teaching and learning (including in a modern online format), a model was built that can be recommended to other universities and scaled throughout the country. The pilot project of the Independent Nationwide Research Portal for Quality Assurance in Higher Education is currently being tested at 6 universities in Ukraine (including 2 displaced from the zone of temporary occupation): 08 Independent Nationwide Research Portal for Quality Assurance in Higher Education
  9. 9. RESEARCH The conducted analysis demonstrated the extremely significant and central place of systematic student surveys on the quality of teaching and learning in the internal system of quality assurance of education. An analysis of international experience in the quality assurance system of higher education through the study of regulations and projects of: • The European University Association (EUA) concerning the development of the culture of quality in European universities, • The European Association for Quality Assurance in Higher Education (ENQA) (formerly the European Network for Quality Assurance in Higher Education), • Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), • Project QUAERE-562013-EPP-1-2015- 1-PL-EPPKA2-CBHE-SP «Quality Assurance System in Ukraine: Development on the Base of ENQA Standards and Guidelines»: (2016 survey conducted in 217 universities in Ukraine), • as well as relevant experience of specific universities: The University of Toronto (Canada), The University of Glasgow (Scotland, UK), Maastricht University (The Netherlands), Stanford University (USA), Princeton University (USA), Rutgers University (USA), et al. The subject of the study was only one key component of the system - student surveys on the quality of teaching, while other relevant components were analyzed (usually indicated by respondents: quality assurance processes in teaching and learning; evaluation; procedures for approval and monitoring of curricula; qualification of professors and faculty; educational resources and student support; information systems, etc. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 09
  10. 10. 25,200 In Ukraine, student surveys are conducted in various forms in many Higher Education Institutions, at least as evidenced by the 2016 survey: "Regular student surveys on the quality of educational activities are conducted in 176 Higher Education Institutions in Ukraine" (see https://bitly.su/erCYVSCY). However, only 20% of these surveys relate to the actual quality of educational programs. As a rule, Higher Education Institutions do not have regulated norms for considering the results of student surveys and taking appropriate measures based on the results of these surveys. A short number of universities provided students who took part in the surveys with information about the results and the course of actions taken based on these results. This trend does not allow students to form an awareness that they are the real participants in the quality assurance system of educational activities. 75% About the quality of work of scientific and pedagogical workers *data of the 2016 survey: "Regular student surveys on the quality of educational activities are conducted in 176 Higher Education Institutions in Ukraine" Does your higher education institution conduct regular student surveys? * 20% About the quality of educational programs 5% Surveys are not conducted Such surveys are an important component of student-centered learning in the world's leading universities. They allow better consideration of students' needs and requests, as well they become a tool for ensuring academic integrity and combating corruption. 10
  11. 11. GENERAL TRENDS IN CONDUCTING SURVEYS Only consistent problem solving of these issues allows us to obtain results that will be reliable, valid, consistent, and representative, and therefore will allow concluding on the quality of teaching disciplines, to assess existing trends. Several main issues commonly relate to organizing such surveys: • Ensuring the confidentiality and anonymity of respondents (students), • Prevention of results manipulation - both at the level of deformation of the numbers or in the act of completion of questionnaires, as well as at the level of interpretation and further results usage, • Ensuring a high level of student participation ("Response Rate") and, e.g. the percentage of responses to the total number of students who had taken the discipline). • Involvement and interest of teachers and administration of the Higher Education Institutions in the constructive use of the results to improve the quality of teaching and educational programs. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 11
  12. 12. NaUKMA EXPERIENCE The single portal, developed following the Project, guarantees the anonymity and confidentiality of each respondent through Authorization, Access Control, and differentiation of Access Levels. Regular student surveys have been conducted at the National University of Kyiv- Mohyla Academy since the mid-1990s and starting from 2016 they have completely switched to the online format. These surveys must be conducted regularly (three times per year, before each session). Under the scope of the project, a detailed analysis of the previous experience of surveys in NaUKMA was conducted, especially concerning the mass involvement of students, the timing of its conduct (this must be done before obtaining the results of the exam or test) or the relevance of the results only to students registered for a course, monitoring results during longer (than one semester) period, correlation of results to overall success, analysis of use and consideration of results for acceptance of academic or administrative events, etc. The methodology of the Survey (preparation of research tools, collection, processing, and interpretation of results, etc.) was analyzed in detail. Based on the study of the NaUKMA experience and the relevant research, a methodological and technical solution (creation of the Independent Nationwide Research Portal for Quality Assurance in Higher Education) is proposed, which allows Ukrainian universities to introduce systematic and consistent student surveys on the quality of teaching. 12
  13. 13. THE LEVEL OF STUDENT PARTICIPATION With the transition from paper to online surveys, the level of student participation (Response Rate) decreases. Many solutions have been proposed to solve the problem, including initiation of the obligatory participation in the discipline and face-to-face interviews using tablets (which turned out to be very expensive). It emerged that the most effective way to increase the level of student participation is to work with the faculty and ask (or directly formulate a requirement) for time to fill in an electronic questionnaire during the last seminar or lecture. Therefore, information work, including advertising campaigns, SMM campaigns and personalized emailing on the importance of the survey should be conducted not only with students but also with the faculty, among whom, unfortunately, sometimes might be an opposition to the idea of such surveys. Increasing the level of student participation (response rate) and, accordingly, the uttermost use of the survey results in the educational process is possible only through the formation and dissemination of a culture of such surveys, understanding that they are not a temporary whim, but are an integral part of learning and educational process at university on the national level and broader for Academia. This entails, in fact, about forming a (corporate/institutional) culture of conducting such systematic surveys and ensuring a high level of trust at all stages. Of course, when it comes to trust, the issue of avoiding manipulation and ensuring the anonymity and confidentiality of respondents becomes fundamental. Information about students' answers, their distributions, and survey results, to which the faculty and administration have access, is always an aggregated, generalized information, by which it is not possible to identify an individual student. Student surveys are an integral part of university life. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 13
  14. 14. It should be noted that with the initiation of authorization, the level of student participation in NaUKMA initially decreased significantly (almost twice). This can be partly explained by the peculiarity of the registration process in the system, the unwillingness to spend extra time on registration or perform any additional actions in general, the vanishing effect of novelty. Among the main reasons for low activity was also the lack of feedback (how the student surveys affected the review of courses of a department, the work of a particular teacher). In the following stages of the Survey, the level of participation (Response Rate) increased slightly (to 21% and then to 43%), considering the average for the university in all specialties and years of study. However, it should be noted that the number of questionnaires in most disciplines is less than 10, only a few disciplines from all over the university had a student participation rate of more than 60%. On the positive side, we note that the level of participation among 1-2- year students is higher than among senior students, especially in master's programs. The constant need to inform students more about the survey appears relevant, as well as on the peculiarities of its conduct and results processing, guarantees of anonymity in the analysis of responses, and in general, the role and importance of survey results on the educational process improvement. Over the four years of online student surveys on the quality of teaching and learning at NaUKMA, the percentage of trust in e- services has grown: from 10-12% of the total number of students in the early stages to over 53% in the last year. We emphasize the importance of the corporate or institutional culture of surveys and providing feedback and constructive criticism in general, both at the level of students and the level of the faculty and the administration of the Higher Education Institutions. It is important to position student surveys as not a temporary phenomenon, but an integral part of learning and, more broadly, university life, as well as one, albeit very important, component of the education quality assurance process. 14
  15. 15. ON THE PECULIARITIES OF THE SURVEY RESULTS USAGE IN THE SYSTEM OF EDUCATION QUALITY ASSURANCE not exceptions, but indicate systematic successful practices or victories or problematic moments or shortcomings. At the same time, when discussing the results and their use, it should be emphasized that it is primarily a question of quality assurance of teaching disciplines, not teachers. The same teacher might receive different "Grades" for the different disciplines, especially if students of different specialties or faculties take these courses. Therefore, "High / Low Grades" of individual courses for a year or two cannot be considered as a sufficient basis for output about teachers, but, of course, is a signal to the department about the need to monitor the situation. It should also be acknowledged that students of different programs and different years may have a different middle or baselines - that is, evaluate all disciplines critically. Accordingly, "Assessments" as such are not sufficient to conclude - their dynamics, trends, "Open Answers" and relevant trends must also be considered. Inquiries about the quality of teaching cannot be expected to be immediate, although some expectations can be met by both students and the faculty, especially those who are actively involved in the educational process. Accordingly, it makes sense to run an information campaign with the emphasis that answering the questionnaire now will help improve the course for students in the future, and that the survey cannot be used as an element of "reconciliation" nor a magic tool for instant improvement. The results of surveys can be used meaningfully and productively only by recording a certain consistency, reliability, durability, and repeatability of the results; making sure that the data obtained are Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 15
  16. 16. All this indicates the need to develop policies for dissemination of survey results, ensuring maximum privacy of all stakeholders, along with the fullest possible use of results in the educational process. It is also important to emphasize the guarantees of anonymity and confidentiality of students who took part in the surveys, even in situations where they do not deny the disclosure of their names. It is essential to form an understanding that students and teachers are not antagonists or opponents, but colleagues and allies who are united by a common goal and common cause. Constructive and positive attitude to the result of all participants in the educational process - ensuring and improving the quality of teaching and learning - is the key to success. It should be noted that there is merit in supplementing the student surveys on the quality of disciplines with surveys of the faculty on their professional activities in the Higher Education Institutions and surveys of teachers and students on the work of administrative departments (for students - for example, deans). 16
  17. 17. The list of Closed-Ended Statements: • During the lecture, the teacher explained the material well. • The course interested me and contributed to my intellectual development. • Overall, I am satisfied with the quality of the course. • I am equally satisfied with the lectures and seminars. • I am satisfied with the way the teacher helped me, provided enough comments and "feedback". • The teacher encouraged the active participation of students. • The thematic material of the course was well structured. • The course was sufficiently supplied with literature, information, materials, and technical resources. • The grading system was clear and consistent with the content of the discipline. • Compared to other disciplines, this was… • Practical classes and/or seminars helped me to better understand the content of the discipline. • The teacher's assessment was fair. THE STRUCTURE OF THE QUESTIONNAIRE ON THE QUALITY OF COURSES IN NaUKMA Initially, the Survey at NaUKMA was conducted based on the Questionnaire with 12 Closed-Ended Statements, with which students (respondents) had to agree or disagree (on a scale from 5 to 1). The volume of the Questionnaire consisting of 12-14 questions is on the verge of optimal, and its increase even by 3-4 questions with a high probability will reduce the level of answers/fillings. The Questionnaire also contained 2 Open-Ended Questions: • What did you like most about studying the discipline? • What did you not like the most and what could be changed? Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 17
  18. 18. The question of assessing those disciplines that are taught by several teachers (in the most common version - lectures and seminars are given by different people) remains as a problem point of the research. At the stage of the Questionnaire and the Survey preparation, it was decided that for such courses students should fill in the Questionnaire once, indicating in the appropriate line the names of both (all) teachers. In terms of content, the statements of the questionnaire can be divided into those related to lectures and those that are more about seminars. This approach based on the assumption that the need to fill in the same questionnaire for each teacher would reduce the level of responses in general. A pilot survey showed that in most cases, students filled in the Questionnaire either by indicating the names of two teachers, as expected, or exclusively by the name of the seminar teacher. In this case, in response to the first Open-Ended Statement, it was often stated that the lectures were given by another teacher and the takeaway from these lectures. In a few isolated cases, students did not bother to fill in the Questionnaire twice for each teacher, but all of them recorded a drastic gap in the assessment of the level of teaching lectures and seminars (in most cases - lectures are rated very well, while seminars - poorly). However, the need to fill in two questionnaires (or even three) for the same subject does not increase the level of student involvement (response rate) and the number of answers. Moreover, it raises the question of how much students can differentiate impressions from lectures and seminars and, accordingly, different teachers from each other. The survey is usually conducted during the last few weeks of the course before the session and in general, at this time students have more or less holistic and integrated impressions of the discipline. 18
  19. 19. EVALUATION SYSTEM Particular attention in the Survey (the Questionnaire) was paid to the System of Evaluation and Fairness of Evaluation. The analysis of the results, in particular with the involvement of answers to Open- Ended Statements, showed that students disagree on the assessment system and its adequacy and transparency, compliance with the content of the discipline ("The grading system was clear and consistent with the content of the discipline") and impressions of the fairness of the grades (“The teacher's assessment was fair”). Whereas the issue of teacher evaluation and attitude is discussed by students in Open-Ended Questions, it is very sufficient that Closed-Ended Questions can record this. For example, very often students stated that the assessment system is adequate and understandable, but the actual grades are not fair. Of course, having an Open-Ended Questions helps to specify the problem, for example, there was a clarification about the availability of the faculty members, conflict of interests, and so on. The Questionnaire "Testing" in real conditions, namely during the several survey staging points at NaUKMA, showed that the Statement "I am equally satisfied with the lectures and seminars" was ambiguously interpreted by the respondents and was accordingly removed from the questionnaire. Thus, to some extent, the problem of interviewing in co-teaching disciplines requires further testing of tools. It is worth mentioning that different solutions may be suitable for different Higher Education Institutions or even different faculties and specialties. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 19
  20. 20. It should be noted that it is important to conduct the Survey before the session, before the final exam, because the actual grade obtained in the discipline should not affect the answers of the respondents. After the first two stages of the Survey, the Questionnaire has been edited. As already mentioned, one question was removed, in exchange for the question of readiness to recommend this course to others, the expected assessment of the discipline (answer in the A-E range in complying with European Credit Transfer and Accumulation System (ECTS) scale) and the subjective assessment of the time required to prepare for classes (Interval Assessment ): • "I would advise others to enroll in this course" • "What grade do you expect to get in this discipline?" • "How many hours a week did you spend preparing for this discipline?" The use of the Questionnaire in the following stages of the survey showed that the Closed-Ended Statements work well, do not duplicate each other. It should also be mentioned that in general the list of the Questionnaire statements and questions as well as the applied methodology of both the Questionnaire and the Survey correspond to and are consistent with those used in Western Universities to assure the quality of teaching disciplines. Theoretically, one or two (1-2) more questions can be added to the Questionnaire. Also, the Questionnaire can be adjusted to the needs of different Higher Education Institutions. At the level of the Independent Nationwide Research Portal and with the participation of the National Agency for Higher Education Quality Assurance (Ukraine), it makes sense to identify the key issues that will be in all questionnaires (Digital Core) and the variable part, which may differ. Although the level of participation (Response Rate) in the Survey at NaUKMA does not allow, in general, to talk about certain correlations or their absence, it is possible to mention certain trends at least for certain disciplines: e.g., the level of satisfaction with the course and willingness to recommend it to other students. Meanwhile, there is no such correspondence between the complexity of the course and / or the expected assessment and the satisfaction and / or willingness to recommend it. However, it should be emphasized again that these are only preliminary conclusions or a hypothesis, while higher student participation (Response Rate) is generally needed for more reliable and representative results. 20
  21. 21. FEATURES OF INTERPRETATION OF THE QUESTIONNAIRE QUALITATIVE PART (OPEN-ENDED QUESTIONS) With great credibility, Ukrainian universities, including NaUKMA, will have to come a long way to ensure consistently high levels of student participation (Response Rate) in the Survey (80%) and to build an appropriate corporate culture of participation and trust. Accordingly, the problem of an insufficient number of answers to conduct the quantitative analysis of Closed-Ended Statements of the Questionnaire is highly relevant. At the same time, the inability of the results quantitative analysis does not exclude the discussion and interpretation of qualitative data, namely the answers to two Open-Ended Questions. The essential points to remember are the limitations imposed by the peculiarities of qualitative methodology - namely, that not so much the prevalence is being recorded as the range of opinions and the main content categories without specifying their level of prevalence. Also, the limitation is the inability to draw unambiguous conclusions about any course or teacher. Moreover, with low participation rates (response rate), the level of open answers is quite high - it can even exceed 50% (ie - more than half of the completed questionnaires have answers to Open- Ended Questions). Usually, this figure is up to 25% (up to a quarter of questionnaires). This is partly due to the phenomenon of self-selection - the first and most active students to fill in the questionnaires are those students who have something to say about the disciplines and their teaching - both good and bad. Therefore, these students are always inclined to fill in the questionnaires. Furthermore, some of them even put their names (in Open-Ended Questions). This proves once again the importance of surveys as a tool/feedback/opportunity to express and convey students' views to the faculty, departments, and administration. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 21
  22. 22. From the analysis of NaUKMA's experience, most of the questionnaires contain meaningful answers to both questions, indicating both the positive aspects of the course and what students would like to change. A separate category consists of questionnaires, where the answer to one of the questions rather continued the main positive aspect, in most cases, the assessment of the course. Together, all this indicates a rather positive and constructive attitude of students. Moreover, if there were several Open- Ended Questions in a questionnaire about the course, then, as a rule, students pointed out similar strengths and weaknesses, systematically mentioned the same issues. That is, a certain repeatable accuracy of answers was recorded. At the same time, the presence of several negative comments does not deny the high assessment of the course in general (as well as vice versa, but the opposite situation is less common). Naturally, this does not preclude the presence of single "emissions", both positive and negative, which may be evidence of a special subjective attitude. All this only emphasizes that to use the obtained results, e.g. by departments, it is necessary to understand that their additional "Verification" (Triangulation) The attitude of students is overwhelmingly constructive, focused on learning, and provides continuous improvement. from other sources is necessary, which will show systematicity and durability concerning these positive or negative aspects of teaching. Moreover, the specifics of the survey generally require repeated results over a long time and preclude the "Immediacy" of conclusions based on only one or two surveys. In addition to their impressions of the courses and teaching, students have mentioned about their success or problems encountered while taking the course, expressed admiration or disagreement, wrote reflections on the specifics of the curriculum and organization of training, reflected on their own experience and experience of colleagues. The general impression from reading the answers to Open-Ended Questions convinces that the attitude of students is mostly constructive and focused on learning and continuous improvement. It seems that this attitude of students should be appreciated and nurtured, through feedback. 22
  23. 23. • Structure of the course - first of all, the presence of Thematic Work Plan / Syllabus and its observance, logical and systematic presentation of material, consistency of lectures, seminars and practicality of classes, correspondence of the name and annotation of the discipline to its actual content and educational goals (focus on achieving or developing), clarity of the content and essence of educational tasks, in particular, written works, availability of methodical instructions and requirements to them. This is exactly what students mention in the first place when pointing out the shortcomings of the course. • The teacher's attitude to students and the evaluation system - its intelligibility and transparency and its fairness is pointed out, also indicating bias or inappropriate attitude towards students. As a rule, the answer to two Closed- Ended Statements of the questionnaire about the evaluation system is revealed in depth. In general, mainly the systematic and consistent remarks (repeated in time and on different stages) and within a separate sample group (within one stage) about the shortcomings and strengths are ANALYSIS & INTERPRETATION The detailed analysis of answers to Open-Ended Questions allows us to group them by main topics or issues. Among them: the key points for the University to pay attention to improve the quality of education. • The level of interest of the teacher. Disadvantages that do not depend on the faculty and the department, namely comments on the place and time of classes, logistics (including laboratory equipment and the convenience of course websites), the number of students in groups. It is worth noting that such comments cannot be rejected or ignored. Whenever possible, the wishes of students should be taken into account, where the situation can be improved by the joint efforts of the faculty, university administration, teachers, and the department. • Therefore, answers to Open-Ended Questions can and should be used in the process of the quality assurance in higher education, in case they are carefully and professionally interpreted, even when the level of student participation (Response Rate) in the survey is 30-40% and insufficient for quantitative (statistical) analysis of answers to Closed-Ended Statements of the questionnaire. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 23
  24. 24. 24 FUNCTIONALITY OF AN INDEPENDENT NATIONWIDE RESEARCH PORTAL FOR QUALITY ASSURANCE IN HIGHER EDUCATION With the help of the portal, educational system users and managers will conduct the focused research and surveys of a wide range of the system beneficiaries. According to the survey results, the portal will form an appropriate analysis of the external quality assurance of education system. Also, the portal functionality will inform the National Agency for Higher Education Quality Assurance and the Ministry of Education and Sciences of Ukraine on operational issues in education, track trends and tendencies. It will also ensure that decision-making criteria are in place in the field of higher education and will facilitate the establishment of accessible and understandable reporting, supported by disregarded data. The research results will inform about the need to conduct operational inspections of quality assurance systems and mechanisms for working with the received recommendations.
  25. 25. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 25 on direct curricula, on the quality of education in the relevant programs, and on their university overall. In the further steps of the portal development, special attention will be paid to the development of a module for monitoring the effectiveness of quality assurance procedures for educational and professional training, as well as automation of preparation and publication of statistical information for reports on quality of higher education in Ukraine. At another point, the beneficiaries of the system will have 24/7 access to a direct impact on the quality of education and their own experience in the higher education system of Ukraine. They will have the opportunity to leave feedback: They will also be able to respond to more strategic decisions in the education quality assurance system. At the university level, heads of institutions will have access to the appropriate level of analytics, which will inform them about possible strategic and operational decisions to ensure the quality of education in their universities. Access to the portal will be restricted to users of the education system only. They will be properly authenticated in the system. This will protect against the provision of inaccurate data, as well as ensure the confidentiality of data, which will eliminate any dishonest impact. In the future, it will be possible to add login through other providers, such as Google, if the Higher Education Institutions is using the corporate environment (the provider must support the OpenID Connect protocol). It is also possible to establish cooperation with id.gov.ua to use other means of identification: EDS, etc.).
  26. 26. 26 THE MAIN STEPS OF THE PORTAL MANAGEMENT Step 1 Login to the system under the roles provided in the system is carried out by following the hyperlink to the authentication page. At the MVP stage, the login is implemented through the Office 365 authentication service. We chose this method of Authentication and Authorization for several reasons. In 2014, Microsoft developed a program for Ukrainian higher education institutions, according to which students and teachers began to use the Office 365 platform on preferential terms. Using a corporate email address to log in allows administrators in Higher Education Institutions accurately identify their user, and higher education seekers can securely access the system. The service development roadmap provides the use of Google+ and BankID for authorization (depending on the dynamics of this authentication method implementation in Ukraine). After filling in the required data (password, login, etc.), the system provides information about the user (name, surname, e-mail, etc.) and uses this information to authorize the user in our system.
  27. 27. Step 2 Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 27 the choice of disciplines for the survey on the quality of teaching; the page on which the survey results are posted; the feedback form; settings for Higher Education Institutions (for administrators). The user gets to the main page, where, depending on his role in the system, the user has access to the functionality of the portal: At the top of the page the user has access to a menu with information about this user and buttons to go to the main menu and settings page. At the bottom of the page, the user has access to a block with information about the owners and organization. The user will also have access to the button, clicking on which will provide this user with help on how to use the service.
  28. 28. 28 Step 3 Presentation of the discipline choice and questionnaire. The feedback page provides the user with a list of available disciplines and writing a review. By clicking on the button next to the selected disciplines, the user goes to the questionnaire filling page. If the user has already filled out the questionnaire for a specific discipline, the fill button will be replaced by an edit button that allows user to edit the data that was entered in the questionnaire for a specific discipline.
  29. 29. An employee of the university's quality assurance department has an additional option "View results" in the main menu, clicking on which the user goes to the university selection page to get the results of a survey on the discipline of a particular university that interests him. Portal guarantees results anonymity. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 29 Step 4 Step 5 The discipline results page displays statistics in the form of graphs and charts, and it is possible to download the report form.
  30. 30. 30 Step 6 If the user fills in the questionnaire for the first time, the questionnaire is presented in the form of various fields with blank or unselected data. If the user has already filled in the questionnaire, the text or variants of the completed questions will already be given. After all questions are answered, at the bottom of the page there will be a button "Send Feedback", which groups all the answers and sends them to the server and the button "Cancel", which completely deletes all the data entered this time and goes to the feedback page. Step 7 If the user selects "Leave Feedback", user gets to a new page. The page displays an input form in which the user writes his question, requests or feedback on the use of the system. If the user is not registered, this user must indicate his contact details in the feedback form. To send a "System Feedback", the user checks the entered data and presses the confirmation button. The functionality of the Independent Nationwide Research Portal for Quality Assurance in Higher Education allows a wide range of users to influence the quality of education in Ukraine.
  31. 31. Further steps in the development of the portal's functionality include the following stages (Product Roadmap): • development of an online system for monitoring the effectiveness of quality assurance procedures for educational and professional training; • development of an online system for accreditation of education and training programs; • creation and maintenance of an IT register of educational and training programs. Quality of education in the universities of Ukraine: Student assessment Kyiv - 2020 31 PRODUCT ROADMAP
  32. 32. QUALITY OF EDUCATION IN THE UNIVERSITIES OF UKRAINE: STUDENT ASSESSMENT. AS PART OF THE PROJECT IMPLEMENTATION «UKRAINIAN QUALITY ASSURANCE E. SERVICE FOR EDUCATION»: 2019-2020 / KVIT S. , YAROSHENKO T. , KOSTYUK O. , OSYPCHUK H. , YAROSHENKO O. , NGO ELIBUKR. – KYIV, 2020. - 28 P. PHOTO CREDENTIALS: NATIONAL UNIVERSITY OF KYIV-MOHYLA ACADEMY, OLEKSANDRA LYSTOPAD (NAUKMA SCIENTIFIC LIBRARY), OLEKSANDRA YAROSHENKO (ELIBUKR), ARTEM SAKH (NAUKMA), IMAGES WITH OPEN LICENSE - PEXELS (PEXELS.COM)

×