Successfully reported this slideshow.
Your SlideShare is downloading. ×

Avoiding Invasive Surveillance, Ensuring Trust: ENSURING TRUST UNED’S AvEx

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 52 Ad

More Related Content

Similar to Avoiding Invasive Surveillance, Ensuring Trust: ENSURING TRUST UNED’S AvEx (20)

More from EADTU (20)

Advertisement

Avoiding Invasive Surveillance, Ensuring Trust: ENSURING TRUST UNED’S AvEx

  1. 1. AVOIDING INVASIVE SURVEILLANCE, ENSURING TRUST: UNED’S VIRTUAL EXAMINATION HALL (AVEX) José L. Aznarte 14/06/2022
  2. 2. OUTLINE About me About my institution: UNED Ethical framework for the use of data-based technologies Assessment under COVID19 1/23
  3. 3. ABOUT ME
  4. 4. TEACHING AND RESEARCH • I’m ”profesor titular” at the Artificial Intelligence Department of UNED. • I teach Data Mining and Machine Learning in master’s level. • I also supervise PhD students and master’s theses. • My main research line: • Time series forecasting with deep learning & statistics • Applied to common/shared problems: air quality, traffic intensity, epidemic propagation… • I coordinate some R&D projects: • SOCAIRE (Municipality of Madrid) • PreCoV2.org (Ministry of Health) • Chair EMT/UNED for Air Quality and Sustainable Mobility 2/23
  5. 5. SERVICE AT RECTORATE • Appointed in 2019 as deputy vicerector in charge of digitalisation and innovation. • Co-leading the ED3 institutional project (”Digital, Distant and Data-powered Education”). • Interface with IT department. 3/23
  6. 6. SERVICE AT RECTORATE • Appointed in 2019 as deputy vicerector in charge of digitalisation and innovation. • Co-leading the ED3 institutional project (”Digital, Distant and Data-powered Education”). • Interface with IT department. Actually… 3/23
  7. 7. SERVICE AT RECTORATE • Appointed in 2019 as deputy vicerector in charge of digitalisation and innovation. • Co-leading the ED3 institutional project (”Digital, Distant and Data-powered Education”). • Interface with IT department. Actually… Many doubts about the use of AI tools for dealing with people’s data. 3/23
  8. 8. ABOUT MY INSTITUTION: UNED
  9. 9. GENERAL INFORMATION ”National Distance Learning University” – UNED • One of the biggest universities in Europe (>150.000 students) • Founded 50 years ago (no digitalization) • Mixed remote and face-to-face model: • Network of regional centers (>50) • Remote teaching during the course • F2F exams • Wide range of different student situations. Mostly not ”freshers” coming from high-school. 4/23
  10. 10. ED3: DISTANT, DIGITAL AND DATA-POWERED EDUCATION (2019) 5/23
  11. 11. ED3: DISTANT, DIGITAL AND DATA-POWERED EDUCATION (2019) «Develop a framework for evidence-based interventions to improve teaching/learning processes through response-able exploitation of data. • Policies which guarantee that use of data takes into account potential social and ethical consequences. • Identify, gather, curate and make accesible all the data sources related with teaching/learning processes, with different access rights for each profile. • Analize data and prepare them for knowledge creation through exploratory analysis and operational models. • Promote interventions over teaching/learning processes based on evidences resulting from data and the knowledge of the different stakeholders». 5/23
  12. 12. ETHICAL FRAMEWORK FOR THE USE OF DATA-BASED TECHNOLOGIES
  13. 13. PARTICIPATORY PROCESS • First things first: we needed a shared ethical common sense, which goes beyond regulation, about what happens with data and how can we extract value from it. • We carried out a participatory process, open to the entire UNED community, to agree upon a series of cautions that should be taken into account when using data-based technologies. 6/23
  14. 14. PARTICIPATORY PROCESS • First things first: we needed a shared ethical common sense, which goes beyond regulation, about what happens with data and how can we extract value from it. • We carried out a participatory process, open to the entire UNED community, to agree upon a series of cautions that should be taken into account when using data-based technologies. • Open during 2 months in 2020, >2.500 people participated. • Used the ”Decidim” participatory software. 6/23
  15. 15. PARTICIPATORY PROCESS • First things first: we needed a shared ethical common sense, which goes beyond regulation, about what happens with data and how can we extract value from it. • We carried out a participatory process, open to the entire UNED community, to agree upon a series of cautions that should be taken into account when using data-based technologies. • Open during 2 months in 2020, >2.500 people participated. • Used the ”Decidim” participatory software. • 9 basic cautions were proposed by the rectorate. • During the process, these were rearranged in terms of the community’s perception of their importance and new 4 with high support were added to the document. 6/23
  16. 16. OUTCOME An official document encoding a set of 13 cautions for the use of data-based technologies, related with: • care • response-ability • transparency • consent • property & control • validity and trust • participation • privacy • preventing potential adverse impacts • effective communication • adaptability • right to explanations 7/23
  17. 17. Then… 8/23
  18. 18. Then… COVID19! :( 8/23
  19. 19. ASSESSMENT UNDER COVID19
  20. 20. PRE-COVID19 EXAMINATION PROCEDURE 9/23
  21. 21. PRE-COVID19 EXAMINATION PROCEDURE • Students choose one of two alternative dates and go to a regional center to undertake an exam. • Exams take place synchronously in halls and are invigilated by ad-hoc teams of faculty staff. • Students might be required to leave bagpacks, purses and other material before entering the hall. • Exams are scanned upon hand-over, gathered in a centralized computer system and distributed to each course’s team. 9/23
  22. 22. AVEX: VIRTUAL EXAMINATION HALL We needed a tool to reproduce the conditions of usual examination under lockdown: • Synchronous questionnaire distribution and response management. • Variety of exam types (test, essay, math…). • Proper identification of students. • Anti-fraud measures. 10/23
  23. 23. AVEX: VIRTUAL EXAMINATION HALL We needed a tool to reproduce the conditions of usual examination under lockdown: • Synchronous questionnaire distribution and response management. • Variety of exam types (test, essay, math…). • Proper identification of students. • Anti-fraud measures. These measures must be non-invasive (at least no more than exam halls). 10/23
  24. 24. AVEX: VIRTUAL EXAMINATION HALL We needed a tool to reproduce the conditions of usual examination under lockdown: • Synchronous questionnaire distribution and response management. • Variety of exam types (test, essay, math…). • Proper identification of students. • Anti-fraud measures. These measures must be non-invasive (at least no more than exam halls). Q: Can we delegate exam invigilation to an AI? 10/23
  25. 25. FACIAL RECOGNITION TECHNOLOGIES (FRT) IN EDUCATION • First step: study the risks of facial recognition technologies for exam invigilation. • Facial recognition systems applied to surveillance are expanding. • There are mounting evidences that these technologies can be problematic: technical, legal and ethical difficulties. 11/23
  26. 26. RISKS OF FACIAL RECOGNITION TECHNOLOGIES 1. There is no clear legal framework for invasive surveillance technologies. 12/23
  27. 27. RISKS OF FACIAL RECOGNITION TECHNOLOGIES 1. There is no clear legal framework for invasive surveillance technologies. 1. The use of FRT might imply a violation of the legal principles of necessity and proportionality. 12/23
  28. 28. RISKS OF FACIAL RECOGNITION TECHNOLOGIES 1. There is no clear legal framework for invasive surveillance technologies. 1. The use of FRT might imply a violation of the legal principles of necessity and proportionality. 1. FRT can violate privacity rights. 12/23
  29. 29. RISKS OF FACIAL RECOGNITION TECHNOLOGIES 1. There is no clear legal framework for invasive surveillance technologies. 1. The use of FRT might imply a violation of the legal principles of necessity and proportionality. 1. FRT can violate privacity rights. 1. FRT are naturally imprecise, and its software is fallible. 12/23
  30. 30. RISKS OF FACIAL RECOGNITION TECHNOLOGIES 1. There is no clear legal framework for invasive surveillance technologies. 1. The use of FRT might imply a violation of the legal principles of necessity and proportionality. 1. FRT can violate privacity rights. 1. FRT are naturally imprecise, and its software is fallible. 1. FRT can produce automatization bias. 12/23
  31. 31. RISKS OF FACIAL RECOGNITION TECHNOLOGIES 1. There is no clear legal framework for invasive surveillance technologies. 1. The use of FRT might imply a violation of the legal principles of necessity and proportionality. 1. FRT can violate privacity rights. 1. FRT are naturally imprecise, and its software is fallible. 1. FRT can produce automatization bias. 1. FRT can produce discriminations and violations of the equality principle. 12/23
  32. 32. RISKS OF FACIAL RECOGNITION TECHNOLOGIES 1. There is no clear legal framework for invasive surveillance technologies. 1. The use of FRT might imply a violation of the legal principles of necessity and proportionality. 1. FRT can violate privacity rights. 1. FRT are naturally imprecise, and its software is fallible. 1. FRT can produce automatization bias. 1. FRT can produce discriminations and violations of the equality principle. 1. FRT can generate discriminations based on different functional abilities. 12/23
  33. 33. FACIAL RECOGNITION TECHNOLOGIES IN EDUCATION 13/23
  34. 34. AVEX: VIRTUAL EXAMINATION HALL 14/23
  35. 35. AVEX: VIRTUAL EXAMINATION HALL • We underwent express development of our own technological solution: AvEx. • In less than 6 weeks, first version operational. • In first exam call, over 400.000 exams with few issues (~1%). 14/23
  36. 36. SECURITY MEASURES Layered security scheme: • Wide question banks • Reasoning-based questions • Open-book exams • Tight timing per question 15/23
  37. 37. SECURITY MEASURES Layered security scheme: • Wide question banks • Reasoning-based questions • Open-book exams • Tight timing per question • Randomization of q&a • Sequential access • Copy/paste disabled • Random photographs 15/23
  38. 38. SECURITY MEASURES Layered security scheme: • Wide question banks • Reasoning-based questions • Open-book exams • Tight timing per question • Randomization of q&a • Sequential access • Copy/paste disabled • Random photographs • Sworn affidavit • Plagiarism: text similarities • Phone interview after the exam 15/23
  39. 39. SECURITY MEASURES Layered security scheme: • Wide question banks • Reasoning-based questions • Open-book exams • Tight timing per question • Randomization of q&a • Sequential access • Copy/paste disabled • Random photographs • Sworn affidavit • Plagiarism: text similarities • Phone interview after the exam 15/23
  40. 40. RESULTS (I) First analysis (after 2 exam calls): 16/23
  41. 41. RESULTS (I) First analysis (after 2 exam calls): • Amount of exams: ↑~20% (of courses) • Pass marks: ↑~40% • Average marks: ↑~50% • Remote sessions attendance: ↑~300% • Disciplinary issues: ↑ 16/23
  42. 42. RESULTS (II) 17/23
  43. 43. RESULTS (III) 18/23
  44. 44. RESULTS (IV) 19/23
  45. 45. DISCUSSION What do these results mean? Is the system less secure? 20/23
  46. 46. DISCUSSION What do these results mean? Is the system less secure? Hypotheses: • More time to study in lockdown • Changes in the assessment design and criteria • More continuous evaluation • Easier to attend exams • Fraud? 20/23
  47. 47. STUDENT SATISFACTION • 86% of students declares negative impacts of lockdown in study. • 40% link it to mood status • Students declare more difficulties to study: • 54% due to work-related issues • 35% due to illness • 20% less time 21/23
  48. 48. STUDENT SATISFACTION • 86% of students declares negative impacts of lockdown in study. • 40% link it to mood status • Students declare more difficulties to study: • 54% due to work-related issues • 35% due to illness • 20% less time • Elements hitting on performance: 21/23
  49. 49. STUDENT SATISFACTION • Academic results: • 50% similar, 25% better or much better • 70% identify some positive element (more time, more motivation…) 22/23
  50. 50. STUDENT SATISFACTION • Academic results: • 50% similar, 25% better or much better • 70% identify some positive element (more time, more motivation…) • 67% is happy or very happy with AvEx • Most students appreciate the efforts of UNED • Young people have worse oppinion 22/23
  51. 51. STUDENT SATISFACTION Privacy protection is enabled under AvEx: 23/23
  52. 52. THANKS! jlaznarte@dia.uned.es 23/23

×