The document discusses evaluating e-learning programs. It describes evaluating at different levels, from student satisfaction to organizational results. Micro and macro perspectives of evaluation are presented. Common evaluation tools include questionnaires, interviews, and forums. Key aspects to evaluate include student satisfaction, learning outcomes, abilities application, and organizational impact. Frameworks like Kirkpatrick's are referenced for guiding evaluation.

1. M-ICTE2009 Conference, Lisbon, 22-24 April José Reis Lagarto

2. evaluating elearning Who am I… ? Professor at Portuguese Catholic University Pedagogical coordinator of the elearning master degree “ICT and Education”related to Teachers Training [email_address] m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

3. Initial question… what really eLearning is ? Distance Education? knowledge mangement? Mobile learning ? Face to face learning? m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

4. eLearning is the use of Internet technologies to distribute a set of solutions that allow the increasing of knowledge and abilities, either at individual level, either at institutional level. Rosenberg (2007) Formal training Knowledge management m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

5. evaluating? Why? Are the learning goals achieved ? Are there areas where improvement is possible? Can we be obtain a better position in the market? m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

6. Evaluating what? Micro vision Macro vision For a component of a course For a specific course Curriculum/program For a organization /SME For a particular activity sector On a social level m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

7. O que avaliar ? m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto Evaluating what? Micro vision Macro vision For a component of a course For a specific course Curriculum/program For a organization /SME For a particular activity sector On a social level

8. evaluating When ? Before training During training delivery After Training Are there conditions for sucess ? Are the training strategy And materials adequate ? The impact of training in the organization m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

9. Paulsen & Vieira (2006) had indicated some criteria to judge the global quality of an elearning program. Credibility of the training institution Guarantee of quality criteria ( standards) Previous information and counseling Costs of the course Support to e-student (technical and counseling) Individual preferences ( learning styles) m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

10. The Kirkpatrick framework levels m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto 1 - satisfaction the students had liked the training? - the drop out rate was low? 2 - learning what had they learned? - had the aims of learning been reached? 3 - abilities - Are the students applying the new abilities in its workplace? 4 - results Is the organization improving its outcomes by the training of its workforce? - how much earns the institution with the training of its collaborators?

11. student’s satisfaction level 1 Coordination of the course Contents related to personal objectives Technology (LMS …) Tutorial model Pedagogical model / training design evaluation criteria ? m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto - - -

12. Evaluation tools Questionnaires Interviews (singles or in focus group) Forums m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto student’s satisfaction level 1

13. m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto Satisfaction questionnaire (example)

14. Results of satisfaction quiz– Research Methodologies in Education - 2007/2008 m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

15. Learning outcomes Level 2 m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto D iagnosis Lists of confidence minimum technical characteristics Scales of abilities formal abilities (academic or professional qualification) specific abilities in some areas (previous linguistics abilities and digital literacy)

16. Mestrado em Ciências da Educação, especialização em Informática Educacional CONDIÇÕES DE ACESSO Edição 2007/2009 a) Habilitações : Pode candidatar-se ao curso de mestrado em Ciências da Educação (Especialização em Informática educacional (e-learning)) os candidatos com o grau de licenciado. b) Competências de utilização ao nível da informática: O curso de informática educacional tem como pressuposto que os alunos detenham competências médias no domínio da utilização dos softwares de utilização corrente (processador de texto, folha de cálculo, apresentações) e da utilização da Internet e da Web. Deste modo, a frequência do curso está dependente da existência dessas mesmas competências. Como a UCP não fará verificação dos pré-requisitos, o candidato será responsável pelas suas capacidades no domínio das ferramentas informáticas. No anexo I define-se o perfil desejável de competências informáticas de entrada no Mestrado. c) Competências linguísticas Embora a língua oficial do Mestrado seja o Português, os candidatos deverão possuir competências de leitura e interpretação de textos escritos em Inglês, dado que alguns materiais pedagógicos distribuídos ou a pesquisar pelos alunos poderão estar nesse idioma. d) Requisitos tecnológicos Para a frequência do curso, os alunos devem dispor, em permanência, de um computador multimédia e acesso à Internet, preferencialmente em banda larga. Anexo I Pré-requisitos de competências informáticas para ac esso ao Mestrado em Informática Educacional Consideram-se pré-requisitos informáticos gerais: Academic qualification Digital literacy Adequate knowledge foreign language Hardware requirements m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto Scale of abilities

17. m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto Learning outcomes Level 2 formative self evaluation hetero-evaluation peers tutor/professor/system sumative self evaluation hetero-evaluation peers tutor/professor/system

18. online self evaluation tests assignments Project work Fóruns participation participation in sincronous sessions m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto Learning outcomes Level 2 Setting of a minimum number of participation Setting of evaluation criteria of each message Interaction Confrontation Lidership Different approaches … Number of messages posted per day

19. Face to face evaluation (quality recognition) Testes online tests Múltiple choice true - false fill in gaps de correspondência … written testes escritos projects discussion Labs work Chats’ participation m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto Learning outcomes Level 2

20. evaluating learning outcomes on-line individual works Particpation in chats forum participation on-line working groups face to face evaluation - tests / projects/ … m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

21. % of different evaluation components in a single Curricular Unit m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto evaluation final evaluation (%) Face to face (teste ou work debate) 50 a 60 Assignments / required along the training period 15 a 20 Colaborative work - written documents sumbited to tutor. 20 a 30 Participation in online sincronous activities (according teacher strategies) 5

22. evaluation of abilities level3 external evaluation department director opinion self evaluation Less complaints Less acidents Raise off sales … m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

23. It is possible to evaluate at this level , if The training is a consequence of clear needs The learner has strong motivation There are logistics conditions allowing th evidence of new skills m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto evaluation of abilities level3

24. Results inside organisation Nível 4 Level 4 measures the results of learning at the institutional level The training contributes to the objectives of the business ? The training responds efficiently to the needs identified ? What level of investment in training is recovered ? ROI m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

25. Evaluating e-learning Evaluating learning Evaluating communication tools Conclusion : Knowledge Skills m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto

26. Thanks for your attention! [email_address] m- ICTE2009 Conference, Lisboa, 22—24 April José Reis Lagarto