Outline of the presentation Introduction Emergo: Design and Implementation Results and Discussion WCCE 2009 – World Conference on Computers in Education Bento Golçalves-RS, Brazil, July 27-31, 2009, www.wcce2009.org Emergo: Academic Performance Assessment and Planning with a Data Mart Vinícius Medina Kern 1 , Luciana Martins Saraiva 2 , Eugênio Rubens Cardoso Braz 3 1 UFSC, Grad. Program in Knowledge Eng. and Management, Brazil – kern.ispeople.org , [email_address] 2 UNIVALI at Biguaçu, Psychology, Brasil – [email_address] 3 UFS, Computing, Brasil – [email_address] Concluding Remarks Appendix: A ggregate results Answer profiles

Outline of the presentation

Introduction

Emergo: Design and Implementation

Results and Discussion

Introduction National assessment of cognitive learning in Brasil PISA/OECD (15 year-olds): below 3 sd from OECD average Higher education : ENC (National Courses Exam), 1996-2003 Lots of irrationalist opposition Some objective criticism (e.g., Landeira-Fernandez & Primi (2002) showed that performance of graduates was due to preparedness at entrance , not graduation). ENADE ( National student’s performance Exam, since 2004 ) ENC/ENADE exams available High quality questions (recognized by both sects), free download This prompted us to undertake EMERGO ( latin: to make emerge )

National assessment of cognitive learning in Brasil

PISA/OECD (15 year-olds): below 3 sd from OECD average

Higher education :

ENC (National Courses Exam), 1996-2003

Lots of irrationalist opposition

Some objective criticism (e.g., Landeira-Fernandez & Primi (2002) showed that performance of graduates was due to preparedness at entrance , not graduation).

ENADE ( National student’s performance Exam, since 2004 )

ENC/ENADE exams available

High quality questions (recognized by both sects), free download

This prompted us to undertake EMERGO ( latin: to make emerge )

Emergo: Design and Implementation Emergo Project @ Psychology (undergrad), Univali Biguaçu-SC: Aim: to support course assessment and planning Way: assessment of all students in 40 5-choice questions (ENC) 2 rounds (2002 and 2003) with meetings with students and faculty, pre- and post-exams No student boycott, but irrational resistance to identification It was very clear that publication of individual results would not be done Univali publishes course grades on the walls with no resistance Assumption: validity of ENC questions

Emergo Project @ Psychology (undergrad), Univali Biguaçu-SC:

Aim: to support course assessment and planning

Way: assessment of all students in 40 5-choice questions (ENC)

2 rounds (2002 and 2003) with meetings with students and faculty, pre- and post-exams

No student boycott, but irrational resistance to identification

It was very clear that publication of individual results would not be done

Univali publishes course grades on the walls with no resistance

Assumption: validity of ENC questions

Emergo: Design and Implementation Supporting data mart Dimensions : EmergoRound (date), Person, SemesterEnrolled (1-10), Question, Topic (subject area), Course (semestral). Facts : expressions of the results achieved by students, crossing several dimensions. Supporting software interface… (next)

Supporting data mart

Dimensions : EmergoRound (date), Person, SemesterEnrolled (1-10), Question, Topic (subject area), Course (semestral).

Facts : expressions of the results achieved by students, crossing several dimensions.

Supporting software interface… (next)

Emergo: Design and Implementation Example question (#6, shorter question from Emergo 2003 = #1 ENC 2002) : According to Jean Piaget, the epistemic subject is (A) the subject himself, independent of the object. (B) determined by the object along his development. (C) what is common to all subjects of knowledge. (D) identical to the kantian transcendental subject. (E) each subject of knowledge, taken individually.

Example question (#6, shorter question from Emergo 2003 = #1 ENC 2002) :

According to Jean Piaget, the epistemic subject is

(A) the subject himself, independent of the object.

(B) determined by the object along his development.

(C) what is common to all subjects of knowledge.

(D) identical to the kantian transcendental subject.

(E) each subject of knowledge, taken individually.

Results and Discussion Emergo respondents and correct answers by semester enrolled in both rounds No direct 2002-2003 comparison (different students, different questions and levels of difficulty) General trend: growing performance along semesters (expected) Patterns for the evolution of correct answers for questions across semester enrolled – Growing, Decreasing, Peak, Constant, and Other (several unexpected)

Emergo respondents and correct answers by semester enrolled in both rounds

No direct 2002-2003 comparison (different students, different questions and levels of difficulty)

General trend: growing performance along semesters (expected)

Patterns for the evolution of correct answers for questions across semester enrolled – Growing, Decreasing, Peak, Constant, and Other (several unexpected)

Results and Discussion Fast feedback: to students (individual) and to students+faculty (aggregate) in collective sessions Call for discussion and reflection (Why...?) Not so much for conclusions at this stage Figures for 2003: number of questions, plausible hypothesis for that performance profile... Growing : 9, PH: Students are progressively building their knowledge along the semesters (in this topic). Peak : 6: PH: It is a memorization question. Decreasing : 7, PH: We are instilling confusion in our students’ minds (in this topic). Constant : 10, PH: Our instruction has no impact whatsoever (in this topic). Other : 7, PH: (None).

Fast feedback: to students (individual) and to students+faculty (aggregate) in collective sessions

Call for discussion and reflection (Why...?)

Not so much for conclusions at this stage

Figures for 2003: number of questions, plausible hypothesis for that performance profile...

Growing : 9, PH: Students are progressively building their knowledge along the semesters (in this topic).

Peak : 6: PH: It is a memorization question.

Decreasing : 7, PH: We are instilling confusion in our students’ minds (in this topic).

Constant : 10, PH: Our instruction has no impact whatsoever (in this topic).

Other : 7, PH: (None).

Results and Discussion In a systemic (or rather systemist--Bunge) view the emergence (or submergence) of a system or a system’s property (in our case, collective excellence in cognitive performance ) depends on the bonds between the system’s components . By challenging the students’ cognitive capacity and then giving them feedback and provoking discussion and reflection, we’ve empowered them and strengthened the communication bonds that could help excellence in cognitive learning to emerge.

In a systemic (or rather systemist--Bunge) view the emergence (or submergence) of a system or a system’s property (in our case, collective excellence in cognitive performance ) depends on the bonds between the system’s components .

By challenging the students’ cognitive capacity and then giving them feedback and provoking discussion and reflection, we’ve empowered them and strengthened the communication bonds that could help excellence in cognitive learning to emerge.

Results and Discussion No conclusion possible, but promising results in actual ENC (not so much in ENADE):

No conclusion possible, but promising results in actual ENC (not so much in ENADE):

Concluding Remarks Exploratory project, aimed at fostering curiosity and reflection ( conclusions are steps away) Emergo discontinued for institutional disinterest (question out of the scope of this paper/presentation) Proposition: Emergo strengthened the bonds necessary for an emerging academic excellence (plausible, but more longitudinal data is necessary to allow for a conclusion). Next ENADE for Psychology: November 8, 2009. Computers manipulated data, people did the thinking. Emergo: potential as part of a broader scheme for measuring effectiveness of undergraduate education, e.g., as proposed by Dunn et al. (2007).

Exploratory project, aimed at fostering curiosity and reflection ( conclusions are steps away)

Emergo discontinued for institutional disinterest (question out of the scope of this paper/presentation)

Proposition: Emergo strengthened the bonds necessary for an emerging academic excellence (plausible, but more longitudinal data is necessary to allow for a conclusion).

Next ENADE for Psychology: November 8, 2009.

Computers manipulated data, people did the thinking.

Emergo: potential as part of a broader scheme for measuring effectiveness of undergraduate education, e.g., as proposed by Dunn et al. (2007).

Araújo, L., Ristoff, D.: O provão deve ser extinto? Sim [Should the ENC be terminated? Yes]. Jornal da Ciência E-mail 2359, Sept. 8th, 2003 (originally published by the newspaper Folha de São Paulo ), http://www.jornaldaciencia.org.br/Detalhe.jsp?id=12578 Bunge, M.: Mechanism and explanation. Phil. Soc. Sci. 27(4), 410--465 (1997) Dunn, D. S., McCarthy, M. A., Baker, S., Halonen, J. S., Hill IV, G. W.: Quality benchmarks in undergraduate psychology programs. Am. Psychol. 62(7), 650--670 (2007) Kimball, R., Reeves, L., Ross, M., Thornthwaite, W.: The data warehouse lifecycle toolkit : expert methods for designing, developing and deploying data warehouses. John Wiley & Sons, New York (1998) Landeira-Fernandez J, Primi R. Performance comparison between freshmen and seniors in ENC Psychology 2000 [in PT]. Psicol. Reflex. Crít. 15(1), 219--234 (2002) Schwartzman, S.: The national assessment of courses in Brazil . In: Public Policy for Academic Quality (PPAQ). Research Program. University of North Carolina, Chapel Hill (2004) References

Appendix: General results and answer profiles (examples) Aggregate results and examples of questions with profiles Growing, Descreasing, Constant, Peak, and Other from Emergo 2003. We start with general averages for each question : Cancelled…..…………………………. Below chance (7/39 or 18% of the questions)

Appendix: Averages according to SemesterEnrolled General results show progress along the semesters, as one could expect… 9 th semester is the champ, with close followers from semesters 10 th , 6 th , and 8 th .

Appendix... But look what happens with the 14 students of the 9 th semester (the champs!): Cancelled…..…………………………. Below chance (7/39 or 18% of the questions plus 5 questions near to the chance rate – 21%) (Yes; the best performers are at a loss or worse in 40% of the questions.)

Appendix: Examples of answer profiles: GROWING (7=e, ENC06-2002) Plausible hypothesis for this kind of answer profile : Students are progressively building their knowledge along the semesters. A 10 years old boy is sent for psychological treatment because of various symptoms: he would not touch anything, so that his mother had to dress him and feed him. If the mother touched some object with one hand, it would have to be placed in the same place and, after that, that same operation would have to be executed with the other hand. Before the disease he was a good student and learned easily. The pathology and the underlying defense mechanism are : (A) Phobia and Refusal . (B) Obsessive neurosis and Rejection . (C) Psychosis and Repression . (D) Psychosis and foreclusion of the Name-of-the-Father. (E) Obsessive neurosis and Repression .

Appendix: Examples of answer profiles: PEAK (5=d, ENC09-2002) Plausible hypothesis: It is a memorization question. The “mirror stage", according to Lacan, is a psychic operation in which predominates the register of the (A) Real, referring to a “remaining part” impossible to transmit. (B) Imaginary, designating the place of the Symbolic that determinates the subject . (C) Symbolic, according to which the signifying is the very essence of the symbolic function . (D) Imaginary, referring to a relation of indistinction with the other . (E) Real, designating the subject’s object of desire, one that escapes symbolization .

The “mirror stage", according to Lacan, is a psychic operation in which predominates the register of the

(A) Real, referring to a “remaining part” impossible to transmit.

(B) Imaginary, designating the place of the Symbolic that determinates the subject .

(C) Symbolic, according to which the signifying is the very essence of the symbolic function .

(D) Imaginary, referring to a relation of indistinction with the other .

(E) Real, designating the subject’s object of desire, one that escapes symbolization .

Appendix: Examples of answer profiles: DECREASING (40=c, ENC40-2002) Plausible hypothesis: We are instilling confusion in our students’ minds. Recent worldwide events lead us to ask again the reason for war making. Consider the statements below . I. If we rebel against war, it is because it is the most obvious opposition to the psychic attitude inculcated in us in the civilizatory process . II. As the shade is the core of the personal unconscious, nucleus of the repressed material, it is necessary to get information on its qualities and intentions. The conflicts can be solved only if we can bear it, and suffering is inevitable . III. Destructivity that is bound to character is nothing more than anger from frustration in general and from the refusal of sexual gratification in particular . Since behavior is due to selective reinforcing, then the engagement of so many men, presenting a belicist repertoire, is a function of the environment that failed in the control of stimuli . The statements correspond, respectively, to (A) Rogers, Jung, Reich, and Skinner. (B) Jung, Freud, Reich, and Rogers. (C) Freud, Jung, Reich, and Skinner. (D) Reich, Freud, Skinner, and Jung. (E) Rogers, Jung, Freud, and Skinner.

Recent worldwide events lead us to ask again the reason for war making. Consider the statements below .

I. If we rebel against war, it is because it is the most obvious opposition to the psychic attitude inculcated in us in the civilizatory process .

II. As the shade is the core of the personal unconscious, nucleus of the repressed material, it is necessary to get information on its qualities and intentions. The conflicts can be solved only if we can bear it, and suffering is inevitable .

III. Destructivity that is bound to character is nothing more than anger from frustration in general and from the refusal of sexual gratification in particular .

Since behavior is due to selective reinforcing, then the engagement of so many men, presenting a belicist repertoire, is a function of the environment that failed in the control of stimuli .

Appendix: Examples of answer profiles: CONSTANT (23=b, ENC21-2002) Plausible hypothesis: Our instruction has no impact whatsoever (!?). A psychologist works in a clinic for treatment of drug dependants and provides psychotherapic attendance to Marcos, Fernanda, and Tânia. In interviews with the psychologist, the patients used the following expressions to describe some effects of the drug each of them used over their behaviour : –– Marcos: sensations of alertness, euphoria, cardiac acceleration. –– Fernanda: diminution of anxiety and tension, sleepiness. –– Tânia: visions, hallucinations, time perception distortions . Psychoactive drugs that typically produce effects corresponding to those described by Marcos, Fernanda and Tânia are, respectively : (A) marijuana, caffeine and morphine . (B) amphetamine, barbiturate, and LSD . (C) barbiturate, LSD, and alcohol . (D) amphetamine, alcohol, and barbiturate . (E) opioid, caffeine, and amphetamine .

Appendix: Examples of answer profiles: OTHER (6=c, ENC01-2002) Plausible hypothesis: What!? According to Jean Piaget, the epistemic subject is (A) the subject himself, independent of the object. (B) determined by the object along his development. (C) what is common to all subjects of knowledge. (D) identical to the kantian transcendental subject. (E) each subject of knowledge, taken individually.

Thank you! Questions? Contact author: Vinícius Medina Kern, kern.ispeople.org (see menu ‘Presentations…’ for these slides) Researcher at Instituto Stela , www.stela.org.br Professor at EGC/UFSC , www.egc.ufsc.br (Both in Florianópolis-SC, Santa Catarina island)