A multiagent system for formative assessment support in Learning
                     Management Systems

monitor, to analyze and to advise the                 researches on education [Hadji 2001]
learning process. This scenario...
To consult other learners’ learning
      profiles, making possible the knowledge                      Course Monitoring A...
type, where m is the number of                       addition of a new Activity Monitoring
      monitored tools in the LM...
Protégé-20002 and Algernon3 tools that are                       implementation, and on the integration of
presented at se...
simplifying and significantly speeding up                    completely independent of the agents Java
the process.       ...
communication tools of this LMS, the                  possible to easily extend the system to
Discussion Forums and the Po...
utilization of Jade, Protégé-2000 and                       (multiple choice questions, true or false
Algernon technologie...
Artificial, vol. 8. Campinas: SBC. 53-102,
     July, 2003.
Upcoming SlideShare
Loading in …5

A multiagent system for formative assessment support in Learning Management Systems


Published on

In: Anais do I Workshop Tidia, São Paulo, novembro, 2004 Em inglês , 9 páginas.

Autores: Heloisa Vieira da Rocha, Joice Lee Otsuka, Vitor Seixas Bernardes

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A multiagent system for formative assessment support in Learning Management Systems

  1. 1. A multiagent system for formative assessment support in Learning Management Systems Joice Lee Otsuka, Vítor Sexto Bernardes, Heloísa Vieira da Rocha Institute of Computing – State University of Campinas Nucleus of Informatics Applied to Education – State University of Campinas P.O. Box 6176 13083-970 - Campinas, SP – Brazil Phone +55 19 3788-5842 {joice,vitor.bernardes,heloisa}@ic.unicamp.br Abstract Formative Assessment has been adopted as an alternative to the traditional model based on cumulative exams and has revealed itself as a crucial importance to distance education, by helping the learners’ behavior perception and the problems identification and making possible more effective advice in the time of learning process, even at distance. Nevertheless, the educators need to take extra steps to monitor learners’ participation and to regulate the learning processes in distance courses, resulting in the educators’ overload and, consequently, the formative assessment high costs. The possibility of saving all interactions in computer-based courses for later analysis can be explored to provide formative support. This paper presents the main results of a doctoral research that proposes a multiagent system (MAS) to support formative assessment in Learning Management Systems. The proposed MAS architecture definition process and some considerations about the implementation of a prototype that has been integrated into TelEduc LMS are discussed. Keywords: multiagent systems, distance education, formative assessment. online assessment presents some intrinsic 1 Introduction difficulties, such as the absence of the face- The formative assessment has been used as to-face interactions feedback, the lack of an alternative to the traditional assessment instructor control over the assessment and based on cumulative exams, and allows the the authentication problem (who is continuous monitoring and orientation of the performing the assessment?). The learning process. This assessment approach informative and regulatory characteristics of presents informative and regulatory roles, this assessment approach have revealed providing information to the two main actors themselves as a crucial importance to of the teaching and learning process: the distance education, by helping the learners’ educator, that will be informed about the behavior perception and the problems real effects of his/her pedagogical actions, identification, making possible more and will be able to regulate them; and the effective advice in the time of learning learner, that will have the opportunity to be process, even at distance. conscious of his/her difficulties and, Nevertheless, nowadays most Learning possibly, to recognize and to review his/her Management Systems (LMSs) do not own mistakes [Hadji 2001]. present features specially designed to help In distance education, the formative this assessment approach. Thus, the assessment is even more important since the educators need to take extra steps to TIDIA First Workshop – E-Learning Project November 2004
  2. 2. monitor, to analyze and to advise the researches on education [Hadji 2001] learning process. This scenario has been the [Perrenoud 1998], the research group’s main problem of formative assessment, face- practical experience in computer-mediated to-face as well as at distance. distance courses [Otsuka & Rocha 2002] [Ferreira, Otsuka & Rocha 2003] and the This paper presents the main results of analysis of other researches and experiences a doctoral research that proposes a on learning assessment in distance education multiagent system (MAS) to support [Thorpe 1998] [Hopper 1998] [Nelson 1998] formative assessment in LMSs, aiming to [Cerny 2001] [Gomez 2001]. reduce the instruct overload and consequently, the formative assessment The requirements of two main actors in costs. formative assessment process, educators and learners, were identified. Thus, the The paper is structured in five sections. requirements to support the educators’ Section 2 shows the main phases of the needs are: definition process that resulted in the proposed MAS architecture. Section 3 To receive, continuously, relevant and presents the agents knowledge base reliable information about learner’s definition and section 4 presents some participation at each proposed activity, considerations about the implementation of throughout the course; a prototype of the proposed architecture that To receive the consolidation of learner’s has been implemented and integrated into participation at all activities proposed, TelEduc1 [Rocha 2002, Rocha et al. 2002], a making possible a global monitoring of LMS that has been developed at State learners’ participation and the University of Campinas since 1997, and identification of the learners’ which is available as free software under the participation and learning styles; open-source GNU Public License. At last, section 5 presents some final considerations To have mechanisms to help the about this work. analysis and interpretation of the above information, just in time, with coherence 2 The Multiagent System and impartiality; Architecture definition To receive help on the assessment of The definition process of a MAS activities, by means of the preanalysis of architecture to support formative assessment learners’ participation based on criteria in learning management systems was defined in the activities plan, and the composed by the following main processes: elaboration of a detailed and consistent the requirements specification and the feedback about these assessments; process of analysis and design based on To have flexibility to define the Gaia methodology [Wooldridge, Jennings & assessment criteria of each activity. Kinny 2000]. The next subsections present the main results obtained in these phases and The requirements to support learners’ the resulted proposed architecture. needs are: To receive, continuously, monitoring 2.1 Requirements Specification reports and their learning profiles, that The first step in the definition of the provide a feedback of their participation proposed MAS architecture was the main in each proposed activities and in the requirements specification. These main whole course, helping them to be requirements have been obtained through conscious of their difficulties and progresses; 1 http://teleduc.nied.unicamp.br TIDIA First Workshop – E-Learning Project November 2004
  3. 3. To consult other learners’ learning profiles, making possible the knowledge Course Monitoring Agent 1.. n of the group learning profile, what might lead to improvements in the collaborative learning. collect and inform consolidate The next step was the specification of update participation global activities indices analysis agents “roles”, that is, the behaviors that need to be implemented by the MAS to support the requirements specified earlier. The roles specification was based on Gaia * 1..m methodology, and it is presented in the next Activities Integrator Agent subsection. Monitoring Agent 2.2 Analysis and design based on Gaia methodology notify new update analyze events with request for The Gaia methodology [Wooldridge, activities indices activities monitoring Jennings & Kinny 2000] was adopted because it is simple and efficient, and Figure 1 – The Agent Model of the proposed MAS: the provides effective support to the agent types are represented by rectangle and the agent roles specification of agents behavior (Roles are represented by rounded rectangles. The agents cardinality is represented by: * (0 or more), 1..n (1 to n) Model), the inter-role protocols definitions (Interaction Model), the agent types definition (Agent Model), as well as the Course Monitoring Agent: this agent identification of communication links type is responsible for ongoing between agent types (Acquaintance Model). monitoring of learners’ participation At analysis phase the Roles Model and throughout a course. It acts accessing the Interaction Model were defined. The the course database and collecting Roles Model specifies a set of agent indices from learners’ participation in behaviors needed to support the each activity proposed (for example, requirements presented at subsection 2.1. synchronous and asynchronous The Interaction Model identifies the discussion, projects, reports and so on). communication protocols that will be The collected indices are sent to the specified to make possible the inter-role appropriate Activity Monitoring Agent interactions. responsible for the analysis of activities. At design phase the Agent Model and This agent type is also responsible for a the Acquaintance Model were defined. The global analysis of learners’ participation Agent Model defines the agent types, that is in one course, creating the learning best thought of as a set of agent roles. To profiles. The MAS can have 1 to n improve the system performance, the roles agents of this type, where n is the were grouped trying to obtain a minimal number of current courses in the LMS; number of agent types. This process resulted Activities Monitoring Agent: each in an Agent Model composed by 3 agent agent of this type is responsible for the types (Figure 1): analysis of learners’ participation indices in every activity developed in one specific tool of the LMS (such as Discussion Forums, Chat, Portfolio, Mail, Bulletin Board and so on). The MAS can have 1 to m agents of this TIDIA First Workshop – E-Learning Project November 2004
  4. 4. type, where m is the number of addition of a new Activity Monitoring monitored tools in the LMS; Agent, responsible for the analysis of learners’ participation in the context of the Integrator Agent: this agent type new tool. The Course Monitoring Agent’s provides the MAS interface with the roles may be slightly changed to be able to educator: it receives events from LMS deal with the new tool monitoring. and informs them to the appropriated Nevertheless, they are precise changes, that agents in the MAS. A new Integrator will not interfere in the previously defined Agent should be created when one roles (see discussion on section 4.3). educator logs in the LMS and should stay “alive” until the end of the session. The Acquaintance Model was obtained Multiagent System Architecture directly from the Agent Model (Figure 2), LMS and identifies the communication links Monitoring criteries <session x> Monitoring request <session 2> among the three different agent types with <session 1> the purpose of identify any potential Integrator Agent communication bottlenecks, that could cause Assessment Agent Communication Chanel problems at run-time [Wooldridge, Jennings Discussion Forum Activities & Kinny 2000]. Discussions Forum Monitoring Agent Chat Chat Activities Monitoring Agent Course Monitoring 1..n * Agent Tool m Tool m Activities Monitoring Agent Integrator Agent Course 1 Monitoring Agent Course1 Activities 1..m Course 2 Monitoring Agent Monitoring Agent Course2 Figure 2 – The Acquaintance Model of the proposed MAS Course n Monitoring Agent Course N 2.3 The multiagent system architecture n The MAS architecture has been obtained Figure 3 – The proposed MAS integrated with a LMS from the two process presented in the previous subsections. Figure 3 shows the proposed architecture integrated into a LMS 3 The knowledge base definition that contains n courses and m tools to be monitored. Thus, to support this LMS the The roles of Course Monitoring Agents and proposed MAS architecture could include n Activities Monitoring Agents involve Course Monitoring Agents and m Activity information analysis and new knowledge Monitoring Agents. At each time an creation. Thus, the definition of these educator logs in the LMS, one Integrator agents’ knowledge bases was needed. A Agent is created to provide the interface knowledge base has been defined for each between the MAS and the educator in the Activity Monitoring Agent to make possible current session. the analysis of learners’ participation in activities developed in the monitored tool. The proposed architecture seems to be easily extended to support the monitoring of The knowledge representation has been new tools incorporated into the LMS. This is done by means of frames and production achieved by the implementation and rules. This process has been supported by TIDIA First Workshop – E-Learning Project November 2004
  5. 5. Protégé-20002 and Algernon3 tools that are implementation, and on the integration of presented at section 4.1.2. An initial set of the proposed MAS into TelEduc. rules has been defined with the participation of researchers of Nucleus of Informatics 4.1 Technology investigation Applied to Education (Nied) at State Some tools have been analyzed in order to University of Campinas, based on group’s make easier the implementation of the experiences with formative assessment in proposed MAS, namely, tools to support the distance education. development of multiagent systems, tools to At the time of an activity creation, the support the construction of the knowledge educator selects some criteria that are going base and inference systems. In the to reflect the rules to be considered on this implementation of the prototype, the activity monitoring. The educator also can following tools have been adopted: the Jade define some reference indices that will be MAS development framework4, the Protégé- used as parameters of rules at the execution 2000 knowledge base editor5 and the time, providing some flexibility to these Algernon rule-based inference system6, all rules. of them are briefly described in the following subsections. Thus, when new monitoring indices are collected through a Discussion Forums’ 4.1.1 Jade Activity and instantiated into the Discussion Forums’ Activities Monitoring Agent’ JADE is free software and is distributed by knowledge base, these rules are executed TILAB, as open source software under the and new knowledge can be produced. terms of the LGPL (Lesser General Public License Version 2). It is implemented in The knowledge base definition is at Java7 and aims to make easier the initial phase and may be refined with users’ development of multiagent systems by suggestions. In order to make this providing a comprehensive set of services refinement possible, these knowledge bases and an API to support the development of modeling have been focused on an easily agents in compliance with the FIPA extended and not source-code dependent specifications [Gluz & Viccari 2003]. solution. In order to make easier the new rules creation, the design of an interface that The JADE framework provides an API makes possible the creation of new rules with several classes that represent common directly by final users is proposed as future entities in MAS, supporting the definition of work. agents, agents' behaviors, FIPA-ACL- compliant messages, interaction protocols, and so on [Bellifemine et al. 2003]. In the 4 Implementation Considerations time of vocabulary and semantics definition A prototype of the proposed MAS has been of the agents’ communication content, a implemented and it is going to be integrated solution based on Java objects usage as the into the TelEduc LMS, composing the content of the messages exchanged among solution for formative assessment support the agents has been adopted, as proposed in proposed for this system in [Otsuka et al. [Vaucher & Ncho 2004]. The JADE 2003]. The following subsections present utilization has been essential to the agents some considerations on the technological implementation in the proposed MAS, support that has been explored in the 4 http://sharon.cselt.it/projects/jade/ 5 http://protege.stanford.edu/ 2 6 http://protege.stanford.edu/ http://smi.stanford.edu/people/hewett/research/ai/ 3 http://smi.stanford.edu/people/hewett/research/ai/ algernon/ 7 algernon/ http://java.sun.com/ TIDIA First Workshop – E-Learning Project November 2004
  6. 6. simplifying and significantly speeding up completely independent of the agents Java the process. code, thus making the maintenance of knowledge bases easier. 4.1.2 Protégé–2000 and Algernon 4.2 Technology integration Protégé-2000 is an extensible, platform- independent ontology and knowledge-base The three different technologies discussed editor, developed by Stanford Medical are Java-based and offer APIs that make Informatics at the Stanford University easier the integration among Jade, Protégé- School of Medicine. Protégé is available as 2000 and Algernon. As seen before, building free software under the open-source Mozilla an agent with the Jade framework is realized Public License [Gennari et al. 2002]. by extending classes already defined in the framework. The manipulation of one agent’s Protégé provides a simple and powerful knowledge base is possible by importing graphical interface, integrating the ontology classes from Algernon's APIs. definition and creation tasks and the insertion of instances into knowledge base The server runs by starting a Jade task. Another very interesting feature is the platform, which is the environment where Protégé-2000's extensibility, since it is a the agents will live in, and then creating platform that can be extended with graphical Activity and Course Monitoring agents. widgets such as tables, diagrams and These agents' knowledge bases edited with animation components. Also, due to its the Protégé-2000 GUI are stored in files that plugin architecture, it is possible to can be accessed using Algernon's API in the customize and extend Protégé's features agents code. This API allows creating an according to the project specific needs instance of the Algernon inference machine, [Musen et al. 2000]. adding a knowledge base to the list of KBs this Algernon instance accesses, and In the prototype implementation, the querying and inserting data into them. Algernon Tab plugin8 has been used, which is responsible for the integration of Protégé Thus, after the definition of frames, and Algernon, an inference system that slots and relations between frames, all using provides backward and forward chaining Protégé-2000's graphical interface, the rules to infer data in knowledge bases. This insertion of instances into an agent's plugin makes possible to execute Algernon knowledge base is done using Algernon's commands directly in the Protégé-2000 API methods. When these data are inserted GUI, and also operates directly on the into the knowledge base, the previously latter’s knowledge bases, requiring no defined rules (using the Algernon Tab mapping between different memory areas. plugin) are automatically triggered in the This direct manipulation of Protégé-2000 Algernon inference machine to process the data is very interesting, because it provides new entries and to try to recognize any some performance gain [Hewett 2003]. pattern. Whenever the inference machine Algernon is available as free software under discovers something, a new instance is the Mozilla Public License as well. created reporting this inferred situation, which can be queried later by the The adoption of Protégé-2000 and community agents. Algernon in the prototype implementation has made possible using Protégé-2000's 4.3 Integration of the proposed MAS graphical interface directly to create into TelEduc ontologies and knowledge-base rules. This is The prototype integration into the TelEduc LMS has started by focusing on the learners' 8 http://algernon-j.sourceforge.net/doc/algernon- participation monitoring in two of the protege.html TIDIA First Workshop – E-Learning Project November 2004
  7. 7. communication tools of this LMS, the possible to easily extend the system to Discussion Forums and the Portfolio. This monitor new tools, since the core of the decision is explained by the relevance of system will already be implemented and these two tools in the context of formative most of the existing classes will possibly be assessment. These tools, since they are used and/or extended to implement new asynchronous, allow deeper reflection by the agents. Thus, the extension of the system to learner at the moment of his/her monitor a new tool is expected to be done in participation, resulting in more relevant and 3 main steps: consistent contribution. The asynchronous Creation of new methods for the Course characteristic also helps the educator’s Monitoring Agent (to collect action in the monitoring and regulating participation indices related to the new learners' participation throughout the tool); activities development, although this process Creation of new methods for the is generally too expensive due to the huge Integrator Agent (to receive requests for volume of information to be analyzed. reports about the new tool monitoring The Discussion Forums tool interface and to present the analysis results); has been adapted to make possible select the Implementation and instantiation of a degree of relevance (not relevant, low new Activity Monitoring Agent and relevance, medium relevance, relevant, high creation of the knowledge base of this relevance) of a message. Thus, it will be new agent to analyze the learners’ possible to record the participation as well participation indices in activities as to analyze the records of the learners’ developed with the new tool. participation assessments. The integration of the proposed MAS 5 Final Considerations into TelEduc has been done through the Integrator agent that runs at an user's This paper presented the definition process browser's applet and is responsible for the of a MAS architecture to support formative communication with server-side agents assessment in learning management notifying relevant events at LMS. The systems. The main goals of this process was Integrator agent is activated when necessary to propose one solution that: (1) tries to by invoking one of its methods, which is minimize the number of agents, avoiding done by a Javascript function call in the one fast explosion of this number when LMS code, that passes on to the applet, and many courses and tools must be monitored; thus to the Integrator agent in it, whatever (2) is extensible, making possible to easily information is needed. The Integrator agent, extend the system to monitor new tools; (3) once receives these data, must pass it to the is generic, making possible to easily adapt it server-side agents (Activity and Course to other LMSs. Monitoring agents), by sending an ACL- The knowledge base definition was message through HTTP using Jade's HTTP concerned with: (1) the flexibility, making MTP (Message Transport Protocol). Once possible the rules adaptation according to the server-side agents receive the data, they the educator’s needs; (2) the easy rule set process it according to predefined protocols, extension, by disconnecting the rules’ and the communication between the edition and system implementation. Integrator agent and the server-side agents is done for that given piece of information that The Gaia methodology employment server-side agents needed to know. was fundamental in the analysis and design phases of the MAS architecture, helping After the implementation of the MAS mainly in the roles and agent types for the monitoring these two tools, it will be definition. In the implementation phase, the TIDIA First Workshop – E-Learning Project November 2004
  8. 8. utilization of Jade, Protégé-2000 and (multiple choice questions, true or false Algernon technologies was fundamental and subjective questions) to supply because they significantly made easier and TelEduc users’ needs [Freitas 2002]. sped up this process. The next steps of this The proposed MAS architecture should project involve: (1) to finish the agents’ be appropriately extended to support behaviors implementation; (2) to refine the these tools monitoring. knowledge bases of Discussion Forums and This model explores the ease of Portfolio Activities Monitoring Agents; (3) recording anything that happens in a to design the MAS user interface, that computer-mediated distance course, in order challenges to present a huge volume of to provide formative assessment support, by information and to maintain the consistence means of agent technology. with the current TelEduc user interface. The proposed MAS is part of the 6 References formative assessment support model Bellifemine, F. ; Caire, G.; Trucco, T.; Rimassa, G proposed to TelEduc in [Otsuka 2002b, (2003) . JADE Programmer’s GUIDE. Otsuka et al. 2003]. The other researches Available at: that compose this model are briefly http://sharon.cselt.it/projects/jade/ described below: Cerny, R.Z. (2001). Uma reflexão sobre a One proposes and implements the avaliação formativa na educação a Assessments Tool, aiming to make distância. UFSC, 2001. Available at: easier the assessment management (by http://www.anped.org.br/24/tp1.htm storing and loading observations and Ferreira, T. B. (2003). Gerenciador de Avaliações: grades assigned by educator during the Uma Ferramenta de Auxílio à Avaliação proposed activities development) Formativa para o Ambiente de Educação a [Ferreira 2003]. These research results Distância TelEduc. Master Thesis, Institute of have been used in the proposed MAS Computing/State University of Campinas, architecture, providing important December, 2003. learners’ participation qualitative indices Ferreira, T. B.; Otsuka, J.; Rocha, H. (2003). to be analyzed by the agents of the Interface para Auxílio à Avaliação Formativa proposed architecture; no Ambiente TelEduc. In: Proceedings of XIV Simpósio Brasileiro de Informática na Another that explores interface agents to Educação (SBIE 2003). Rio de Janeiro, do an initial selection of the relevant November, 2003. messages of one chat session, according Freitas, C. E. F. (2002). Desenvolvimento de uma to educator’s interests [Lachi, Otsuka e Ferramenta de Avaliação Objetiva para Rocha 2002; Lachi 2003], in order to Ambientes Computacionais de EaD. Master reduce the educator overload during the Qualify, Institute of Computing/ State analysis of a chat session log. In order to University of Campinas, December, 2002. integrate this new chat tool into the Gennari, J. ; Musen, M. A.; Fergerson, R. W. ; proposed architecture, its interface Grosso, W. E.; Crubézy, M.; Eriksson, H. ; should be adapted to make possible Noy, N. F. ; Tu, S. W. (2002) The Evolution indicate the relevance degree of each of Protégé: An Environment for Knowledge- selected message (not relevant, low Based Systems Development, 2002. relevance, medium relevance, relevant, Available at: http://protege.stanford.edu/ high relevance). Besides, the proposed Gluz, J. C.; Viccari, R. M. (2003) Linguagens de architecture should be extended to make Comunicação entre Agentes: Fundamentos possible monitor this new chat tool; Padrões e Perspectivas. In Vieira, R., Osório, The last one aims the development of a F., and Rezende, S., eds., Proceedings of III set of exercises management tools Jornada de Mini-Cursos de Inteligência TIDIA First Workshop – E-Learning Project November 2004
  9. 9. Artificial, vol. 8. Campinas: SBC. 53-102, July, 2003. Otsuka, J. L.; Rocha, H. V. (2002) Avaliação Gomez, M.V. Avaliação Formativa e Continuada Formativa em Ambientes de EaD. In: da Educação Baseada na Internet. Proceedings Proceedings of XIII Simpósio Brasileiro de of VI Congresso Internacional de Educação a Informática na Educação (SBIE 2002). São Distância. ABED, 1999. Leopoldo, 12-14 de novembro, 2002 Hadji, C. (2001). Avaliação Desmistificada. Porto Otsuka, J. L., Ferreira, T. B., Lachi, R. L., Rocha, Alegre: Artmed, 2001. H. V.(2003) Um Modelo de Suporte à Avaliação Formativa no Ambiente TelEduc. Hewett M.S. (2003). Algernon in Java Revista Brasileira de Informática na Documentation – Available at: http://algernon- Educação, SBC, 2003. j.sourceforge.net/doc Perrenoud, P.(1999). Avaliação: da excelência à Hopper, M. (1998) Assessment in WWW-Based regulação das aprendizagens entre duas Learning Systems: Opportunities and lógicas. Porto Alegre: Artes Médicas, 1999. Challenges. Journal of Universal Computer Science 4, no. 4 (1998): 329-347. Available at: Rocha et al.(2002). Projeto TelEduc: Pesquisa e http://www.jucs.org/jucs_4_4/assessment_in_ Desenvolvimento de Tecnologia para www_based/paper.html Educação a Distância. Prêmio de Excelência ABED/Embratel 2002. IX Congresso Lachi, R. L. Chapa: Um Agente de Interface para Internacional de Educação a Distância da Ferramentas de Bate-papo em Ambientes de ABED, September, 2002. Ensino à Distância na Web. Master Thesis, Institute of Computing/ State University of Rocha, H. (2002). O ambiente TelEduc para Campinas, 2003. Educação à Distância baseada na Web: Princípios, Funcionalidades e Perspectivas de Lachi, R; Otsuka, J.; Rocha, H. (2002). Uso de desenvolvimento. In: Moraes, M.C. (Org). Agentes de Interface no Suporte à Análise de Educação à Distância: Fundamentos e Sessões de Bate-Papo. In: Proceedings of 5th Práticas. Campinas, SP:Unicamp/Nied, 2002, Workshop on Human Factors in Computer pp. 197-212. Systems (IHC 2002). Fortaleza, October, 2002. Thorpe, M. (1998) Assessment and 'Third Generation' Distance Education. Distance Musen, M. A.; Fergerson, R. W. ; Grosso, W. E.; Education 19, n. 2, 1998: 265-286. (EJ 582 Noy, N. F.; Crubezy M., & Gennari, J. 131). H..(2000). Component-Based Support for Building Knowledge-Acquisition Systems. In: Vaucher, J. Ncho, A. (2004). Jade Primer. Proceedings of Conference on Intelligent Available at: Information Processing (IIP 2000) of the http://www.iro.umontreal.ca/%7Evaucher/Age International Federation for Information nts/Jade/JadePrimer.html. Processing World Computer Congress (WCC Wooldridge, M.; Jennings, N.R.; Kinny, D. (2000) 2000), Beijing, 2000. The Gaia Methodology for Agent-Oriented Nelson, G. E. (1998). On-Line Evaluation: Analysis and Design. Journal of Autonomous Multiple Choice, Discussion Questions, Essay, Agents and Multi-Agent Systems Vol. 3. No. and Authentic Projects. In: Third Teaching in 3 285-312. the Community Colleges Online Conference, Kapiolani Community College, Hawaii, April 7-9, 1998. (ED 430 695). Otsuka, J. L. (2002). Um Modelo Baseado em Agentes de Interface para o Suporte à Avaliação Formativa em Ambientes de Educação a Distância. Campinas: Instituto de Computação da UNICAMP. Doctoral Qualify. Institute of Computing / State University of TIDIA First Workshop – E-Learning Project November 2004