The document discusses standards for the design and delivery of online engineering programs in South Africa. It outlines ECSA's accreditation standards for online programs, which include additional requirements beyond traditional in-person programs. Some key points discussed are the legislative framework for accreditation, planning assessments at the program level to map learning outcomes to graduate attributes, designing multiple choice questions and rubrics, and examples of documentation required for accreditation of online programs.

2. DESIGNING HIGH
QUALITY ASSESSMENTS
CSET
Presenter: Dr. P Govender (Salosh)
• ADOVH : DHET PROJECT

3. OVERVIEW OF PRESENTATION
Legislative framework:
ECSA’s Accreditation
standards for online
module
Planning at Programme
level: Graduate
attributes
Designing MCQs
Assessment tool
Rubrics
3

4. ACCREDITATION STANDARDS FOR ONLINE PROGRAMME: ENGINEERING
COUNCIL OF SOUTH AFRICA ( REVISED 10 FEB 2021)
Purpose of the document
• Outlines the standards in the design
and delivery of online programmes
in order to meet the requirements
towards registration in ECSA.
• This Standard document highlights
the additional requirements of
online programmes to meet the
ECSA accreditation criteria described
in the E-03-CRI-P policy.
https://ecsa.co.za/ECSADocuments/Shared%20Document
s/E-24-
STA%20Standard%20for%20Accreditation%20of%20Onlin
e%20Programmes%20.pdf
4

5. 6. RECORDS AND DOCUMENTATION TO BE SUBMITTED FOR ACCREDITATION OF
ONLINE PROGRAMMES
• Students’ scripts
• Course material
• Assessment materials
• Marked examination, project and assignment scripts
• A selection of the best, the average and just passed/failed students’ scripts at each
level and in each course.
• Examples of final-year design and/or laboratory/investigational projects for the
most recent year
• Access to individual students’ academic records on request
• Evidence of internal quality assurance process, including sample paper trails for
selected courses and all exit-level courses.
• Evidence of moderation process (internal and external), including the moderators’
high-level reports for the most recent examination.
5

6. 7. Accreditation criteria in the context of online programme
There are 4 accreditation criteria specified in the document
• 7.1 Criterion 1 : Credits , knowledge profile and coherent design ( no additional elements required)
• 7.2 Criterion 2: Assessment of Graduate Attribute ( No additional elements required)
• 7.3 Criterion 3: Quality of online teaching and learning
Related to Assessment ( Additional to online offerings)
• The method of assessment for each module of the online programme must be defined and documented in order to
achieve and ensure assessment integrity
• Method of identity verification for each student must be defined and documented to ensure assessment integrity.
• Where the method of assessment makes use of continuous evaluation, the engineering education provider must
ensure that the complexity is at the correct NQF level and that the evaluation can be externally moderated
• External moderation: The moderator is not in the employ of the provider, has made no input into the programme and has no
prior contact with the students.
• The Assessment of Graduate Attributes may be conducted online. The engineering education provider must
demonstrate the integrity of the online assessment.
• Note: Engineering education providers are accorded flexibility in the use of either the set of exemplary assessment
criteria in the relevant Standard or an alternative and fully documented set that demonstrates achievement of each of
the learning outcomes at the specified level.
6

7. Graduate attributes and outcomes
GA OUTCOMES
Problem solving Demonstrate competence to identify, formulate, analyse and solve complex engineering problems creatively
and innovatively
Application of scientific and
engineering knowledge
Apply knowledge of mathematics, natural science and engineering fundamentals and an engineering speciality
to solve complex engineering problems
Engineering Design Demonstrate competence to perform creative, procedural and non-procedural design and synthesis of
components, systems, engineering works, products or
Investigations, experiments and
data analysis :
Demonstrate competence to design and conduct investigations and experiments
Engineering methods, skills and
tools, including Information
Technology :
Demonstrate competence to use appropriate engineering methods, skills and tools, including those based on
information technology.
Professional and technical
communication
Demonstrate competence to communicate effectively, both orally and in writing, with engineering audiences
and the community at large
7

8. Graduate attributes and outcomes
GA OUTCOMES
7. Impact of Engineering activity Demonstrate critical awareness of the sustainability and the impact of engineering activity on the social,
industrial and physical environment
8. Individual, team and
multidisciplinary working
Demonstrate competence to work effectively as an individual, in teams and in multidisciplinary environments.
9. Independent learning ability Demonstrate competence to engage in independent learning through well-developed learning skills.
10. Engineering Professionalism Demonstrate critical awareness of the need to act professionally and ethically and to exercise judgment and take
responsibility within own limits of competence
11. Engineering Management Demonstrate knowledge and understanding of engineering management principles and economic decision-
making.
8

9. SAQA LEVEL DESCRIPTORS AND NQF
LEVELS
SAQA Graduate attributes
1. KNOWLEDGE
2. KNOWLEDGE LITERACY
3. METHOD AND PROCEDURE
4. PROBLEM SOLVING
5. ETHICS AND PROFESSIONAL PRACTICE
6. ACCESSING, PROCESSING AND
MANAGING INFORMATION, SYMBOLIC
SYSTEMS
7. PRODUCING AND COMMUNICATING
INFORMATION,
8. CONTEXTS AND SYSTEMS
9. MANAGING LEARNING
10. ACCOUNTABILITY
9

10. SAQA LEVEL DESCRIPTORS
10

11. Graduate attributes and level descriptors
Level: A measure of learning demands expressed in terms of level descriptors and encompassing types of
problems, knowledge required, skills and responsibility.
Level Descriptor: Well-defined engineering
problems: NQFL5
Level Descriptor: Broadly-Defined engineering
problems: NQF L6
Level Descriptor: Complex Engineering
Problems: NQF L7
a) Can be solved mainly by practical
engineering knowledge, underpinned by
related theory;
and have one or more of the characteristics:
a) require coherent and detailed engineering
knowledge underpinning the technology
area;
and one or more of the characteristics:
a) require in-depth fundamental and
specialized engineering knowledge;
and have one or more of the characteristics:
a) may require clarification;
b) are discrete, focussed tasks within
engineering systems;
c) are routine, frequently encountered, may
be unfamiliar but in familiar context;
and have one or more of the characteristics:
a) are ill-posed, or under or over specified,
requiring identification and interpretation
into the technology area;
b) encompass systems within complex
engineering systems;
c) belong to families of problems which are
solved in well-accepted but innovative ways;
and one or more of the characteristics:
b) are ill-posed, under- or over specified, or
require identification and refinement;
c) are high-level problems including
component parts or sub-problems;
d) are unfamiliar or involve infrequently
encountered issues;
and their solutions have one or more of the
characteristics:
a) can be solved in standardized or prescribed
ways;
b) are encompassed by standards, codes and
documented procedures; requires
authorization to work outside limits;
c) information is concrete and largely
complete, but requires checking and
possible supplementation;
d) involve several issues but few of these
imposing conflicting constraints and a
limited range of interested and affected
parties.
a) can be solved by structured analysis
techniques;
b) may be partially outside standards and
codes; must provide justification to operate
outside;
c) require information from practice area and
source interfacing with the practice area that
is incomplete;
d) involves a variety of issues which may
impose conflicting needs and constraints;
technical, engineering and interested or
affected parties
e) are not obvious, require originality or
analysis based on fundamentals;
f) are outside the scope of standards and
codes;
g) require information from variety of sources
that is complex, abstract or incomplete;
h) involve wide-ranging or conflicting issues:
technical, engineering and interested or
affected parties.

12. PLANNING AT PROGRAMME LEVEL
ECSA Exit Level Outcome Assessment Details
1: Problem solving Learning outcome: Demonstrate competence to identify, formulate, analyse and solve complex engineering problems creatively
and innovatively.
Where is outcome assessed?
How is this outcome assessed?
What is satisfactory performance?
What is the consequence of unsatisfactory performance?
2. Application of scientific and engineering knowledge : Apply knowledge of mathematics, natural science and engineering fundamentals and an
engineering speciality to solve complex engineering problems.
Where is outcome assessed?
How is this outcome assessed?
What is satisfactory performance?
What is the consequence of unsatisfactory performance?
12

13. ELO -----> 1 2 3 4 5 6 7 8 9 10
Chem Plant 3A X X
Thermodyn X X
Chem Eng Tech 3A X X
Eng Math 2 X
Stats 2 X
Mgmt Skills 1A X X X
Chem Plant 3B X X X
Applied Thermo X
Chem Eng Tech 3B X X X
Chem Design Prin X X X X X X
Mgmt Skills 1B X X X
Experiential 1 X X X X X X X X X X
Experiential 2 X X X X X X X X X X
Graduate attributes in a table– excerpts from UJ Chem Tech

14. 14
DESIGNING MCQS AND RUBRICS

15. Why Advanced skills?
❑ Measures high order thinking (HOT)
skills
❑ Aligns to higher NQF level
❑ Includes authentic assessments aimed
at producing graduates who are
responsive to the labour marker.
❑ Promotes problem solving, critical
thinking which is essential for every
profession.
❑ Application of knowledge minimises
the risk of cheating
About randomization.
❑ A number of questions are randomly
drawn from the question pool. Each
student is presented with a unique test
which minimizes the chances of
cheating during the exam. The auto
submit function in the LMS deters
students from copying their question
set and passing it along to their peers
who may not have started the exam
❑ MCQs can be used to assess
students’ knowledge and skills by
incorporating lower, middle and higher
order thinking questions, as set out in
Bloom’s Taxonomy.
❑ HOT MCQs coupled with an open book,
can challenge students and reduce the
risk of cheating.

16. BLOOMING HOT MCQs
• High ordered thinking skills (HOTS) refers to
thinking that goes beyond the basic recall
of memorized information
• Requires cognitive processing such as
critical thinking, problem solving,
application, analysis, evaluation, and
creation.
• General perception that MCQs are
incapable of assessing cognitive processes
beyond recall or recognition of knowledge.
16

17. STEPS IN PLANNING QUALITY RANDOMIZED
MCQS : PROCESS
17
• Align to module
outcomes
• Align to NQF
• Content
mapping
Choosing the content
and level of difficulty
• Higher ordered
• Different types
• Question stem and
plausible distractors
Formulating question
• Quality measures
• Objectivity
• Reliability
• Validity
Evaluating question
• Create
Question pools
• Create
subpools for
randomization
• Adding
questions
• Creating
assessment
Creating randomized
assessments

18. STEP1:
CHOOSING THE
CONTENT AND
LEVEL OF
DIFFICULTY
18

19. STEP 1: Plan / organize your item bank
FMT3701
Module outcomes:
Approximately 5-6
MO
Unit 1 :
Number
operations
and
relationships
Outcomes
Low
order
thinking
(LOT)
High
order
thinking
(HOT)
Unit 2
outcomes
Patterns and
Algebra
LOT HOT
Unit 3
Outcomes
Measureme
nt
LOT HOT
Unit 4
Outcomes
Space and
Shape
LOT HOT
Unit 5
Outcomes
Data
Handling
LOT HOT
19

20. STEP 2: FORMULATING HOT MCQs: REVISED BLOOMS
(ANDERSON AND KRATHWOHL)

21. STRATEGIES FOR PLANNING HOT MCQS
Multi-logical
thinking
Uses visuals:
Graphs,
cartoons,
pictures
Real world
scenarios
and Case
studies
2 Tier
questions

22. Strategy 1: Two tier multiple choice question
• The first tier pertains to a
knowledge or
comprehension statement
and the second tier
facilitates testing of higher-
order thinking.
• It is in effect asking a lower-
order question and then
asking students to provide a
reason (which requires
higher-order thinking).
Suppose you are given two clay balls of equal size and shape.
The two clay balls also weigh the same. One ball is flattened
into a pancake-shaped piece and the other is round.
Select the correct option.
1st tier: knowledge or comprehension statement
a. The pancake-shaped piece weighs more than the ball
b. The two pieces still weigh the same
c. The ball weighs more than the pancake-shaped piece
because ( 2nd tier facilitates thinking of higher order thinking)
d. the flattened piece covers a larger area.
e. the ball pushes down more on one spot.
f when something is flattened it loses weight.
g. clay has not been added or taken away.
h. when something is flattened it gains weight.
Options
1. a and d
2. b and h
3.c and i
4. a and g

23. Strategy 2: Real world scenarios/ case studies
Before ( LOT)
What is the first concern of an emergency
worker?
After ( HOT): Using a scenario
You arrive at the scene of an accident where
people are panicked and yelling. Three people
appear to be injured. Which person will you
take care of first?
• One person has scrapes and bruises,
• A second is lying on the ground moaning in
pain and
• A third is losing blood quickly from a skull
injury.
One of the best ways to
promote and assess higher-
order thinking is to use case
studies and scenario-based
questions. This changes a
question from requiring simple
recall to application of
knowledge

24. EXAMPLES FROM TDC3602
• QUESTION 4
• Arnot Power Station in Mpumalanga, South
Africa, is a coal-fired power plant operated by
Eskom. The plant operates with a heat-source
reservoir at 350 °C and a heat sink-reservoir at
30 °C. It has a thermal efficiency equal to 55 %
of the Carnot-engine thermal efficiency for the
same temperatures.
• 4.1What is the thermal efficiency of the plant?
(4)
• 4. What temperature must the heat-source
reservoir be raised to increase the thermal
efficiency of the plant to 35 %?
(4)
•
[8 marks]
•
• MCQ
• Question 4.1 and 4.2 are based on the
scenario below:
• Scenario
• 4.1 Select the option that best describes the
thermal efficiency of the plant
• Provide 5 options
4.2 What temperature must the heat-source
reservoir be raised to increase the thermal
efficiency of the plant to 35 %
Provide options

25. Strategy 3: Analysis of visuals
Analyse and
interpret visuals
from diagrams or
graphs require
higher-order
thinking to solve.
An ideal monatomic gas is taken round the
cycle ABCDA as shown below in the P-
V diagram. The work done during the cycle is
1. PV
2. 2PV
3. PV/2
4. 0

26. Strategy 4: Multi-logical thinking: Case
study
• Multi-logical thinking questions
are questions with more than
one premise, from which a
student must make the correct
conclusion.
• Requires a student to consider
several facts and aspects before
making a decision.
• In the example given, the
student is required to both
diagnose the symptoms and
decide on what the best way
would be to break the news to a
parent.
Tim’s Grade 2 teacher is concerned about Tim because of the
following behaviour in class:
• Withdraws from peers on the playground and during group
work
• Often confuses syllables in words (ex: says mazagine instead of
magazine)
• Often confuses b and d, p and q, etc. when writing or
recognizing letters
• Loses his place when reading
The teacher has arranged a meeting with Tim’s mother to discuss
these concerns. Which is the best statements to communicate to
Tim’s parents?
• A. Tim needs extra practice reading and writing problematic
letters and words at home at least 30 minutes per day.
B. Please discuss the importance of schoolwork to Tim so that
he will increase his efforts in classwork.
C. These are possible symptoms of dyslexia so I would like to
refer him to a specialist for diagnosis.
D. Please adjust Tim’s diet because he most likely showing
symptoms of ADHD due to food allergies.

27. Item Shells: Lower/ middle order
MCQs
27

28. Item shells for HOT MCQs
28

29. ITEM SHELLS FOR HOT MCQs
Summarisation: A statement which specifies
main idea or main points
Which option best describes the summary of
the article/ paragraph
Which statement best summarises the results
of experiment 2
Conclusion : a judgement formed after critical
thought
What might the reader conclude based on
Thabo’s action?
Based on the data on the chart/ graph, one may
conclude…
Based on the results of the proble, what might
one conclude?
Justifying motives/ reasons : Consider actions,
ideas, reasons, etc
What is the most likely reason Thabo walked
out of the meeting?
Which of the following best justifies Tom’s
decision to sue the employer
Providing support ( reasoning or evidence
behind ideas)
What is the most likely reason that the author
included ….
29

30. CHANGING QUESTIONS TO MCQS
EXAMPLES FROM TDC3602
• The gas turbine is the engine at the heart of
the power plant that produces electric current.
A gas turbine develops 13 000 kW with gas
flows of 16 kg/s. The enthalpy at the inlet is 1
220 kJ/kg and velocity of 55 m/s. At the outlet
the enthalpy is 350 kJ/kg and the velocity are
145 m/s. Determine:
• 2.1 The rate at which heat is rejected from
the turbine. (6)
• 2.2 The diameter of the turbine inlet given
a specific volume of the gas of 0.5 m3/kg.(6)
• Answer question 2.1 and 2.2
which is based on the scenario
below
• Scenario
• 2.1 Determine the rate at which
heat is rejected from the turbine
• Options
• 2.2 The diameter of the turbine
inlet given a specific volume of
the gas of 0.5 m3/kg.
• Options

31. A piece of 40 kg steel casting at a
temperature of 450 °C is quenched
in 150 kg of oil at 25 °C. Assume
there are no heat losses. The specific
heat for steel is Cp steel = 0.5 kJ/kg.
K and for oil are Cp oil = 2.5 kJ/kg. K.
Calculate the change in:
• 5.1 Entropy for the steel casting.
(8)
• 5.2 Entropy for the oil.
(2)
• 5.3 Total entropy of the system.
(2)
• Select the correct option
• 5.1 Calculations illustrating
the change in entropy for the
steel casting is ….
• Options
• 5.2 Calculations illustrating
the change in entropy for the
oil is ….
• Options
• 5.3 Calculations illustrating
the change in total entropy of
the system ….
• Options

32. Essays: Use of rubrics
• Scoring tool that describes the performance expectations
for an assignment.
Identifies the
➢ criteria / aspects of work that will be assessed
➢descriptors: the characteristics associated with each
dimension (e.g., argument is demonstrable and original,
evidence is diverse and compelling)
➢performance levels: a rating scale that identifies students’
level of mastery within each criterion
32

33. Problem solving rubric ( adapted from Dept of Chemical engineering CRCD Project)
33

34. Criteria for problem solving
34

35. Criteria for problem solving
35

36. 36

37. Criteria for problem solving
37

38. GA 6: Professional and technical communication
Learning outcome: Demonstrate competence to
communicate effectively, both orally and in writing
38

39. GA 6: Professional and technical communication
Learning outcome: Demonstrate competence to communicate effectively, both orally and in writing
39

40. THANK YOU
40