Richter, T. (2012). Feedback: Learners’ Understanding and Preferences. In: Bastiaens, T., & Marks, G. (Eds.), Proceedings of the World Conference on E-Learning in Corporate, Government, Healthcare, & Higher Education (e-Learn) 2012 (Montreal, Canada), Chesapeake, VA: AACE, pp.1268-1273.

1. Feedback: Learners’ Understanding and Preferences
Thomas Richter
University of Duisburg-Essen
Information Systems for Production and Operations Management
Germany
thomas.richter@icb.uni-due.de
Abstract: We investigated learners’ understanding and preferences regarding feedback in scenarios of higher
education amongst five different national cultures. As e-Learning technologies do not limit education on na-tional
contexts but perfectly can support the implementation of educational scenarios on an international plat-form,
local and particularly cultural differences between educational contexts are to be focused in order to
support the learners’ efforts as much as possible. For learners, feedback can be both, a strong motivator but in
case it fails its purpose due to a culturally insensitive implementation, it can be a strong reason for frustration
and dropouts as well. In this paper, we firstly discuss the state of the art regarding feedback in educational
scenarios. Afterwards, we introduce our multinational study, discuss the results, and drive conclusions.
Introduction
In e-Learning scenarios, learning is understood as a self-directed process (Rey 2009, p.33). Schwartz
and Bilsky, (1987, p.552) describe “self-direction” as referring ’to reliance on and gratification from one's in-dependent
capacities for decision-making, creativity and action’. Lenartowicz and Roth (2001, p.311) write that
‘self-directed individuals rely on themselves for achieving desired outcomes; they place great importance on
logical and intellectual-based reasoning as a basis for behavior.’ Konrad and Traub (1999, p.13) introduce
„self-directed learning“ as a form of learning, in which the learner (depending on the kind of his motivation)
decides himself which methods are to be taken in order to check, control, and evaluate the own (metacognitive)
learning process (freely translated from German). Ehlers (2004, p.181) lists the support of the learners’ skill in
self-directed learning as one of the five major quality goals, which proper e-Learning should achieve.
In such a self-directed educational context, motivation is the most crucial success factor (Richter &
Adelsberger 2011, p.1603). If learners lose their motivation in a face-to-face scenario, the educator still has a
chance to recognize it and, to intervene and/or support the regain of motivation (Rothkrantz et al. 2009, p.1). In
contrast, in e-Learning scenarios, this chance is rarely given; without recognizing the learners’ mimics and ges-tures
as tools to communicate satisfaction or frustration (Sandanayake & Madurapperuma 2011, p.72), the edu-cators
depend on the explicit communication of threats against the motivation of the learners. However, learners
from Western countries, rarely communicate irritations. Within most eastern cultures, in which the educator is
highly to be respected and not to be put into question, such an open confession may even be understood as mis-behavior.
Ways to achieve a bit more control over the level of motivation of the learners are monitoring their ef-forts
(Jain 2002) and keeping their motivation on a high level by providing reasonable feedback.
Ackerman & Gross (2010) consider feedback in educational processes being ‘essential for effective
learning’ (p.173). While in educational face-to-face scenarios, we believe to understand our local learners’ per-ceptions
to feedback, we are almost blind in the international context. What, from the perspective of the learners,
is to be considered being “reasonable” feedback? When does feedback help and when might it be a threat by it-self?
According to Carless (2006), Feedback ‘is central to the development of effective learning, yet is compara-tively
underresearched’ and the ‘feedback process is more complex than is sometimes acknowledged’ (p.219).
Hounsell (2003) indicates that ‘feedback plays a decisive role in learning and development’ (p.67). Hig-gins
et al. (2002) criticize that the literature does almost not consider the effects of feedback and Poulos and
Mahony (2008) express that ‘research on students’ perceptions of feedback remains thin’ (p.144).
The study of Ackerman and Gross (2010, p.179) revealed that the amount of given feedback plays a vi-tal
role for the learners’ level of satisfaction. James (2000) argues that feedback can have a negative impact on
the student’s self perception and confidence, if it is provided inappropriately. Yorke (2003) claims that not just
the content of feedback is relevant for its’ appropriateness, but also the psychology of giving and receiving
feedback is crucial to be understood. Providing too little feedback may cause uncertainty (Ackerman and Gross

2. 2010, p.179), but too much feedback also revealed having a negative effect on the learning success (Lilly &
Tippins 2002). What seems to be clear, is that designing a culturally sensible feedback-strategy to preserve or
even raise the learners’ motivation requires understanding and considering the learners’ perceptions of feedback.
Questionnaire (Operationalization)
In contrast to most other studies, we did not focus the efficiency of feedback, but the learners’ under-standing
what kind of feedback may be appropriate (Rae & Cochrane 2008, p.218) for them and when it is to be
provided. We designed our study amongst the results of the studies from Rae & Cochrane (2008) and Pokorny
& Pickford (2010). However, in contrast to their studies, we used a questionnaire instead of conducting target
group interviews, extended the scope of both studies from a national (GB and US) to a multinational level, and
added some further aspects that have been left untouched in both studies. We asked university students in dif-ferent
national contexts to evaluate statements regarding the point of time (immediate or delayed) when feed-back
should be given (Parson & Shils 1951, p.494; Smits et al. 2008, p.186), the subject (item-related or task-related)
to which feedback may be related to (Smits et al. 2008, p. 186), the extent (just point out failures or ex-plain
how it could be done better) of feedback (Rae & Cochrane 2008, p.225), the context (Pokorny & Pickford
2010, p.24), in which feedback may be given (is it considered being a private matter or can it be stated in front
of other learners?), if laud and critique are motivating or discouraging for them (Hofstede & Hofstede 2005),
and if the students feel lost without any feedback (Rae & Cochrane 2008, p.224).
The students were asked to evaluate the following statements on a 4 point Likert-Scale from (1) fully
agree to (4) fully disagree. We did not implement a neutral middle-position because we wanted to force the par-ticipants
to take a clear position in terms of showing a ‘preference’. For the case that a certain statement did not
apply to a specific context, we separately provided the option to choose ‘does not apply to my national context’.
1 Point of time
a. I prefer receiving critical feedback at the time when I do something wrong. (immediate feedback)
b. I prefer receiving critical feedback after I fully completed the task. (delayed feedback)
2 Situation
a. I prefer feedback which refers to the single components of a task. (item related feedback)
b. I prefer feedback which refers to complete tasks. (task related feedback)
c. For me, it is ok when critical feedback is given in front of my colleagues. (immediate feedback provid-ed
in the group)
3 Transmission code
a. I prefer explicitly expressed feedback which simply tells what is wrong.
b. When receiving critical feedback, I like to be informed on how my results could be improved.
4 Form
a. I do not want to know about concrete mistakes, but solely expect to receive a fair evaluation.
b. I like to get information on mistakes and get to know what has been wrong.
c. I like being informed about a mistake without further explanations.
d. I like to get information on what is wrong (name mistakes), why it is wrong (explain), and also like to
know how to do it better.
5 Treatment
5.1 Laud affects my learning motivation
a. positively b. not at all c. negatively
5.2 Admonition (critic) affects my learning motivation
a. positively b. not at all c. negatively
6 No Feedback
No feedback makes me feel insecure in the learning process.
Study Setting
We investigated the national contexts in Bulgaria, Germany, Turkey, Ukraine, and South Korea. The
study was provided in each country’s national language: Bulgarian, German, Turkish, Russian, and Korean. For

3. the translation, we asked local university students to translate the English questionnaire version into the local
language/context. As for the German and the Korean language versions, we had the chance to conduct profes-sional
back-translations in order to get more evidence that the translations were appropriate. Apart from South
Korea, the questionnaire was conducted in its online-form. In South Korea, we conducted the survey in paper-form
as a street survey. We chose the South Korean context to conduct the study on a horizontal (broad) level
(many universities) in order to investigate the issue of inter-institutional differences. By directly addressing the
students in face-to-face situations in Seoul’s subway and staying present during completion, we received results
from 39 universities in and around Seoul. In this context, we used a random route algorithm for the choice of
the participants. As for the German survey, we conducted the questionnaire in its online form on a vertical level
(in-depth study, many students per university). We had the chance to directly involve three German universities,
which supported us by internally inviting all of their students to participate via each university’s e-Mail-distribution
services. In the German context, we wanted to find out if specific faculty cultures show a significant
influence on the answer patterns. The response-rate in Germany was typical for online studies, which was be-tween
2-5 percent of the total number of students. In South Korea, the non-response rate was about 50%. Doing
the adjustment of the questionnaire in those two national contexts appeared meaningful because firstly, the cul-tural
contexts are considered being very different (what promised receiving clearly distinguishable results), sec-ondly,
both national contexts are considered being culturally more or less homogenous (we did not have to dis-tinguish
between different races or tribes) and thirdly, both countries have a singe national language. Regarding
the technological development, both countries are on a similar level. While in Germany, we achieved a total of
1817 fully completed sample elements (2400 responses in total), we received 286 (completed) answers in Korea
(325 total). We only evaluated fully completed questionnaires.
Although within the national context of Germany, a certain spectrum of differences in the answers was
found, only a very small number of faculties (and just related to a few isolated items) actually showed signifi-cant
differences to the other ones. The same pattern-homogeneity could be monitored in the South Korean (SK)
context. Besides rare cases, where the answers concentrated on a single side (positive or negative), a Gaussian
distribution was found. A detailed analysis of the adjustment/scope of the data, including an investigation in the
German context of adult education, can be found in Richter & Adelsberger (2012). After investigating randomly
chosen parts of the full samples from the German context, we found that a larger sample size mainly influences
the answer-spectrum but actually, the general answer-patterns (tendencies) kept very similar: For this investiga-tion,
we randomly chose 100 response-clusters with a size of 30 sample elements of the German overall sample;
in 2% of the results, we found accumulations of extreme values. For the other randomly built response-clusters,
the data per item reflected the patterns of the full sample.
Considering the results of this last described test, we addressed student unions in the academic contexts
of Bulgaria, Turkey, and Ukraine and asked them for support by inviting local students to contribute to our
study. We received 30/40/53 completed questionnaires from Bulgaria/Turkey/Ukraine. Even though, because of
the relatively small number of responses, particularly the results of the studies in Bulgaria, Turkey and the
Ukraine may not be representative for the countries/regions, they appear to be sound and allow us a first under-standing
on the spectrum of possible regional differences regarding the students’ perceptions of feedback.
Study Results
The “Item Nr.” in the following Table 1 refers to the initial numbers at each item of the above listed
statements. Apart of the results (percent positive answers of total: “% (p.)”) per item and country, Table 1 dis-plays
the means (m) and the largest found distance between the highest and the lowest results (amongst coun-tries).
Before evaluation, the responses were binarized in positive (1 & 2) and negative (3 & 4) answers.
item Nr. Bulgaria Germany Korea Turkey Ukraine max.
distance
m. % (p.) m. % (p.) m. % (p.) m. % (p.) m. % (p.) % total
1.a 1.47 76.67 1.80 78.36 1.78 83.57 2.30 60.0 1.62 79.25 23.57
1.b 2.17 53.33 2.51 39.95 2.07 73.43 1.80 85.0 2.47 39.62 45.38
2.a 1.67 93.33 1.76 85.60 2.22 66.43 1.85 92.5 2.22 56.60 36.73
2.b 1.67 96.67 2.04 72.89 1.78 82.87 2.18 67.5 1.68 81.13 29.17
2.c 2.34 53.33 2.08 71.35 2.36 58.39 2.25 75.0 2.13 67.92 21.67
3.a 1.63 86.67 1.72 86.95 1.95 79.37 1.78 85.0 1.34 96.23 16.86

4. 3.b 1.37 96.67 1.21 98.76 1.75 82.87 1.45 95.0 1.26 96.23 15.89
4.a 3.26 16.67 3.58 3.75 3.11 16.78 3.00 20.0 3.26 13.21 16.09
4.b 1.40 90.00 1.36 97.89 1.55 93.01 1.58 100.0 1.36 96.23 10.00
4.c 3.00 23.33 3.54 5.97 3.24 15.38 3.00 20.0 2.96 22.64 17.36
4.d 1.37 90.00 1.23 96.86 1.44 90.91 1.00 97.5 1.36 94.34 7.5
5.1.a 1.33 96.67 1.30 95.15 1.34 97.20 1.50 97.5 1.38 90.57 6.93
5.1.b 3.29 10.00 3.57 4.48 3.56 7.69 3.18 10.0 3.23 13.21 8.73
5.1.c 3.64 3.33 3.74 2.57 3.42 14.34 3.61 2.50 3.64 5.66 11.84
5.2.a 1.90 76.67 2.29 64.95 1.88 87.76 1.90 82.50 2.15 67.92 22.81
5.2.b 3.41 6.67 3.42 8.04 3.44 8.74 3.10 7.50 3.19 7.55 2.07
5.2.c 3.31 13.33 2.80 32.42 2.89 34.62 3.20 17.5 2.70 26.42 7.96
6 1.67 93.33 1.88 79.81 2.08 74.48 2.10 72.5 2.09 67.92 25.41
Table 1: Students’ perceptions of feedback: Study results (mean, % positive answers, max. absolute distance)
In the following Figure 1, the percentage values of positive answers are displayed in a net-diagram. While the
axes represent the items, the net elements (“circles”) show each country’s answers. Although just the crossing
points with each axis actually represent defined values, displaying the national pattern in this particular form re-vealed
being helpful to recognize differences (Richter & Adelsberger 2012).
Figure 1: Learners’ perceptions of feedback: Visualization (% positive answers)
Findings
Regarding some of the investigated aspects of feedback, remarkable national differences were found.
However, the majority of investigated aspects surprisingly did not reveal as large differences as we expected.
Unless the different concepts of failure regarding shame and guilt (e. g., Hofstede and Hofstede 2005, p.102),
the participating students from all national contexts had a very similar understanding of laud and critique: Both,
in the average of all contexts, are considered having a positive effect. In example, the variance regarding the
motivating effect of critique (item 5.2.a) had its extreme values in South Korea (pos. max) and Germany (pos.
min). With 22.81 %, this difference cannot be considered high but reflects the average answer spectrum within
national contexts. This is amazing, not just because the literature says that Asian people generally may experi-ence
difficulties standing critique; ‘learning from mistakes’ is a central concept for knowledge improvement in

5. Germany. In South Korea, this idea of the usefulness of mistakes is uncommon. Hofstede and Hofstede (2005,
p.98) claim that mistakes in the Asian context mean personal failures and can lead to face loss.
The most significant national differences have been found in the items related to the point of time
when feedback is to be given (1.b; delayed: 45.38%) and if feedback should be given item related (2.a; 36.73%)
or task related (2.b; 29.17%). Regarding the question if students feel irritated if they do not receive any feed-back
at all (6), we found a difference of 25.41% between the investigated national contexts. Unless this variance,
the mean in all contexts shows that all involved students generally feel confused if no feedback is provided
(1.76-2.10). Receiving just a fair evaluation (4.a) or only getting shown, at which point the students failed (4.c),
is considered insufficient. Most students are not satisfied with just knowing if they have done something wrong
(4.c) or what it has been in detail (4.b - although most at least like to have this information), but they also like to
know how they can improve their results (3b, 4d). Regarding a more or less private atmosphere, in which feed-back
should be given, the German and the Turkish student seemed least to care if other students also know
about their failures (2.c). This is quite different in the South Korean and the Bulgarian contexts where almost an
equal distribution for or against providing feedback amongst other students has been found.
Conclusions
The results of our study showed that amongst our investigated topics, just a few aspects actually need a
higher sensibility when designing feedback in internationally distributed e-learning scenarios. In the study, the
students of the investigated countries generally considered qualified feedback in form of laud as well as critique
being motivating. They explicitly wanted to be evaluated in detail, as long as they also are informed on how
they can improve. Major national differences were found in terms of the subjects that are to be evaluated
(item/task related), the point of time when those are evaluated (directly/delayed), and the situation (private/in
group), in which the evaluation takes place. A contextually sensible design regarding those aspects is strongly
recommended in order to keep the learners’ motivation on a high level. This sensibility also is required regard-ing
a decision if feedback is considered being a private matter. Even if a student expects receiving critical feed-back
from a trusted educator this does not mean that the student does not feel ashamed because of the failure.
Limitations: We found significant differences between the contexts of adult and higher education
(Richter & Adelsberger 2012). Thus, the herein discussed results must be limited to the investigated context.
We did not yet investigate the basic education sector. We plan to focus on this context in one of our next re-search
steps, particularly in order to get an idea about when this kind of cultural imprinting is built.
References
Ackerman, D.S. & Gross, B.L. (2010). Instructor Feedback: How much do students really want? Journal of
Marketing Education, 32(1), pp.172-181.
Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31(2), pp.219-
233.
Ehlers, U-D. (2004). Qualität im E-Learning aus Lernersicht: Grundlagen, Empirie und Modellkonzeption sub-jektiver
Qualität. Wiesbaden: VS Verlag für Sozialwissenschaften.
Higgins, R., Hartley, P., & Skelton, A. (2002). The conscientious consumer: Reconsidering the role of assess-ment
feedback in student learning. Studies in Higher Education, 27(1), 53-64.
Hofstede, G., Hofstede, G.J., (2005). Cultures and Organizations. Intercultural Cooperation and Its Importance
for Survival. 2nd edition, USA: McGraw-Hill.
Hounsell, D. (2003). Student feedback, learning and development. In: M. Slowey & D. Watson (Eds.), Higher
education and the lifecourse, Buckingham: Open University Press, pp.67-78.

6. Jain, K.K (2002). Motivating E-Learners - A Case Study of E-Learning Seeking an Advanced Degree From
UNITAR. Student Affairs Online, 3(4). http://studentaffairs.com/ejournal/Fall_2002/motivatingelearners.htm
James, D. (2000). Making the graduate: perspectives on student experience of assessment in higher education.
In: A. Filer (Ed.), Assessment: social practice and social product, New York, RoutledgeFalmer, pp.151-168.
Konrad, K. & Traub, S. (1999). Selbstgesteuertes Lernen in Theorie und Praxis. München: Oldenbourg Schul-buchverlag.
Lenartowicz, T. & Roth, K. (2001). Does subculture within a country mater? A cross-cultural study of motiva-tional
domains and business performance in Brazil. Journal of International Business Studies, 32(2), pp.305-
325.
Lilly, B. & Tippins, M. J (2002). Enhancing student motivation in marketing classes: Using student manage-ment
groups. Journal of Marketing Education, 24(3), pp.253-264.
Parson T, Shils E (1951). Toward a general theory of action. Cambridge, MA: Harward University Press.
Pokorny, H., Pickford, P. (2010). Complexity, cues and relationships: Students perceptions of feedback. Active
Learning in Higher Education, 11(1), pp.21-30.
Poulos, A. & Mahony, M.J. (2008). Effectiveness of Feedback: The Students’ Perspective. Assessment and
Evaluation in Higher Education, 33(2), pp.143-154.
Rey, G.D (2009). E-Learning. Bern: Verlag Hans Huber.
Rae, A.M., & Cochrane, D.K. (2008). Listening to students: How to make written assessment feedback useful.
Active Learning in Higher Education, 9(3), pp.217-230.
Richter, T. & Adelsberger, H.H (2011). E-Learning: Education for Everyone? Special Requirements on Learn-ers
in Internet-based Learning Environments. In: T. Bastiaens & M. Ebner (Eds.), Proceedings of the World
Conference on Educational Multimedia, Hypermedia and Telecommunications 2011, Chesapeake, VA: AACE,
pp.1598-1604.
Richter, T. & Adelsberger, H.H (2012). On the myth of a general national culture: Making specific cultural
characteristics of learners in different educational contexts in Germany visible. In: F. Sudweeks, H. Hrachovec,
& C. Ess (Eds.), CATaC'12 Proceedings (Aarhus, Denmark): Cultural Attitudes towards Technology and
Communication. School of Information Technology, Murdoch University: Murdoch, Australia.
Rothkranz, L., Dactu, D., Chiriacescu, I., & Chitu, A.G (2009). Assessment of the emotional states of students
during e-Learning. In: A. Smirkarov, W. Bodrow, & A. Ivanova (Eds.), Proceedings of the International Con-ference
on e-Learning and Knowledge Society, pp.77-82.
Sandanayake, T.C & Madurapperuma, A.P (2011). Novel Approach for Online Learning Through Affect
Recognition. In: Proceedings of 5th International Conference on Distance Learning and Education IPCSIT
vol.12, Singapore: IACSIT Press, pp.72-77.
Schwartz, S.H & Bilsky, W. (1987). Toward a Universal Psychological Structure of Human Values. Jour-nal
of Personality and Social Psychology. 53(3), pp.550-562.
Smits, M.H.S.B., Boon, J., Sluijsmans, D.M.A, & van Gog, T. (2008). Content and timing of feedback in a web-based
learning environment: effects on learning as a function of prior knowledge. Interactive Learning Envi-ronments,
16(2), pp.183-193.
Yorke, M. (2003). Formative assessment in higher education: moves towards theory and the enhancement of
pedagogic practice. Higher Education, 45(4), pp.477–501.