Presented at the 8th Conference of the International Test Commission in Amsterdam, The Netherlands, 3-5 July 2012.

1. Effect of Computer-based Testing on
Candidate Perception of Selection
Justice
Aylin Dincer, Levent Sevinç,
Merve Şuşut, İpek Altıntaş
8th ITC Conference, Amsterdam 2012

2. Introduction
• Recent advancements in technology have led
organizations to make use of technology
widely in their recruitment process.
• As a result, there is an increasing adoption of
online test applications in employee
selection procedures.

3. Introduction
• Online recruitment is a reality across Europe
with the more technologically advanced
European community countries leading the
way in these developments. (Lievens et al.,
2002)
• Parallel to these developments in Europe,
online recruitment has been much more
utilized in Turkey as well.
– Nevertheless, online testing has started being used more
recently, yet the demands of organizations for online test
applications are increasing rapidly.

4. Introduction
• In online testing, human resources
professionals mostly focus on the reliability
of the results of applicants.
– Although validity and reliability of the test results are
crucial, only focusing on psychometric issues results in
neglection of non-psychometric factors.
• In order to acquire a holistic view, it is also
important to examine applicant reactions to
online testing.

5. Aim
• To this end, this study aimed at examining
the effect of computer-based testing on
candidates’ perception of fairness of
selection process.
– For this purpose, this study was carried out in the
selection process of a bank in which paper-and-
pencil and online tests were administered to
candidates.

6. METHOD

7. Method
• Research was conducted on a group of
candidates of a private bank’s credit card
sales department (n=235) attended a general
ability test.
– Candidates were separated into two groups: One group
(n=162) were administered a computerized version of the
test and the other group (n=73) were administered a
paper-and-pencil version.
• After completing the test, each candidate
group filled out a scale evaluating selection
justice.

8. Method
• Candidates were selected by the human
resources department through CV screening
among the job applicants.
• The bank carries out the test application
online and proctored in the office
environment (fully managed).
– For this research, test was administered to some of the
candidates in paper-and-pencil version. The candidates
were assigned to each group according to the bank’s
decision.

9. Measures:
Ability Test
• In the study, all candidates were assessed
with the same cognitive ability test.
– FVAT Cognitive Ability Test used in this process was
developed by Assessment Systems.
• Test-retest reliability .89
• Split-half reliability .87
• Predict job performance (manager rating) r=.51 p<.01
• Correlation coefficients of online and paper-pencil versions .91
– The test consists of 40 face valid questions, with a duration
of 60 minutes. It measures 4 different abilities: Numerical,
verbal, analytical thinking and attention.
• Average difficulty index .55

10. Measures:
Ability Test
• Psychometric properties and results of the
test did not indicate a significant difference
between the two groups.
– The split-half reliability of the test did not significantly
differ between the two groups.
• Computer-based = .87; Paper-and-pencil =.86
– The difficulty index did not indicate a significant
difference between the two groups.
• Computer-based = .57; Paper-and-pencil =.56
– There was no significant difference between the number of
correct answers in each group.
• Computer-based : M=22.66 (SD=7.2); Paper-and-pencil: M= 22.41
(SD=8.1); t=-.24 p=.81 >.05

11. Selection Justice Scale
• In this study, Gilliland’s organizational
justice model (1993) was used to examine
the reactions of the applicants in terms of
selection justice.
• Studies of applicant perceptions of selection
processes mostly use an organizational
justice point of view (Ryan & Wessel, 2008).
– Gilliland (1993) pioneered in applying organizational
justice theory to understand how applicants react to the
selection process.

12. Gilliland’s
Selection Justice Model
• Distributive justice, concerns applicant
reactions of whether a specific selection
decision (hiring or rejection) is fair.
• Procedural justice, concerns applicant
reactions to
– the tests and techniques used in evaluating their
qualifications.
– the organizational policies implemented for processing
applicants, and
– the interactions that applicants have with various members
of the organization during the hiring process.

13. Gilliland’s
Selection Justice Model
• Gilliland mentioned 10 rules, can be grouped
in 3 categories, concerning procedural
justice
– The formal characteristics of the selection process which
includes job relatedness, opportunity to perform.
consistency
– Information given to applicants about the hiring process
and decision, it includes rules like feedback and honesty in
communications with candidates.
– Interpersonal treatment which includes treatment during
the selection process, two-way communication, and
propriety of questions.

14. Selection Justice Scale
• The Selection Justice Questionnaire
developed by Gilliland ve Honig (1994) was
adapted for this study.
– Procedural justice was considered within 3
categories rather than 10 sub-scales.
– Accordingly, the scale consists of 4 dimensions
together with distributive justice.

15. Selection Justice Scale
• The scale is composed of 14 items arranged
along a 5-point Likert-type Scale.
– Distributive Justice, 4 items.
• “I think assessment based on this test will be fair.”
– Procedural Justice, 10 items.
• “This test gave me the opportunity to demonstrate my abilities to the
authorities of the organization.”
• “The candidates were sufficiently informed about the announcement
date of results.”
• “During the test application, the authorities answered the questions
sincerely.”

16. Selection Justice Scale
• Factor Analysis revealed that the scale
consists of 3 dimensions.
– Two categories of Procedural Justice (interpersonal and
informational) were combined into one dimension and
named as «interactional».
– KMO=.71; Bartlett’s Test chi square 693.47 p=.00<.01; total
variance explained=57%
• The Cronbach’s Alpha values calculated for
each dimension indicate that the scale has
high internal consistency.
– Distributive= .88; Procedural=.82; Interactional=.87

17. Participants
• The distribution of the participants (n=235)
in terms of their demographic characteristics
is as follows:
– 100% university graduate
– 52% males
– Mean age 22.4 (SD=1.8)
• The distribution of each group in terms of
their demographic characteristics is similar.

18. RESULTS

19. Results
• In both groups, candidates perceived the process
positive (>4.00). In candidates who were
administered the computerized version,
perceptions of procedural and distributive justice
were significantly higher (<.01).
Pen-and-Paper Computer-based
t Test
(n=73) (n=162)
M SD M SD t p
Distributive Justice 4.23 .75 4.62 .44 -4.96 .00**
Procedural Justice 4.10 .92 4.56 .49 -4.95 .00**
Interactional Justice 4.70 .41 4.68 .44 .39 .70
**p<.01

20. Results
• According to the results, candidates who
were administered the computerized version
mentioned that;
– Assessment based on the test would be much more fair.
(distributive justice)
– The test used gave them the opportunity to highly
demonstrate their abilities and test was strongly job
related. (procedural justice)

21. Results
• On the other hand, interactional justice
perception was not significantly (p>.05)
different between two groups.
– In both groups candidates stated that proctors permitted
them to ask questions and answered the questions
sincerely.
• Insignificance may be explained by
employing proctor in both of the groups.
– Because of the organizational policies, proctors’ behaviors
would have probably been similar to each other.

22. Comparison
• When results are compared to a similar study
on perception of justice about selection
process (Sevinç & Yılmaz, 2005);
– Researchers examined selection justice perceptions of a
bank candidates (n=97) according to Gilliland’s model as
well, but the test had no online version, it was
administered in paper-pencil form.
– In this study, it is revealed that the perception justice
scores of candidates who were administered paper-pencil
test was parallel to 2005 study results.
– Thus, the increase in the perception of justice scores may
be explained by online test administration.

23. CONCLUSION

24. Conclusion
• When the candidates take the test online,
they evaluate the selection process and the
content of the assessment tools positively.
– Philo and Green (2007) showed that the candidates of a
company which used online test for the first time judged
the content to be more fair when compared to traditional
testing methods. Thus, it can be said that applicants found
the online test experience positive.
– Online testing is relatively a new concept in Turkey.
Candidates would have taken online test for the very first
time and this situation would have affected them to
evaluate the process positively.

25. Conclusion
• The perception of fairness is much more
important than whether the test is fair or
not.
– Today, it is not sufficient for the assessment tools to be
valid and reliable. The perception of candidates towards
the tools eliminates the importance of psychometric
properties and influences perceptions about the
organization.
– Therefore, even if we have considerably strong tools as
developers and practitioners, it may lead adverse
perceptions if implementation of these tools is not
combined with modern technology.

26. Conclusion
• Each new generation makes use of
technology more widely compared with the
previous generation.
– Online testing may be considered as new and different.
However it is obvious that it will become second nature for
next generations in the near future.
– Therefore it can be foreseen that the positive impact of
using technology in testing will disappear in the future.

27. Suggestions
• It is suggested that new methods should be
implemented in online testing in order to
manage the perceptions of candidates.
– In the near future, it will be a must for the tests to be
available on social networks like Facebook and mobile
devices which are indispensable for the new generation.

28. Thank you!