This document summarizes a comparison of two plagiarism detection tools: TurnItIn and Urkund. It outlines criteria for evaluation such as cost, functionality, matching ability, turnaround time, and support. Data is presented on usage from 2014-2017 showing Urkund requiring less support tickets and having faster turnaround. While TurnItIn has more functionality, Urkund has a lower cost. Based on the evaluation, the next steps outlined are to terminate TurnItIn in June 2017 and develop an audio feedback plugin, while negotiating a better price for Urkund by increasing its user base.
6. @glynnmark #heie
Support required
Both systems produce a similarity score
23019 reports generated through
submissions made to TurnItIn. In that
time we had 52 tickets to our helpdesk
related to TurnItIn
In the same period we ran 55000+ submissions
through Urkund and we had 4 tickets.
8. @glynnmark #heie
Matching Ability
90% of the documents analysed had a difference in
similarity scores of less than 10% (with 80% of the
documents having less than 5% difference in similarity
score).
The differences in the score can be due to three main
factors.
Similarity
Score
75 – 100% 50 – 74% 25 – 49% 0 – 24%
% of
Reports
1.65% 7.67% 27.5% 67.98%
9. @glynnmark #heie
Functionality
Feature TurnItIn Moodle +
Urkund
Annotation*
Can the lecturer annotate a student’s submission
x x
Marking Guides
Structured feedback on specified criteria with a marks range for each student
x
Rubrics
In Built Rubrics
x x
Group work x
Similarity reports x x
Peer Review x x
Rubric / Marking guides reports**
Can the lecturer get a report providing a breakdown of all of the marks and
comments for each criteria within a rubric/marking guide given to the entire class
x
Audio Feedback x
Extensions
Can the lecturer grant deadline extensions to one or more students easily
x
11. @glynnmark #heie
Next steps
We terminate TurnItIn in June 2017.
We have developed an audio plugin to
facilitate audio feedback similar to TurnItIn.
We get more people buying Urkund to build the
database and help us negotiate a better price.
Editor's Notes
Students enoy the use of Moodle in one class –will demand it in others
Both TurnItIn and Urkund are integrated in Loop via plugin. Both systems produce a similarity score which is produced as part of the originality report and fed back to Loop. For the duration of the evaluation there were 23019 reports generated through submissions made to TurnItIn. In that time we had 52 tickets to our helpdesk related to TurnItIn. In the same period we ran 55000+ submissions through Urkund and we had 4 tickets.
Without exception TurnItIn produced a higher similarity score for each submission, however the 90% of the documents analysed had a difference in similarity scores of less than 10% (with 80% of the documents having less than 5% difference in similarity score).
The differences in the score can be due to three main factors:
Each system uses their own algorithm which leads to slightly different results. This is manifested by TurnItIn counting “matches” in documents with a smaller number of words relative to what Urkund classifies as a “match”
TurnItIn has a larger dataset to chose from as it has a larger customer base particularly in higher education in the UK therefore would have more documents at their disposal
Furthermore TurnItIn analyses relatively more journal databases that are not openly available.
While both TurnItIn and Urkund offer similar services i.e. text matching, it is important to note that TurnItIn also provides feedback functionality. Key features are illustrated in Table 3 below. The feedback feature within TurnItIn is an excellent addition to the product but currently is only used by 89 (8.5%) of our academic staff members for feedback to students. The majority of the staff use it just for similarity scores. It is worth noting that while Urkund does not have feedback functionality Loop does and there will be significant improvements in the next version of Loop scheduled for upgrade in June 2017. Therefore the table below includes Loop in the analysis as well. The question now becomes is the better functionality in TurnItIn for feedback worth the additional year on year cost
The final factor considered in this evaluation is cost. TurnItIn cost €23,406 in 2014, €40247 in 2015 and €33497 in 2016. The actual invoice is quoted in sterling and while our rate was increased from 2014 to 2015 (£23,302) the rate remained stable for 2016 (£25,247) but fluctuations in currency strength resulted in a cost saving.
The cost for Urkund is €10,720 per annum and that is the normal ongoing price i.e. not a special introductory offer. The cost saving between TurnItIn and Urkund is €22,777 per annum. This difference can be put towards improving the feedback functionality of Loop to match if not exceed the current capabilities of TurnItIn as a once off cost.