Recommended
More Related Content
Similar to Detect Praise and Criticism in Research Reviews
Similar to Detect Praise and Criticism in Research Reviews (20)
More from Verena139
More from Verena139 (16)
Recently uploaded
Recently uploaded (20)
Detect Praise and Criticism in Research Reviews
- 1. PeerJudge: Praise and Criticism Detection in F1000Research reviews Mike Thelwall University of Wolverhampton
- 2. PeerJudge Overview • Based on a dictionary of review sentiment terms and phrases from F1000Research reviews • Each dictionary term or phrase has a praise or criticism score • Well written: +2 • Flawed: -4 • Reviews given the maximum positive and negative scores of words or phrases found in each sentence. • -1: no criticism … -5: very strong criticism • 1: no praise …. 5: very strong praise • Also 12 linguistic rules to cope with negation, booster words (very, slightly)
- 3. PeerJudge Example • The paper is well written but the study is poorly designed. • Praise: 2; Criticism: -4. • Try: online • http://sentistrength.wlv.ac.uk/PeerJudge.html
- 4. Part of the dictionary acceptabl* 3 accurate 2 adequat* 3 appropriate 3 arbitrary -2 balanced 2 bewilder* -3 but 1 careful* 3 clarify -2 clear 4 clearer -3 compelling 3
- 5. Technical details • Java jar program • portable • Dictionaries are external plain text files • Easily customizable • Fast • 14,000 reviews per second • Explains its judgement • So is transparent and the owner can adjust the dictionary for recurrent problems • Agrees above random chance with reviewer scores • Because based on a dictionary, does not “cheat” by identifying hot topics, fields, affiliations or jargon
- 6. Where is the dictionary from? • Human evaluation of a development dataset of F1000Research reviews • Machine learning to suggest extra terms and different weights
- 7. Limitations • Designed for F1000Research decisions – needs dictionary modification for good performance on other review datasets. • F1000Research reviews are unbalanced – few negative decisions • F1000Research reviews have standard concluding text that had to be removed – so referees might not conclude • Referees often give judgements in field-specialist languages, avoiding general conclusions • More substantial modifications may be needed for technical domains. • Difficult to do this in advance because very few outlets publish reviews and scores
- 8. Applications • Warning reviewers if their judgements are apparently out of line with their scores? • Warning reviewers if they have not given any praise. • As above for editors • On a larger scale, allow publishers to check for anomalies in the reviewer process, such as by identifying journals with uncritical referees (low average criticism scores).