An experiment was conducted to test using voice recognition software to automatically monitor call center interviews for correctly recording responses. Interviews from 42 respondents were analyzed. Voice recognition was able to correctly recognize responses in about 71% of cases. Using word filtering and synonym matching increased the recognition rate to about 91%. Recognition rates varied by question type and respondent gender. The results were promising but improvements are still needed, such as automating the separation of interviewer and respondent audio. Future work could explore monitoring additional question types and languages.

1. An Experiment with Voice Recognition to
Improve Call Center Quality
Lew Berman, PhD, MS
Don Allen
Chuck Akin
Josh Duell
Matt Jans, PhD
May 21, 2018
2018 International Field Directors and Technology Conference
Denver, Colorado

2. Agenda
 Framing the problem
 Monitoring metrics
 Experimental design
 Results
 Discussion
2

3. Framing the problem
 Call center monitoring for large-scale national phone surveys
 Current approach
 Record all interviews and save recordings for 60 days
 Randomly select ~10% of ‘survey time’
 Include a variety of dispositions
 Challenges
 Many files and considerable disk storage, much of which is not actively used
 Manually monitor calls
 Costly to review calls and store files
 Difficult to track performance over time or in real-time
 Thus, investigating automation
3

4. Current Approaches
 Common elements, varying by mode
 Silent monitoring during call
 Direct observations
 Telephone verification interviews
 Record and review interview (CARI)
 Data checks
 Typically review about 5-10% of all interviews [FEDCASIC 2018]
 RTI Quality Monitoring System [SPEIZER 2008]
 Standardized approach to telephone and in-person survey
 Monitoring of protocols, metrics, and feedback
 Collection of trend data
 Increased efficiency of monitoring operations
4

5. Monitoring Metrics – Anticipated
Automation Challenges
Automation Challenge
Least challenging,
but not simple
Moderately
challenging
Very challenging
Interviewer Speech
Acceptable speech tempo 
Reads questions verbatim 
Enunciation is clear and not breathy, crackly, strained, etc. 
Avoids monotonic speech patterns 
Avoids repeating words, pauses, filled pauses (“um”), dead air 
Maintains neutrality / impartial and not leading 
Data Capture
Correctly record data 
Correctly record dispositions 
Demeanor / behavior
Addresses resistance 
Converts refusals appropriately 
Engages respondent 
Maintains control of the interview 
Stays on script and does not lead 
Tone is professional and pleasant 
Transitions in acceptable manner 
5

6. Our Study: Correctly Record Data - Low
Hanging Fruit
 Operational Definition: interviewer correctly records respondent answer and
does not inadvertently or purposefully miscode answers or dispositions
 Reasons to monitor incorrectly recorded data
 Attention diverted
 Interviewers moving between surveys and gets stuck in keystroke patterns
 Falsify data (changing or omitting data)
 Fabricate data (make up data)
 Improve production rates
 Measurement
 Compare response in database to that provided by respondent
6

7. Experimental Design
 Respondents: 51 ICF employees, friends, and family invited
 42 participated: 20 female (48%), 22 male (52%)
 Native and non-native English speakers
 Interviewers: 3 experienced female interviews
 Received basic training on study
 Study conducted June 2017, Voxco platform, cell phones only
 7-item instrument, but respondents provided with random responses
 Cell phones
 Carriers primarily Verizon (48%) and AT&T (31%)
 Devices primarily iPhones (79%)
 Determined to be non-human subjects research
7

8. Conceptual View of Voice Recognition Process
for Correctly Record Data
Synonym
Matching
Male same as boy, mail
Female same as girl, woman
* Photos by Unknown Author and are licensed under
CC BY-NC-SA. Changes have been made to
diagrams.
JSON File Output with
Voice Recognition Data
Phone
Interview
with
Respondent
Voxco Survey
Software Platform
Microsoft Voice
Recognition REST API
Wav files for recorded
questions and
responses
Word
Filtering
Hesitations, pausing,
emotional expressions, etc.
8

9. Voice Recognition Results: Correctly
recorded data exact matches
Total Female Male
Category N % Correct N % Correct N % Correct
Gender Question 42 20 22
Incorrectly Recognized 29 69.05% 11 55.00% 18 81.82%
Correctly Recognized 13 30.95% 9 45.00% 4 18.18%
Number Question 42 20 22
Incorrectly Recognized 14 33.33% 7 35.00% 7 31.82%
Correctly Recognized 28 66.67% 13 65.00% 15 68.18%
Yes-No Question 210 100 110
Incorrectly Recognized 42 20.00% 24 24.00% 18 16.36%
Correctly Recognized 168 80.00% 76 76.00% 92 83.64%
Total 294 71.09% 140 70.00% 154 72.08%
9

10. Impact of Word Filtering & Synonym
Matching
45%
65%
76%
18%
68%
84%
31%
67%
80%
80%
80%
95%
73%
77%
99%
76%
79%
97%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Gender Number Yes-No Gender Number Yes-No Gender Number Yes-No
Female Male Total
% Respondent Answers Correctly Recognized and Synonym Matched
by Respondent Gender and Question Type
% Correctly Recognized % Correctly Recognized + Synonym Matched
10

11. Impact of Word Filtering & Synonym
Matching
Respondent
Group /
Question Type
N
% Correctly
Recognized
# Incorrectly
Recognized
# Synonym
Matches
% Synonym
Matched
Total
Recognized
+ Matched
% Recognized +
Synonym
Matched
Female
Gender 20 45.00% 11 7 63.64% 16 80.00%
Number 20 65.00% 7 3 42.86% 16 80.00%
Yes-No 100 76.00% 24 19 79.17% 95 95.00%
140 70% 42 29 69.05% 127 90.71%
Male
Gender 22 18.18% 18 12 66.67% 16 72.73%
Number 22 68.18% 7 2 28.57% 17 77.27%
Yes-No 110 83.64% 18 17 94.44% 109 99.09%
154 72.08% 43 31 72.09% 142 92.21%
Total
Gender 42 30.95% 29 19 65.52% 32 76.19%
Number 42 66.67% 14 5 35.71% 33 78.57%
Yes-No 210 80.00% 42 36 85.71% 204 97.14%
Total 294 71.09% 85 60 70.59% 269 91.50%
Impact of Synonym Matching Overall Improvement 11

12. Discussion – Technical Aspects
 Results are promising for verification of correctly recording responses
 Filtering
 Currently manual operation, but can be automated
 Requires word corpus (e.g., “um”) and rules such as removing repeats
 Synonym corpus
 Well understood for gender, yes-no
 However, other questions such as scales / categorical questions will need additional effort
12

13. Discussion – Human Impacts
 Interviewers
 Provide timely feedback for new interviewers
 Follow-up with objective information for interviewers with poor performance
 Should not be used in a punitive manner, but as a tool for corrective action
 Management
 Understand impact of coaching and training
 Produce daily, weekly, monthly trends reports for supervisors, interviewers, and clients
 Sponsor perspective
 Increase level of monitoring
 Focus on key interview questions for 100% monitoring
 Utilize objective measures
13

14. General Limitations
 No noise-filtering done on sound files
 Occasionally no clean break between interviewer and respondent
 Only considered English interviews
 Did not account for non-native speakers in analyses
 Occasionally the Microsoft Cognition API would respond with errors
14

15. Voxco Limitations
 Typically records interview as one “.wav” file
 Prefer separate “.wav” files per question and response
 Implications for setup, interviewer adjustments, and run-on question/response
 Manually broke up “.wav” files into interviewer and respondent files
 Future work required to differentiate
 Signal & noise
 Interviewer, respondent, dog, baby, …
15

16. Future
 Automate the division between interviewer and respondent
 Utilize noise filtering prior to voice recognition
 Look at other question types such as Likert scales and dates
 Automate speech tempo
16

17. Lew Berman, PhD, MS
Email: lewis.berman@icf.com
Phone: 301-407-6833
17

18. References
[FEDCASIC 2018] Federal Computer Assisted Survey Information Collection (FedCASIC)Workshops: Roundtable: What Makes a Good
Interviewer? Metrics and Methods for Ensuring Data Quality. (2018) Organizers: Matt Jans, Lew Berman. Moderator: Matt Jans. April
18, 2018, Suitland, MD. https://www.census.gov/fedcasic/fc2018/ppt/full_program_2018.pdf.
[JACEWICZ 2009] Jacewicz E, Fox RA, O’Neill C, Salmons J. Articulation rate across dialect, age, and gender. Language variation and
change. 2009;21(2):233-256. doi:10.1017/S0954394509990093.
[NCVS 150] National Center for Voice and Speech. Tutorials - Voice Production. Voice Qualities.
http://www.ncvs.org/ncvs/tutorials/voiceprod/tutorial/quality.html
[SPEIZER 2008] Speizer H, Kinsey S, Heman-Ackah R, Thissen M. Rita. (2008) Developing a Common, Mode-independent, Approach
for Evaluating Interview Quality and Interviewer Performance. Research Triangle Institute Internal Report.
https://www.rti.org/sites/default/files/resources/kinsey_iii-c.pdf.
[TSAO 2006] Tsao Ying-Chiao, Weismer Gary, Iqbal Kamran. Interspeaker variation in habitual speaking rate: Additional evidence.
Journal of Speech, Language, and Hearing Research. 2006;49:1156–1164.
18