Generative AI on Enterprise Cloud with NiFi and Milvus
Β
IFD&TC 2019: Automating Call Center Monitoring
1. Automating Call Center Monitoring
IFD&TC 2019
Lew Berman, MS, PhD
John Boyle, PhD
Don Allen
Josh Duell
Matt Jans, PhD
Ronaldo Iachan, PhD
Josiah McCoy
May 21, 2019
2. Problem
ο§ Systematic call center monitoring and behavior coding can be costly, slow,
(somewhat) subjective, and is done on a small sample.
ο§ However, cloud based speech recognition and machine learning tools are
available at modest cost and elastic scale. Can these tools be used to
overcome monitoring challenges?
ο§ Experiment 1: What is the level of agreement between call center monitors
(humans) and tools to classify the subject response to a question?
ο§ Experiment 2: How accurate are these tools in computing interviewer
speech rate and articulation rate? [underway]
2
3. Solution β Speech recognition and machine
learning
3
* Photos by Unknown Author and are licensed under CC BY-NC-SA. Changes have been made to diagrams.
Speech Recognition Language Understanding
Question
Response
Text Cleanup
Phone
Interview
Auto-Parse
Interview into
Question
Snippets
Have you smoked 100 cigarettes in your lifetime? no
0.5s 1.2s 2.3s 2.7s 3.4s 4.2s 5.3s 8.5s
Microsoft Azure
Amazon Web Services
5. Results: Response agreement between
monitors and tools
5
BRFSS Question Type Agreement
Would you say that in general your health is excellent, very good,
good, fair or poor?
Straight forward wording
Multi-category
Straight forward response
96%
(N=23)
Have you smoked at least 100 cigarettes in your entire life? Straight forward wording
Multi-Category (yes/no)
Straight forward response
70%/100%
(N=23)
Are you currently employed for wages, self-employed, out of work for
1 year or more, out of work for less than 1 year, a homemaker, β¦
Complex wording
Multi-Category
Complex response
61%
(N=23)
How often do you use seat belts when you drive or ride in a car -
would you say always, nearly always, sometimes, seldom or never?
Moderately complex wording
Multi-Category
Moderately complex
response
70%
(N=23)
Please think of the past 12 months. How many times did you reduce
or change your outdoor activity level based on the air quality index or
air quality alerts? For example avoiding outdoor exercise or
strenuous outdoor activity. Please do not include β¦
Complex wording
Long question
Multi-Category
Complex response
20%
(N=5)
οΎ
οΎ
οΎ
6. Speech & Articulation Rate
Q. How many times did the child eat fast
food or pizza at school, at home, or at fast-
food restaurants, carryout or drive thru?
1: XX per day (range 1 - 15)
2: YY per week (range 1 - 84)
Interviewer time = 6.45 seconds
Items detected = 27
Words detected = 25
Speech Rate =
# ππ‘πππ πππ‘πππ‘ππ
π πππβππ π‘πππ‘)
=
27 ππ‘πππ
6.45 β0.0) π
= π. ππ ππππ π/ππππππ
Articulation Rate =
# π€ππππ πππ‘πππ‘ππ
π πππβππ π‘πππ‘)
=
25 π€ππππ
6.45 β0.0) π
= 3.88 ππππ π/ππππππ
β‘ Item Timestamping β’ Behavioral Coding
6
β Question
7. Automating Call Center Monitoring
IFD&TC 2019
Lew Berman, MS, PhD
lewis.berman@icf.com
Office: 301-407-6833
May 21, 2019