QueueMetrics - Tips and Tricks

Presented by:
Lenz Emilitri
Founder, Loway
@lenz
Effective
Call-Center monitoring and reporting
QueueMetrics Tips and Tricks
Asterisk Africa – Johannesburg, Feb 16, 2016

Contents
●Introduction
–Who we are
–What we do
●Running QueueMetrics
●Integrating QueueMetrics
●Open discussion

Who we are
Since 2004, Loway is leading the way in development of advanced software solutions for
the Asterisk® PBX.
With QueueMetrics we set up modern standards for call center performance
measurement.
Our Mission is to put Swiss passion for precision and reliability at customers' service.
Together with Loway, you can provide your clients with the most reliable, flexible, and
sophisticated call-center management solutions available today.

What we do
Our History:
● Started working with Asterisk in 2003
● Developed QueueMetrics in 2005
● Developed WombatDialer in 2012
● Launched QueueMetrics Live in 2015
Installed base:
QueueMetrics currently deployed in thousands of
call-centers worldwide
Average site: ~50 agents
Largest sites: ~1000 agents live (on Asterisk clusters)
WombatDialer deployed in ~300 sites
Average site: ~80 channels
Largest site: ~4000 channels
Client base:
● 30% USA / Canada
● 25% Europe
● 20% LATAM
● 10% Africa
● 10% Asia
● 5% Middle East

Contents
●Introduction
–Monitoring & Performance
–Configuration best practices
–Extending call tracking
–Quality Assessment (QA)
●Open discussion

The key process to improve run-time performace is:
Monitoring → Assessing → Understanding → Fixing.
●Monitoring builds a baseline of data
●Assessing lets you define how the problem appears
●Understanding – that's the hard part!
●Fixing is often trivial.
●Simply throwing better hardware at the problem will not usually help in
the long run.
●Don't wait for problems to start monitoring!
Monitoring & Performance

QueueMetrics was designed to be run in call-centers with up to
1,000 live agents.
Most performance problems in mid-sized CC's (up to 150 agents)
are caused by:
●Java memory (mis)configuration
●QM data caching turned off
●Broken MySQL indexes
●Wrong audio storage model
●JSON or XML-RPC wallboards
Performance problems?

Monitoring – Java Memory
QueueMetrics is written in Java.
●Java uses a fixed memory
pool that you need to configure.
●Target: no stop-the-world major collections. That's where the
system appears to hang.
●The RPM defaults are NOT okay for larger systems
●Sensible defaults:
-Xms4096M -Xmx4096M -server -XX:+UseParallelOldGC -
XX:PermSize=512M -XX:MaxPermSize=512M
●But you need to monitor them!

It is very easy to monitor remotely a production JVM.
Monitoring - jVisualVM
●Check heap size and parameters
●Monitor GC activity vs CPU usage
●Monitor threads:
–1 thread ~ 1 active request
–>5 users per thread
●Normal for memory to go up until GC'd
●Thread dump: what is QM doing now?
–Standard QM scripts let you take thread dumps
●CPU Sampler: why is it taking so long?

Internal tools: Admin page → System diagnostic tools
Monitoring – QM internals
RAM caches
–Data for all partitions
–SQL cache efficiency > 90%
–Cached strings < 1M
Live inspector
–See data being loaded
–Recent activity
–Quite expensive to run

Sometimes we find random MySQL issues.
●Symptoms
–Performance degrades strongly and all of a sudden
–Very high MySQL CPU usage and light Java CPU usage
–Lots of open threads
–Very high disk I/O on MySQL server
●Root cause
–Access indexes on tables present but broken / unused.
●What to do:
–Manual reindexing (drop indexes and recreate)
–Might take a while on very large databases
–..assuming I/O on the server is adequate (SSD!)
Monitoring – MySQL issues

If your audio recordings are all stored in a single folder, scan times can
get high...
●Symptoms
–QueueMetrics very slow when
opening the call pop-up window
●Root cause:
–Improper file storage
●Solution
–Split your recording in multiple folders, per day and queue, e.g.
/mount/audio/2016-01/30/queue1/….
–Use LocalFilesPerDay as the audio playback PM
You can also have multiple locations scanned in sequence, e.g. local
first, NAS next.
Monitoring – recordings

If you use a remote wallboard or any other piece of integration though JSON
or XML-RPC webservices, it should be well behaved.
●Symptoms
–Overall performance degradation
–VisualVM shows tons of open threads.
●Root cause
–Too many requests sent per unit of time
●Solution
–Services should cache local results if they need to feed multiple consumers /
wallboards at once
–Services must wait for an answer before submitting another – if not, it's sure
„Death by a Thousand Threads“.
Monitoring – Services

QueueMetrics offers a plain service to be used by
monitoring apps:
http://my.qm:8080/queuemetrics/sysup.jsp
●Lightweight JSON response
●Checks memory and DB connectivity
●Easy to integrate with existing monitoring platforms (Influx,
Riemann, Zabbix… you name it)
Monitoring – Sysup

We run hundreds of QueueMetrics instances in our service
QueueMetrics Live.
Monitoring – what we do
●Chose very basic metrics:
–total memory,
–free memory,
–web hits
–loader hits
–errors
●Easily spot 95% of issues
●Manually instrument if
needed

QueueMetrics is built around an all-encompassing security model
based on keys:
●System keys (access QM functions and pages)
●Custom keys
Users have a keyring:
–Default keyring = security class
–Can add an revoke keys for each user
–Masterkey
●All functionalities have a required system key
●All objects have optional keys ( = locks)
–Blank key = visible to everybody
The security model

Create different classes for different kinds of users
●Different clients
●Different access roles
●Start with the default ones (AGENT, USER, etc.)
Edit keys with the new editor:
●Plain-English view
●Set individual keys
●View inherited keys
Do not use the master key
Security best practices

Configuration
All configuration stored in the system folder.
●You can see which one is used in the License page
A text file holds the current configuration keys:
●File: configuration.properties
●General defaults
●General settings
●Picked up on user log-in
A different configuration file holds licensing information:
●File: tpf.properties
●Picked up on system restart

Config. best practices
Keep a backup of your current configuration!
Edit the configuration.properties file through Home → Edit system
parameters.
Edit common properties through the Explorer GUI: Home → Explore
system parameters
View the current configuration through the Home → System
Diagnostic Tools → View Configuration.

QueueMetrics was born to analyze inbound queues...
●In order to analyze outbound, we used special „outbound queues“
that used a piece of dialplan to do the tracking
●When you do outbound in a call-center context, you need a
„reason“ to distinguish calls
● You can originate outbound calls through phones or though the
GUI.
Outbound calls in QM

For best results...
●Use an underlying physical queue for outbound.
–Presence
–Pauses
–Hotdesking
–No calls!
●Name the queue „q-XXX“ where „XXX“ is the inbound queue
●Use the Icon page to originate, not phones
●If you need to place a lot of calls, use a dialer solution that works
with QM (e.g. WombatDialer)
Outbound – best practices

Agent configuration
Agents are used by QueueMetrics to
map names and properties to a PBX
code. An agent has...
●An agent code
●A name
●A supervisor
●A group
●A set of queues
●Etc...
Users are used by QueueMetrics to allow
access to the GUI.
A user has...
●A login
●A password
●Belongs to a class
●Has a set of security keys
●Etc...
What is the difference between an Agent and a User?
In order to let an agent use the GUI, its Login must match the Agent code (e.g. Agent/10
The QueueMetrics licensing model is based on Agents, not Users.

Agents – best practices
For best results...
●Use simple SIP phones
–Configure max lines = 1
–Check agent state on queues
●If using Local channels, avoid excess dialplan
●Use hot-desking
●Use the Icon page
–Easy log-on multiple Q's
–Easy status tracking
–Agent messages
–Outcomes

Asterisk logs the basic events for queue. QueueMetrics improves on this,
by tracking:
●Start of call (as opposed to queueing of call)
●DNIS
●IVR traversal and menus
●IVR goals
●Music on Hold
As they are not provided by Asterisk, they need to be added at the
dialplan level by adding INFO records.
Tracking more information

QueueMetrics is able to track the full life cycle of a call.
You need to:
●Send events when keys are pressed
●Send events when goals are reached
●Queueing is automatically considered a goal
Tracking IVRs

How often were your IVR
menus traversed?
When and where did people
stop?
How many self-service goals
were reached?
Tracking IVRs: Reports
How much did it take to
traverse each menu?

For many call-centers, measuring Music-on-Hold on answered calls is
important:
●Directly affects the perceived quality of service
●Anomalous MOH durations are an „alarm bell“ for underlying
problems
●QueueMetrics fully supports it
–Specific reports
–Call details
So… where is the catch?
●Asterisk does not produce these events for queues
Tracking Music-on-Hold

So how do we do it?
●Patches available for 1.4 & 1.8
–Never accepted into Asterisk
–Used in production in a number of 100+ channel CCs for years
–Hard to upgrade
●Passive monitoring daemon - MohTracker
–Uses AMI and works with any version of Asterisk
–Available as alpha software, free of charge
–Contact us if interested
Tracking MOH (2)

Contents
●Introduction
–Monitoring & Performance
–Configuration best practices
–Extending call tracking
–Quality Assessment (QA)
●Basics of QA monitoring
●Effective scoring
●Performance tracking

Quality Matters!
●Track calls
–Find issues before they find you.
●Track agents
–Improve training
–Assess strenghts and weaknesses
●Track queues
–Are we doing what we are expected to be doing?
–Can we show it?
If you are not using QA now, you are losing out.
QA – Introduction

Each call is graded on a set of metrics you define
–Each metrics has its own Engagement Code
–They map a numeric score (0-100)
–Same metric be used in multiple forms
Metrics are grouped into Forms
–Up to 10 sections with up to 130 questions
–Metrics are grouped into four grade bands (Issue, Req.Impr, Meets exp.,
Exceeds exp.)
–For each form, a score is computed
–Forms are immutable (more or less)
–Each call can be graded on multiple forms
QA – General ideas

●Form input is flexible
–Questions can be hidden or shown
–Questions can be weighted in order to form a score
–Shortcut questions
●Immediate agent feedback
–Receive a task when a call is scored
–Agents are immediately engaged
–Agents must acknowledge or dispute it
QA – General ideas (2)

QA is a process and requires some planning.
●Define what you expect
–Don't make things too complex at first
–Tell your agents!
●Plan a call review process
–Don't oversample issues!
●Define initial QA review targets
●Plan in advance for corrective action to be taken
QA – Getting started

A simple form to review issues
●Track whether the problem is solved
●Track the sex of the caller
●If problem not solved…
–Track conversation metrics
●Different data input
●Non-scoring questions
●Show-hide questions
QA – A real-life sample

Start by defining questions...
●Engagement code
●Description
●Weight
●Shortcuts
●Value types
–Number
–Yes/No
–Combo (drop-down)
QA – Creating items

Then create forms...
●Security keys
●Allowed queues
●Open / Locked
●Sections
●Thresholds
QA – Creating forms

Define which items are a part of the form.
QA – Creating forms (2)
●Group items into
sections
●Move up and down
●Score contributions
●Visibility
Once a form has items in it, it cannot be changed.

●Form selector
●Call information
●Recordings
●Timer
●Items per type
●Overview per section
●Score
●Questions & notes on questions
Anatomy of a form
What does a form look like?

Notes
●General notes
●Notes per question
Audio player
●Speed controls
●Playback controls
●Markers
Anatomy of a form (2)
What does a form look like? (continued)

Run Report → Queue details → Call Detail → QA
Form appears → Listen to the call → Fill it.
QA Input
Entering QA is really simple.

Results:
●Per agent
●Per queue
●Per group
●Per analyst
Showing:
●Averages
●Items by threshold
QA Results
To see results: Run QA reports → Criteria

Divided by section:
●By agent:
–Calls
–Average per question
●By queue:
–Calls
–Average per question
QA Results (2)
More results...

See all the graded calls that contribuited to a result.
Click on a call to see its QA form (questions, notes, associated audio files).
QA Results (3)
How did a result came to be?

●Overall summary
●Scoring questions
●N. forms scored
–Averages
–% per threshold
–% shortcuts
●Non scoring
–Averages
–Histograms
QA Results - Summary
Overview of results

Deciding which calls to grade
Just clicking on random calls is not especially effective.
●Poor ergonomics
●Selection biases:
–Issues / „interesting“ calls
–Short / long calls
–First / last calls
Solution:
●Create a policy
–By call outcome
–By agent group
●Use the Grader's page
–Weighted random sampling
QA – Grading calls

Find efficent sets of calls to be graded
●Random sampling in call space
●Prefers calls matching multiple criteria
●Multiple Quality Analysts can work on
the same issue at once
●Shows target statistics
●First line of defence against quality
issues
–Ask for coaching
–Ask for CBTs to be taken
QA – Effective grading

Long-term agent performance management
●Agents need to be managed in order to work effectively
–Performance targets
–Quality policies
–Agents need training
●Agent lifecycle
–Training
–Validation period
–Review
–Production
–Periodic Review
QA – Agent Performance

Comprehensive Performance Rules
●Performance
–Sales, SPH
–Contacts
●Call duration
–Averages
–Anomalies
●Quality results
Red and yellow alarms with relative importance scores.
QA – Agent performance

The Performance tracker lets you:
●Apply Performance rules and find anomalies
●Send training and coaching
●Move agents between groups
●Be reminded for further review
QA – Performance tracker

Contents
●Introduction
–Automated QA input
–Queue recalls with WombatDialer
–Handling VIP callers
●Open discussion

QA is an effective process, but...
–Reviewing calls manually is very expensive
–Most calls are not very informative
–Biased sampling
–What if we could automate it?
●How can this work?
–Agent tranfers call to IVR at the end of the interaction – 70-80% of callers will accept
–IVR gathers simple information („Are you satisfied?“ - „Problem solved?“)
–Asterisk pushes this information to QM as a QA Form
●Targets:
–You have a way to review issues immediately
–You can monitor quality fairly and continuously
Surveys: Automating QA

●Create a QA form called „SATISF“
–One section
–One single yes/no question - „Are you satisfied with the
interaction?“ (item SAT)
●Create a remote user in QM
–User „qasubmit“ password „passw0rd“
–Class ROBOTS
–Custom security key „QATRACK“
Surveys – QM set-up

Asterisk issue: IVR happen on a different channel:
●When calling the queue, store the original UniqueId of the call and
the Queue used on inheritable channel variables
●Allow unattended transfers on the queue (# + xxx)
●Download script pushQA.sh from the Open QueueMetrics Add-
ons on GitHub
●Make it executable and edits its configuration
●Create a simple IVR script to transfer to
Surveys – Asterisk set-up

exten => _999,1,Answer
exten => _999,n,Background(ivr_satisfaction)
exten => _999,n,Wait(10)
exten => _999,n,Hangup()
; Not happy
exten => 0,1,Playback(beep)
exten => 0,n,System(/var/lib/asterisk/agi-bin/pushQA.sh ${UID} ${QUE} 0)
exten => 0,n,Playback(thank-you)
exten => 0,n,Hangup
; Happy
exten => 1,1,Playback(beep)
exten => 1,n,System(/var/lib/asterisk/agi-bin/pushQA.sh ${UID} ${QUE} 100)
exten => 1,n,Playback(thank-you)
exten => 1,n,Hangup
Surveys – Asterisk set-up
exten => _900,1,Wait(1)
exten => _900,n,Answer
exten => _900,n,Playback(queue-welcome)
exten => _900,n,Set(CHANNEL(musicclass)=mymu
exten => _900,n,Set(__QUE=myqueue)
exten => _900,n,Set(__UID=${UNIQUEID})
exten => _900,n,Queue(${QUE},t,,,300)
exten => _900,n,Hangup
Asterisk dialplan
Push QA to QueueMetrics
Store Queue & UID

Your services have a known peak time.
●You need to staff agents based on the load at peak time
●Your wait time SLA is stressed at peak time
●Harder interaction with callers who have waited too long
Solution: offer people the choice to be recalled
●Better SLAs – less staff required
●Happier customers
●Can be extended to web-based „call me back“
Automated recalls

Caller is offered the option to press '1' to be recalled
●If they do, a script starts to get their phone number
●When done, the number is piped to WombatDialer
WombatDialer monitors the recall queue for free agents
●When agents are available on it, recalls starts
●Wombat knows how to handle busy, no answers, etc.
When the call is started...
●A script plays a welcome message on connection
●The call is queued on the Recalls queue
Auto recalls: the logic

QueueMetrics
●Situational awareness
●Agent management
Asterisk
●Script to track caller's numbers
●Script to greet people being recalled
WombatDialer
●Monitors presence on Asterisk queues
●Implements recalls
●Handles busy, no answers, invalid numbers, etc.
Auto recalls: components

QueueMetrics configuration
●Create two queues – one for inbound and one for recalls
●Monitor them through the Real-Time page
●Supervisor moves agents to and from the Recalls queue based on
the state of the Inbound queue
●Everything else happens automatically
Auto recalls: QM

WombatDialer
●Runs multiple independent campaigns in the background
●Receives numbers to be called and extra information for handling
them over HTTP
●Monitors agent continuously
–Are they available? Paused? Busy?
●Handles recalls with sensible rules
–Retry in 10 minutes if you get a busy
–Retry in 30 minutes if the number is free
–Call only during allowed times
●When successful, connects to the welcome script
Auto recalls: WombatDialer

●Solution can be implemented in very little time
●Does not require extensive changes to the PBX config
●Use the same tools you use for inbound
●Very extensible solution
–Reverse IVRs
–Voice synthesis
●Excellent ROI
●
Recalls – all together

How can we handle VIP callers?
●Special low-latency queues
●Different products
●Automatic routing of open cases
●How can we identify callers?
Usually done with dedicated DIDs, but...
●Hard to scale to hundreds or thousands of cases
●We can definitely do better!
VIP callers

VIP callers in QM
QueueMetrics offers „Known numbers“, an easy-to -query
database of known numbers.
●Associates an action and an identity
to the caller
●Action can be time-limited
●Optional „agent affinity“ to first try the
agent who is currently handling the
case
●Database can be fed through JSON
API

VIP callers in QM (2)
For security reasons, you need a special user to acces
QueueMetrics through the APIs.
●Create a remote user in QM
–User „pbxapi“ password „api123“
–Class ROBOTS
–Custom security key „PBXAPI“

VIP callers in Asterisk
You query QueueMetrics before routing a call:
same => n,Set(CURLOPT(hashcompat)=yes)
same => n,Set(NUM=${CALLERID(number)})
same => n,Set(URL=http://my.qm/queuemetrics/numberLookup.do
?mode=hash&user=pbxapi:api123))
same => n,Set(HASH(resp)=${CURL(${URL}&number=${NUM})})
same => n,Set(CALLERID(name)={HASH(res,name})
same => n,GotoIf($["${HASH(resp,action)}" = "VIP"]?supervip)
same => n,GotoIf($["${HASH(resp,action)}" = "BLACKLIST"]?blklist)
same => ....
So with one single call you get:
●Name of the caller
●VIP or Blacklist
●Agent affinity
And you can route accordingly!

Thank you for attending!
QueueMetrics www.queuemetrics.com
Loway www.loway.ch
A real programmer puts two glasses on his bedside table before going to sleep.
A full one, in case he gets thirsty, and an empty one, in case he doesn’t.

Contact us
pinterest.com/QueueMetrics
twitter.com/queuemetrics
linkedin.com/company/loway
slideshare.net/QueueMetrics
youtube.com/user/QueueMetrics
plus.google.com/+QueuemetricsCallCenter
facebook.com/QueueMetrics
loway.ch/rss.jsp

QueueMetrics - Tips and Tricks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to QueueMetrics - Tips and Tricks

Similar to QueueMetrics - Tips and Tricks (20)

More from Clarotech_Events

More from Clarotech_Events (11)

Recently uploaded

Recently uploaded (20)

QueueMetrics - Tips and Tricks