This document discusses automating telecom customer support processes using a system called DARE (Diagnosis And Repair Engine). Key points: - DARE aims to simplify and standardize support, improve first-time resolution, and increase efficiency. It provides a single system for automated and interactive support. - DARE collects data from various backend systems, applies business rules to diagnose issues, and provides guidance to call center operators and field technicians. - The system aims to improve customer experience, support consistency, and allow remote diagnostics and repairs to reduce field visits.

1. Telecom Customer Support
Process Automation
erwin.de.ley@isencia.be

2. Setting the scene
We work with Belgium's largest telecom provider
Traditional telecom services :
• voice, internet, IDTV, mobile
Nomadic services :
• personal cloud, streaming, video-on-demand, gaming, …
iSencia Belgium NV 2
Millions of customers

3. Setting the scene
Technical support & repair
• 15K calls / day
• 10K field interventions
• 700 call center operators
• 2500 field technicians
After-sales, billing support
• 30K calls / day
• 1500 call center operators
iSencia Belgium NV 3

4. Setting the scene
The network is the service.
Geographically distributed network, multitude of street cabinets, devices, …
Many variations of complex products and line technologies
Rapidly evolving products and services
Long half-life of old products and devices
iSencia Belgium NV 4

5. What was in the past
iSencia Belgium NV 5

6. Old-style repair support process
Rem.intervention
Customer calls 0800-...
Gets an operator (after a while)
Operator tries to understand what the customer's problem is
And starts consulting all kinds of BSS/OSS support systems, launching
network measurements, retrieving equipment states...
Manual analysis of the information
On-the-spot diagnosis and resolution decision
iSencia Belgium NV 6
Call
Manual
Collect
data
Manual
Analyse and
diagnose
Field intervention
Solution

7. This doesn't help either
Large organization with many separate teams
Each historically had own tools and methods
Customers get sent around between teams
Operators and technicians are often subcontracted
=> Volumes i.o. customer satisfaction
BSS/OSS systems owned by many application teams
Mix of legacy systems, new technologies,
packages, in-house dvp ...
iSencia Belgium NV 7

8. Issues
Customer often fails to correctly describe problem
Manual data retrieval and line measurements are slow
Customer and operator are blocked waiting for results
Approach and results depend on individual skills
Long average call handling times
Risk for errors & inconsistent support decisions
Technician's tools inconsistent with operator decision
Low first-time-right ratio
Low customer satisfaction
Huge learning curve and high recurrent training costs
iSencia Belgium NV 8

9. Improving the support processes
iSencia Belgium NV 9

10. Where do we want to go?
iSencia Belgium NV 10
Customer Experience
• Consistent support answers and decisions
• Improved FTR
Simplification
• Integrated view on customer, services, network
• Automate steps where possible
• Support the interactions with the customer
Efficiency
• Always access to up-to-date best practices & formalized knowledge
• Shorter handling times
• Less rework, training, knowledge transfer

11. How to get there
DARE : Diagnosis And Repair Engine
Single system for automated and interactive support processes
• Call Center Operators for retail and professional customers
iSencia Belgium NV 11
• Field Technicians
• OLO Operators and Technicians for regulated and commercial products
• Customer Self Repair
• Proactive repair, automated line profiling, ...
Consolidated, up to date business logic with 360°customer view
• Frequent business logic updates
• Data collection and remote repair in the network, even in home equipments
• Integrated data views and interactive guidance

12. How to feed it
Call center coaches and field intervention experts extract
support&repair cases and best practices
iSencia Belgium NV 12
Organize support tasks :
• Automated data collection
• Interactive forms
• Analysis modules
• Diagnostic decisions
• Advises for (automated) repair actions

13. How to feed it
Define and build a library of :
• Data connectors to OSS/BSS systems
iSencia Belgium NV 13
• Analysis modules
• Reusable sub-processes
Define domain-specific support processes :
• reusing connectors and modules
• specializing diagnostic decisions

14. Example
Refactoring the end-2-end process for repair support
iSencia Belgium NV 14
X
DARE
IVR
1
2
3
4
4

15. Pretroubleshooting &
troubleshooting
While customer on queue :
• automated pre-troubleshooting
• data collection
• rules-based problem analysis
• root-cause diagnosis
• identify and prioritize troubleshooting processes
• advise on possible repair actions
iSencia Belgium NV 15

16. Pretroubleshooting &
troubleshooting
When the operator gets the customer call :
• diagnostic report available
• detailed underlying data
• guided interactive troubleshooting
iSencia Belgium NV 16

17. Diagnostic report
iSencia Belgium NV 17

18. Interactive guidance
iSencia Belgium NV 18

19. Decision examples
iSencia Belgium NV 19
IVR
Filtering and routing decisions based on
• Global outages, planned works in the area ?
• Customer info : Bad payer? SLA? …
• Customer's products
• Language

20. Decision examples
Pre-troubleshooting process
• Check customer's ordered products
• Measure the physical line characteristics
• Read network configuration, device status
• Reach to customer modem and other CPEs
• Check service configurations FIA, IDTV, VOIP, …
• Compare configuration in the network and service management
systems with expected values
(e.g. activation of TV packages, VOIP features, ...)
iSencia Belgium NV 20

21. Decision examples
Troubleshooting process
• Is the PTS diagnosis what the customer is calling for?
• Can the issue be solved by the operator?
iSencia Belgium NV 21
• Must a ticket be created?
• Is a field intervention required?

22. And now a bit more technical...
iSencia Belgium NV 22

23. DARE system overview
IVR
Service
Gateway
BIQ
E2E testing
SALY UTS/eTS
Callback
manager
Pretrouble shooting
ABR
Technical situation
iSencia Belgium NV 23
Backends
NPS
TBF
Procodac
Hlog & XDSL
parameters
SAFE
expected attenuation
ROSY
(SALY GUI)
DSLAM 4TEL
BAS
ABD
FH
NFH
PSTE
Customer facing
serv/Assigned prod
LAM
Local Address
Management
SGW
Modem Interaction
Homer
Skynet accounts
SOCCABIS
Skynet
login/password
DARE
DARE Rules Engine DARE Process Engine
DARE Manager
Syst.
Admin
2
STB
HGW
IMS
NGN
UMS
…
Product catalog PILA
LLD
OFA WFM
Operator Cockpit
CLI/CLE
1
Troubleshooting
Console Advisor
DATAMINER
STB Info

24. DARE modules
Built on iSencia's
Passerelle EDM
Orange boxes : extensions
Process
editor
iSencia Belgium NV 24
Passerelle
process engine
Jboss Drools
rules engine
Asset
repository
Application
modules
Backend
adapters
Task form entry
raw data views
Role-based
security
Process
scheduler
Application
Preferences
Execution
traces
Support
modules
Web
Services
(SOAP & REST)
Operator
Console
Admin
Manager
Interface
layer
Backend
simulator
Diagnostic
processes
Analysis
rules
End-user Console
Task form entry
Web
Services
(SOAP & REST)

25. Asset repository
• Process models, rules packages, images, translations …
iSencia Belgium NV 25
• 2-digit versioning schema
• Packaging/grouping of related groups of assets
• Root-level : Project
• Simple preparation, activation, roll-back of new project releases
• Export/import of project packages between test envs & PROD

26. Backend adapters
iSencia Belgium NV 26
• >150 adapters
• Typically slow tasks (2s – 2min)
• Asynch/synch bridge with internal buffering, resource pools
(minimize internal resource blocking)
• Configurable throttling settings to protect backends from overload
• New/updated adapters can be uploaded without downtime

27. Application preferences
iSencia Belgium NV 27
• Stored in DB
• Web UI to import/export, CRUD, …
• DARE modules & services can be reconfigured “live”
• Used for analysis thresholds, refdata,
backend connection settings etc

28. Some usage scenarios
iSencia Belgium NV 28

29. Collaborative process definition
Workflow
editor
Task form entry
raw data views
iSencia Belgium NV 29
Take advantage of :
• Web-based access
• Role-based security
• Fully-featured graphical model editor
• Asset versioning, personal workspace,
model locking
• Activate,update,rollback
sets of related assets atomically
• Test & simulation modes
Passerelle
process engine
Drools
rules engine
Asset
repository
Application
modules
Backend
adapters
Role-based
security
Process
scheduler
Application
Preferences
Execution
traces
Support
modules
Web
Services
(SOAP & REST)
Operator
Console
Admin
Manager
Interface
layer
Backend
simulator
Workflow
processes
Analysis
rules
End-user Console
Task form entry

30. (Don't) work at night
Workflow
editor
Task form entry
raw data views
iSencia Belgium NV 30
For automated/pro-active/batch runs
• Schedule runs day-and-night
• Not just remote in location, but also in time !
• Integrate automated self-diagnostics
in your workflows
• Automated notifications when needed
• Role-based security for secure support
and collaboration
• Connect to your sysadmin monitoring tools
via adapters, web-services, event logs etc
Passerelle
process engine
Drools
rules engine
Asset
repository
Application
modules
Backend
adapters
Role-based
security
Process
scheduler
Application
Preferences
Execution
traces
Support
modules
Web
Services
(SOAP & REST)
Operator
Console
Admin
Manager
Interface
layer
Backend
simulator
Workflow
processes
Analysis
rules
End-user Console
Task form entry

31. Execution monitoring & support
Workflow
editor
Task form entry
raw data views
iSencia Belgium NV 31
The platform enables :
• Browse active workflows
• Integrated execution tracing,
provenance info
raw data views, graphical execution
traces,
Task form replay ...
• Secure pause/stop/resume
• Collaboration between business
analysts
and IT support
• Testing options : mock testing,
replay etc
Passerelle
process engine
Drools
rules engine
Asset
repository
Application
modules
Backend
adapters
Role-based
security
Process
scheduler
Application
Preferences
Execution
traces
Support
modules
Web
Services
(SOAP & REST)
Operator
Console
Admin
Manager
Interface
layer
Backend
simulator
Workflow
processes
Analysis
rules
End-user Console
Task form entry

32. Traceability approach
• Each process execution gets unique ID,
communicated to client apps
• Each task's lifecylce is stored
iSencia Belgium NV 32
• With backend data,
analysis results etc
• Internal monitoring,
timing & error statistics

33. Execution browser
& raw data views
iSencia Belgium NV 33

34. Graphical execution trace
iSencia Belgium NV 34

35. Finally, log files!
• All those application features are nice but...
• In the end sys admins want log files!
• Errors, timeouts etc get logged with specific error codes
• Monitoring tools have their own rule-based decisions to react on error
frequencies, assign tickets to application teams...
iSencia Belgium NV 35

36. Lessons learned
iSencia Belgium NV 36

37. Graphical tools: Sales pitch vs facts
• Graphical process models, rules-based logic …
often presented as a way to let non-developers build
business solutions
• We've noticed that this works for contained scopes
• But for larger contexts, with long-term maintenance,
performance requirements etc …?
iSencia Belgium NV 37

38. Graphical tools: Sales pitch vs facts
What works best in our situation :
• Close collaboration
• Process experts formalize the definitions
• Functional experts turned developers maintain :
• Interactive processes with simpler decision rules
• Threshold values for complex decision modules
• Reference data
iSencia Belgium NV 38
• IT team maintains :
• Complex, high-throughput flows, modules
• Builds adapters
• & coaches the business developers

39. Agility and Testing
• Built for fast implementation & maintenance cycles.
But errors have large impacts.
• Gradually evolved to traditional release-bound test cycles
for non-minor changes
• Live changes still allowed for :
iSencia Belgium NV 39
• Threshold changes
• Ref data
• Urgent rules fixes

40. Data issues
• Data distributed across many applications, databases...
• For repair processes we need live data from
network, devices, ongoing provisioning flows, …
• Confronted with many intermediate layers & conventions
• Backend systems all have different backgrounds, data
structures
• Data quality issues become visible in your projects
(as in all integration solutions)
• Where to find the authorative data source
(batch replications, plain duplication, ...)
iSencia Belgium NV 40

41. Data issues
• We store lots of data per diagnostic run
(approx 150M data items / day)
• Maintain 6 weeks of results in the operational DB
• Older weekly partitions off-loaded to DWH
• Decision analytics done there
iSencia Belgium NV 41

42. Organization issues
• Meeting culture, budget allocations etc all slow down the
real implementation work
• Many different teams are impacted
• They may see your integration work as a threat
• End-users will see your work as threat
(your decision automation project is paid for
by cost-cutting programs)
iSencia Belgium NV 42

43. Thank you
iSencia Belgium NV 43