3. Why learn about ACCEPTANCE & KPI…s
› Acceptance is part of Business
Critical Terms & Conditions
› Control of KPI commitments is
control of cash flow and
profitability
› Ericsson had 80BSEK locked in
Acceptances in 2008
› If all projects finished one day
earlier = 500 MSEK/day for
Ericsson
4. Scope and objectives
› Explain how Acceptance & KPI’s are
impacting business
› Describe Ericsson view on LTE
Acceptance principles and KPI’s
› List LTE KPI approach and point to
further information
Objectives
Scope
› Customer and Ericsson view on
Acceptance and KPI’s -> business Impact
› KPI Based Acceptance
› Understanding the KPI’s
› LTE KPI’s
6. LTE ACCeptance principles
› HW should be accepted on delivery. The LTE product HW is tested at the factory
according to Ericsson quality procedure. Normal claims and warranty naturally
apply.
› HW Activation Codes should be accepted when the codes are made available to
the customer in ELIS, the license management system.
› SW should typically be accepted after successful demonstration in the customer
lab, following the Demo Object List (DOL). This should include the mechanisms for
both HW Activation Codes, and for SW Feature Activation Licenses.
› Installation services should be accepted when the node has been installed and
stand-alone tests have been successfully completed.
› Integration services should be accepted when the node has been configured and
integrated into the network, and stand-alone tests have been successfully
completed.
› Tuning services should be accepted when tuning service has been completed
and documented, ideally on a cluster per cluster basis.
Separate payments for HW, SW and Services !
7. LTE Acceptance tips
Not recommended:
› Contractual formulations including delay of fraction of payment of
HW/SW coupled to fulfillment of service
› Contractual formulations including liquidated damages (referring to
payment of HW/SW) coupled to fulfillment of service
Wrong acceptance approach has:
› Effect of tied capital –increases our Employed capital/Net sales ratio
› Effect of lowered margins – increased costs
If we tie up KPI…s with the HW and SW sales we might end up like this...
8. CASH FLOW Scenario
Upfront 15%
HW
75%
Delivery
SW
75%
Delivery
Services
75%
Monthly
Progress
Final Acceptance
10%
Recommended Terms
10%
75%
15%
Penalties
Discounts
!
High risk
Bad Cash flow
Cost blow out
KPI Based Acceptance Terms
Upfront 15%
HW
x%
Delivery
SW
y%
Delivery
Services
z%
Progress
Final Acceptance
10%
KPI
Targets
25% plus
50% or less
KPI linked
payment risk
11. Why ARE CUSTOMERS DEMANDING KPIS ?
It…s not just about performance IT…S CONTROL AND COST!
TO CONTROL...
RISKS COSTS
› Internal pressure
› Don’t understand the
technology -> LTE !
› Force Ericsson to fix their
performance issues
› Avoid risk, shift risk to
Ericsson
› Own cash flow
issues
› Extra services at no
cost
› Penalties, discounts
or things for free
› Avoid paying
Ericsson
12. Ericsson Driving Forces
› Profitability, ability to fulfill contract
› Cash flow, timely invoicing
› Separate payment terms for HW, SW, and Services
› Close contract with clear responsibility, scope and acceptance
criteria
› Control of contract fulfillment
› Additional sales
13. Contractual Acceptance directive
1. Acceptance should have a defined scope, procedure and
timetable agreed in advance,
2. To be based upon objective and measurable criteria
3. KPI based acceptance should be limited to network and
technology consulting part of rollout contracts and limited to KPIs
defined in the KPI library at the Acceptance Gateway.
4. If products or parts are put into operation, automatic final
acceptance of such parts
Reference: Business Critical Terms&Conditions documents
14. ContracTual considerations and mu…s
› Contractual Considerations: The MU shall always strive to follow the
Acceptance Principles (Acceptance Approach and the Business Critical Terms
and Conditions).
› Restriction: If the MU deviates from the Acceptance Principles, it should
clearly state the deviations and estimate the risks and associated impact on
the business case for the contract.
Deviations from the Acceptance Principles must be presented to the LTE Deal
Strategy meeting, pre-SDP2.
The MU is responsible to secure required input, Price Governance, material (PPT,
Xls )
Set up of this meeting is done by contacting Christer Gustafsson or Linda Lindberg
(manager EAB/FRP).
16. ACR
Strategy
Payments
Satisfaction
KAM
CSR
System
Solution
Service
Solution
CFR
Cash Flow
Delivery
Execution
Ensure link between
Terms of Payment
and Acceptance
Procedure.
(Particularly Cashflow)
Align Services (SoW)
and Solution with
Acceptance
Commitment incl. KPIs
Ensuring KPI Based
Acceptance is
integrated in total
Acceptance Procedure
Core 3 Responsibility
KPI Based Acceptance
• Overall Decision
• Ensure Acceptance risks are
understood by team and push
for flow between solution and
Acceptance
• Promote use of KPI Champion
KPI Based Acceptance Champion
18. Basic facts KPI…s
› Operator thinking: More KPI’s, Operator
is getting more control
› Ericsson thinking: More KPI’s, higher
costs, longer procedure, risk for delays in
payment and penalties
› LTE is new, we can expect a lot of KPI’s
since operators want to protect
themselves
› We can be active and suggest KPI’s,
rather than wait for them - We know what
we are doing
› Right support is provided from the LTE
start – look at the Acceptance gateway
What KPI’s and what values to discuss
Acceptance method
19. WHAT IS IMPACTING KPI VALUES ?
TARGETS
PRE-CONDITIONS
HOW TO
MEASURE
CONDITIONS
EXCLUSIONS
PROCESSES
21. Kpi…S AND PROJECT TYPES
› In Large Rollout Projects where we have a lot of
risks, unknowns and dependancies and where
services are small part - Minimise KPI risks and
costs
› In Standalone projects where we have mostly
services, and we have control - Use KPIs as
business driver
KPIs can result in good business !
23. LTE kpi Acceptance
Types of Acceptance
› The scope of a contract will determine the type of acceptance used
› Four different types of acceptance are proposed in this document
Product (Supply) Acceptance
Proof of Concept Acceptance
Network (or Cluster) Acceptance
Stability Acceptance
› A network contract may stipulate more than one of these types of
acceptance
› It is desirable to minimise the acceptance testing – it can be
consuming and costly
KPI’s handling will depend on the type of Acceptance
24. There are 4 KPI recommended for use in LTE acceptance
Session Setup Success Rate
Session Abnormal Release Rate
ONLY used for drive tests with short calls
or
Minutes per Abnormal Release
Used for stats based KPI & drive tests with long calls
DL User Throughput
UL User Throughput
Recommended KPI
For use in LTE Acceptance
1
2
3
4
25. Recommended KPI
Recommended Location for KPI Measurement
No Recommended KPI
Controlled
Environment Golden Cluster
Network
(Cluster)
Drive Test
Counters /
Statistics
1 Session Setup Success Rate X X X X
2a Session Abnormal Release Rate X X X
2b Minutes per Abnormal Release X X
3 DL User Throughput X X X
4 UL User Throughput X X X
Product (Supply) /
Proof of Concept
Acceptance
Network
Rollout
Acceptance
Network
Stability
Acceptance
26. Recommended KPI example
Session Setup Success Rate
98.00%
99.0%
99.3%
99.50%
97.50%
98.00%
98.25%
98.50%
96.00%
96.50%
97.00%
97.50%
98.00%
98.50%
99.00%
99.50%
100.00%
L10A L10B L11 L12 =>
Lab / Controlled
Field / Drive test
Comments and Recommendations for Acceptance:
› SSSR should be tested using terminals which have undergone IODT with Ericsson
› Short calls are recommended for field test acceptance
› Lab and Controlled environment testing is to be undertaken in good radio conditions
(RSRP > -85dBm, SINR > 6dB)
› Exclusions should be included for drive testing where the setup failure is caused by
a factor beyond Ericsson’s control (eg. No coverage due to radio network design
constraints)
27. Throughput KPI
High Level Summary: Acceptance Process
DL Throughput (EPA 5Hz) vs SNR 10MHz
0
10000
20000
30000
40000
50000
60000
70000
80000
-5 0 5 10 15 20 25 30 35 40
SNR (dB)
Throughput(kbps)
Simulation value
Maximum Acceptance Value
Note: Graph is only an example, real values should be
calculated using Coverage and Dimensioning
Guideline
› Special methodology for the
KPI User throughputs
› Applications, measurement
methods, exclusions
explained
› Theoretical max value
calculations (based on NW
configuration) vs. Practical
value
› Throughput KPI’s are well
documented
› Safe way for Ericsson to
take lead in the KPI
negotiations
28. Optional KPI
For use in LTE Acceptance
There are 6 optional KPI for possible use in LTE acceptance:
Simultaneous Connected Users per RBS
Intra LTE HO Success rate
Round Trip Time (RTT)
Terminating Session Success Rate (Paging)
Control Plane Latency (Idle to Active)
Cell Availability
9
8
7
6
5
10
30. ERTIES
ssing, 'Finish' button: Goes to Next Slide
ling, 'Finish' button: Goes to Next Slide
user to leave quiz: At any time
may view slides after quiz: At any time
may attempt quiz: Unlimited times
32. Summary LTE Acceptance & KPI…s
› KPI’s are often financial driver, rather than technical
› Keep it simple, minimize number of KPI’s
› Consult "LTE Acceptance Toolbox" for overall
procedure
› For KPI’s listing (Recommended and Optional), and
their definitions consult "Acceptance KPIs - methods
and values"
› KPI Target values are last in chain - consequence of
pre-conditions
› Special care is given to the throughput related KPI’s
– systematic and structured approach
› Negotiate with customer
› For more information about KPI Based Acceptance
please contact Graham Allen or Marius Sirbu
