Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright 2009 QuEST Forum. All Rights Reserved.
1
1
TL 9000 Measurements and Requirements
Interactive Workshop
APAC Best ...
Topics for Today
• Questions were provided in advance in
the following Areas:
– General
– QuEST Forum Membership
– TL 9000...
“GENERAL” QUESTIONS
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights Reserved.
3
Acquiring TL 9000 Certification
• Tell me about how my business can
become TL 9000 certified
• How long does certification...
Copyright 2012 QuEST Forum. All rights reserved. 5
TL 9000 Implementation Path
6. Continual
Improvement
5. External
Assess...
6
I would like to have realistic examples of
measures to meet TL 9000 Requirements:
Access Examples Utilizing “Links” ( tl...
What Kinds of Questions Does QuEST Forum
Get on TL 9000 Measurements and Trends?
Copyright 2009 QuEST Forum. All Rights Re...
General Questions –
ISO 9001:2015 and TL 9000
Requirements R6
• The requirements for documentation and
records have been r...
ISO 9001:2015
Status Update
9
CD available
2013
DIS available
05/2014
FDIS expected
AFTER 07/2015
ISO 9001:2015 to
be publ...
QuEST Forum ISO 9001:2015 Position
What actions have QuEST Forum
taken regarding ISO 9001:2015?
Solicited, reviewed and submitted
comments on ISO 9001:2015 C...
“Key” Detailed Changes:
• New version of Figure 1; ISO 9001 still follows PDCA
• “documented information” remains
• “proce...
General Questions –
Third Party Process
• How does QuEST Forum take
action with all certification
bodies in order to have ...
Regional Findings / Audit Day
ABCB Sub-Team Review:
Overview of Sub-Team Charter
• CB Statistics
• AB Approval / Re-approval
• Code of Practice
– Includ...
ABCB Sub-Team Review:
Validation Audits
• Plan: 2008-9
– Decision to investigate the effectiveness of
CB audits in identif...
Action Thresholds
Copyright 2009 QuEST Forum. All Rights Reserved Copyright
2009 QuEST Forum. All Rights Reserved.
18
90%
...
ABCB Sub-Team Review:
Validation Audits
• Act: 2014-15 – Approved
1. Code of Practice Changes
• Additional qualifications ...
ABCB Sub-Team Review:
Validation Audits
• Act: 2014-15 – Proposals Under Discussion
1. Upgrade CB Technical Review of audi...
General Section – Other Questions
• What is the difference between "Tier 1 Supplier" and "Tier 2
Supplier"?
– It depends!!...
QUEST FORUM MEMBERSHIP
QUESTIONS
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights Res...
TL 9000 Certification & QuEST
Forum Membership Questions
• TL 9000 Certification does not require
Forum membership
• QuEST...
QuEST Forum Membership –
Other Questions
• Can an Operator Receive IA/BIC/WIC for their
suppliers and the suppliers of oth...
TL 9000 CERTIFICATION
QUESTIONS
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights Rese...
TL 9000 Certification –
Audit Days
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights R...
Is there a case where a service provider
requires its suppliers to require their
suppliers to meet TL 9000?
• Yes! - See E...
TL 9000 Certification
- Scope Questions
• The clause 3.2 b) in the Requirement Handbook requires in the Scope
Statement th...
TL 9000 Certification
- Other Questions
• An organization wants to limit its scope of registration to products for one
cus...
APPLICATION OF TL 9000
ADDERS
QUESTIONS
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rig...
Inputs to TL 9000 from Other
Standards
• What part of the items in TL 9000
Requirements and Measurements reflect
the stand...
Copyright 2012 QuEST Forum. All rights reserved. 32
TL 9000 is based on ISO 9001
ISO
9001
Telcordia
(Bellcore)
ISO/IEC
122...
33
CMMI and TL 9000
Full TL 9000 Certification Approximates CMMI Level 3. Standards
Comparisons including CMMI can be foun...
7.1.C.2 Disaster Recovery
• We have Business Continuity Plan for disaster
recovery planning. Could it be applied for this
...
7.1.C.3 Product Security
• As we are merely producing and supplying the
products based on the specification which the
Tele...
7.3.1HS.1 Migration Planning
• As we do not intentionally change the OS of the
product and just follow the customer reques...
7.5.5.S.1 Software Virus Protection
• As we merely produce/supply customer (service
provider) specified products, we take ...
8.2.4.H.1 Periodic Retesting
8.2.4.H.2 Content of Testing
8.2.4.H.3 Frequency of Testing
• Although our production plant i...
MEASUREMENT
QUESTIONS
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights Reserved.
39
• What if the "units shipped" period does not actually
encompass the past full 12 months, because of a
newly introduced pr...
• As per the product category 6.2.1.2.4 Wireless Terminal Software Applications,
is it not necessary to register this prod...
Measurements Questions –
Linked Certifications
• My company and its subsidiary company are now on the way to get TL 9000
c...
DATA SUBMISSION
QUESTIONS
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights Reserved.
...
Data Submission Questions
- Initial Certification
• Do the three consecutive months of data imply measurement
data as defi...
Data Submission Questions
- Customer Data
• There is a concern about that we cannot submit data without
the permission of ...
Copyright 2012 QuEST Forum. All rights reserved. 46
Repository Systems
- Identities of Companies
- Contact Information
- C...
47
Secure Data Submission Process
Secure Network
Internet
Switch
Front-end
Web Server
Registration
Management
System (RMS)...
Data Submission Questions
- Other
• As for the product delivery, the delivery to the end customer is
done through our sale...
Data Submission Questions
- Other
• What is the reason why the field data and delivery data for the
products submission is...
Data Submission Questions -
Advisories
• What is the purpose of advisories?
• What shall an organization do if an advisory...
Copyright © 2014 QuEST Forum
Measurements Advisories (page 1)
QuEST Forum strives to continually improve the value of the ...
Copyright © 2014 QuEST Forum
Measurements Advisories (page 2)
The “Advisor” checks look at data across measurements within...
Copyright © 2014 QuEST Forum
Advisory # 1 - The calculated Measurement over the
smoothed period is perfect – no longer app...
Copyright © 2014 QuEST Forum
Advisory # 16 – Downtime reported less than minimum expected
Advisory # 17 – Downtime reporte...
Copyright © 2014 QuEST Forum
What is expected of CB Auditors?
 Adhere to the QuEST Forum Code of Practice for TL 9000 Reg...
PDR Improvement
Executive Board
Strategic Initiative
Improve Member Satisfaction with Performance Data
And Performance Data Reports (PDRs).
EB SI #5 - PDR Improvement Project ...
TL 9000 Industry Reports
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights Reserved.
5...
Network Field Maintenance Study
Copyright 2009 QuEST Forum. All Rights Reserved Copyright
2009 QuEST Forum. All Rights Res...
60
Problem Report Industry Average
Industry Average has Strong Performance
showing an improvement trend
0
0.0005
0.001
0.0...
61
Fix Response Time Industry Average
Better than 95% in 2014
Showing an improvement trend
50
55
60
65
70
75
80
85
90
95
1...
62
OFR4 Industry Average
Consistent Performance
50
55
60
65
70
75
80
85
90
95
100
%OverdueFixResponseTime
Month
7.3.1 OFR4...
63
On Time Service Industry Average
Approaching 100% in 2014
50
55
60
65
70
75
80
85
90
95
100
%OnTimeService
Month
7.3.1 ...
64
Service Quality Industry Average
Improving to less than 0.5%
0
0.5
1
1.5
2
2.5
%DefectiveServiceTransactions
Month
7.3....
65
Support Service Caused Outages Industry Average
SSO Improving in 2014
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
%SupportService...
Wireless Study
Copyright 2009 QuEST Forum. All Rights Reserved Copyright
2009 QuEST Forum. All Rights Reserved.
66
Three W...
67
Wireless Study
50% Decrease in 3G Return Rate
GOOD
0
0.5
1
1.5
2
2.5
%ReturnRate
Month
YRR Industry Average
3.3.2.1 (2G...
68
Wireless Study
2G and 3G at a mature level. 4G Trend Improving
0
0.0001
0.0002
0.0003
0.0004
0.0005
0.0006
0.0007
0.000...
69
Wireless Study
4G SPR2 IA Improving; 2G and 3G have mature IA
0
0.0001
0.0002
0.0003
0.0004
0.0005
0.0006
0.0007
0.0008...
70
Wireless Study
4G Approaching 2G and 3G Major Fix Response Time
GOOD
0
10
20
30
40
50
60
70
80
90
100
%FixResponseTime
...
71
Wireless Study
OTI Trend lines at about 70%. 3G & 4G Degrading
0
10
20
30
40
50
60
70
80
90
100
%OnTimeItems
Month
OTI ...
72
Wireless Study
4G YRR, SPR2, FRT2 maturing rivaling 2G and 3G;
OTI degrading for 3G and 4G and not in line with custome...
Investigation Letter Trial Example (PC 1.2.2)
73
Dear TL 9000 Registration Administrator,
It has been brought to the atten...
1.2.2 Example Continued –
Investigation Letters Resulted in ERI
Rates Returning to Typical Levels
Copyright 2009 QuEST For...
2014 Investigations Conducted
75
Product
Category Name Issue # of Certs. # Responded Suspended
Dropped
Cert
Still
Suspende...
Asking Companies to Check Their Data When
Anomalies Occur, Can Improve Data Accuracy
76
0
10
20
30
40
50
60
70
80
90
100
4...
77
Correct?
Category added
to profile
QF approval
noted in profile
Pop-up asks for
info
Info
Complete?
No
Yes
Verification...
Useful PDR Whitepapers
• PDR Usage
• PDR Measurements Outputs and
Calculations
• Product Category Selection
• TL 9000 Data...
QuEST FORUM
Member Value Study
2014 Results
© 2014 Walker Information - proprietary and confidential 80
2014 MEMBER VALUE SURVEY RESULTS
Response Rate
Invitations
Sen...
© 2014 Walker Information - proprietary and confidential 81
2014 MEMBER VALUE SURVEY RESULTS
Overall Membership Satisfacti...
© 2014 Walker Information - proprietary and confidential 82
2014 MEMBER VALUE SURVEY RESULTS
Performance Data Reports
The ...
© 2014 Walker Information - proprietary and confidential 83
2014 MEMBER VALUE SURVEY RESULTS
Performance Data Reports
QuEST Forum Member Value Survey
PDR Specific Data Trends
2008 2009 2010 2011 2012 2013 2014
Overall PDR Satisfaction 49 60...
Contact Information:
Ken Koffman
JDSU
+1 (240) 404-1249
ken.koffman@jdsu.com
John Wronka
Alcatel-Lucent
+1 (727) 204-9508
...
TL 9000 Measurements and Requirements Interactive Workshop
Upcoming SlideShare
Loading in …5
×

TL 9000 Measurements and Requirements Interactive Workshop

984 views

Published on

TL 9000 Measurements and Requirements Interactive Workshop presented by John Wronka - Alcatel-Lucent and Ken Koffman - JDSU. Learn how your business can become TL 9000 certified. TL 9000 is a two-part quality system with significant management and measurement components.

Published in: Technology
  • DOWNLOAD FULL BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

TL 9000 Measurements and Requirements Interactive Workshop

  1. 1. Copyright 2009 QuEST Forum. All Rights Reserved. 1 1 TL 9000 Measurements and Requirements Interactive Workshop APAC Best Practices Conference Tokyo, Japan 16 April, 2015 Ken Koffman – JDSU John Wronka – Alcatel-Lucent
  2. 2. Topics for Today • Questions were provided in advance in the following Areas: – General – QuEST Forum Membership – TL 9000 Certification – TL 9000 “Adder” Applications – Measurements – Data Submissions • Time Permitting: – PDR Strategic Initiative – Member Value Survey 2
  3. 3. “GENERAL” QUESTIONS Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 3
  4. 4. Acquiring TL 9000 Certification • Tell me about how my business can become TL 9000 certified • How long does certification take? • How many personnel are required? Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 4
  5. 5. Copyright 2012 QuEST Forum. All rights reserved. 5 TL 9000 Implementation Path 6. Continual Improvement 5. External Assessment 4. Assess and review 3. Process Improvement 1.Management Commitment • Understand TL 9000 and Customer Needs, Set Goals, Commit Resources • Core Team Selection and Training, Gap Assessment, Project Planning • Process Mapping, Data Collection and Analysis, Process Changes to Meet New Requirements • Internal Audit, Management Review, Corrective and Preventive Action • Certification Body Pre- Assessment and Registration Audit Measurements Reporting – Often a gating item. Plan at least 9 months to prepare and submit. Start Suggested Implementation Time Line 4-6 Months 3-5 Months Ongoing 2. Organizational Preparation 2-3 Months 1-2 Months
  6. 6. 6 I would like to have realistic examples of measures to meet TL 9000 Requirements: Access Examples Utilizing “Links” ( tl9000.org/links.html). Use “Contact Us” to Suggest New Examples for IGQ Examples Subteam Scroll Down
  7. 7. What Kinds of Questions Does QuEST Forum Get on TL 9000 Measurements and Trends? Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 7 http://tl9000.org/alerts/qa/overview.html
  8. 8. General Questions – ISO 9001:2015 and TL 9000 Requirements R6 • The requirements for documentation and records have been reduced version by version in ISO 9001 QMS. On the other hand, in the TL 9000 system which include ISO 9001, documentation is mandatory for many requirements. In the future, will TL 9000 be aligned to ISO 9001:2015 in which the documentation requirements are supposed to be much more reduced? Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 8
  9. 9. ISO 9001:2015 Status Update 9 CD available 2013 DIS available 05/2014 FDIS expected AFTER 07/2015 ISO 9001:2015 to be published 09/2015 3-Year Transition Proposed ISOStandardsDevelopmentProcess IAF Informative Document - Transition Planning Guidance for ISO 9001:2015 Published TC 176 SC 2/WG 23 Communication & Product Support – Ashok Dandekar (QuEST Forum Representative) International ballot on the DIS closed on Oct. 10, 2014 – Results were APPROVE DIS by 90% ISO/NP TS 9002 “Guidelines for the application of ISO 9001” may be published in 2015
  10. 10. QuEST Forum ISO 9001:2015 Position
  11. 11. What actions have QuEST Forum taken regarding ISO 9001:2015? Solicited, reviewed and submitted comments on ISO 9001:2015 CD QuEST Forum agreed continued alignment to ISO 9001 QuEST Forum Academy ISO 9001:2015 Webinar by Nigel Croft Solicited, reviewed and submitted comments on ISO 9001:2015 DIS Mapped R5.5 TL 9000 Adders to ISO 9001:2015 DIS clauses (now 10!) 08/2013 06/2014 07/2014 07/2014 11/2014 May REPEAT upon release of the FDIS!
  12. 12. “Key” Detailed Changes: • New version of Figure 1; ISO 9001 still follows PDCA • “documented information” remains • “procedures” and “records” are not mentioned instead “maintain documented information” and “retain documented information” • A quality manual and a management representative are no longer required • “products and services” replaces products • ISO 9000 will be updated to address changes in terms & definitions • “applicability” replaces “exclusions” • risks are to be determined and addressed, but there is no requirement for formal risk management or risk approach • New/different requirements from ISO 9001:2008 include organizational context and knowledge management • Annex A to provide general guidance on implementation and other Annexes will be included such as a summarized analysis of key enhancements between ISO 9001:2008 and ISO 9001:2015 13 ISO 9001:2015 Revision Overview
  13. 13. General Questions – Third Party Process • How does QuEST Forum take action with all certification bodies in order to have a uniform level of TL 9000 certification audits? • Or, is there no level difference of the audit among all certification bodies in the world? Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 14
  14. 14. Regional Findings / Audit Day
  15. 15. ABCB Sub-Team Review: Overview of Sub-Team Charter • CB Statistics • AB Approval / Re-approval • Code of Practice – Including TL 9000 Auditor Qualification and Experience Requirements for CBs • Validation Audits Copyright 2011 QuEST Forum. All Rights Reserved 16
  16. 16. ABCB Sub-Team Review: Validation Audits • Plan: 2008-9 – Decision to investigate the effectiveness of CB audits in identifying major non- conformities • Do: 2010-11 – ABs conducted validation audits at randomly selected sites • Check: 2012-13 – ABCB Sub-team analyzed data and developed recommendations Copyright 2011 17
  17. 17. Action Thresholds Copyright 2009 QuEST Forum. All Rights Reserved Copyright 2009 QuEST Forum. All Rights Reserved. 18 90% Failures Certs=800 Sites Problem Sites High Midpoint Low Site=1700 12 0 22.1% 11.0% 0.0% 3rd Party Process Function Well 1 32.7% 16.5% 0.4% Individual Issue, 3rd Party Process OK 2 40.8% 21.9% 2.9% Investigation Required 3 47.6% 27.1% 6.6% 3rd Party Process Review Required 4 53.4% 32.1% 10.8% 5 58.4% 36.8% 15.1% 6 62.7% 41.1% 19.6% 7 66.6% 45.3% 23.9% 8 70.0% 49.1% 28.2% 9 73.0% 52.7% 32.4% 10 75.7% 56.0% 36.4% QuEST Forum Validation Audit Confidence Interval Sites Audited Problemsite count fromVA Audits Problemsite estrapolationestimates 5 of 12 audits Failed: Team determines there is a high risk of CB audits failing to detect some major non-conformities to TL 9000
  18. 18. ABCB Sub-Team Review: Validation Audits • Act: 2014-15 – Approved 1. Code of Practice Changes • Additional qualifications for new auditors • Additional continuing education requirements • CBs to identify and respond to soft grading • CBs to collect additional pre-audit information from sites • On site audits required for installation processes 2. QF to ensure validity of product category selections 3. Optional checklist for Measurements auditing 4. Optional webinars – Auditor Rotation – REJECTED by OSWG Vote 19
  19. 19. ABCB Sub-Team Review: Validation Audits • Act: 2014-15 – Proposals Under Discussion 1. Upgrade CB Technical Review of audit reports 2. Prepare a list of optional questions for CB auditors 3. MHB Checklist improvements 4. Anonymous feedback on CBs 5. Library of recorded webinars 6. Provide additional training for CB auditors 7. Reconsider AB oversight requirements 8. Perform further validation audits 20
  20. 20. General Section – Other Questions • What is the difference between "Tier 1 Supplier" and "Tier 2 Supplier"? – It depends!!! • Does QuEST Forum have any plans to set standard objectives for each measurement based on PDR Industry Statistics? – No • Thoughts on estimating future measurement trends based data collected from all organizations. What is the most effective way? – Depends on maturity level, data systems in place, connection with improvement initiatives • What part of the TL 9000 Adders should be emphasized or enhanced in the future? – Some of the newer “adders” such as security, sustainability, risk management, supplier performance management Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 21
  21. 21. QUEST FORUM MEMBERSHIP QUESTIONS Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 22
  22. 22. TL 9000 Certification & QuEST Forum Membership Questions • TL 9000 Certification does not require Forum membership • QuEST Forum membership does not require TL 9000 Certification Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 23
  23. 23. QuEST Forum Membership – Other Questions • Can an Operator Receive IA/BIC/WIC for their suppliers and the suppliers of other operators? – Yes • Can a Forum member or TL 9000 certified supplier obtain industry statistics solely for the carrier which it supplies products? – No • Does being a QuEST Forum member reduce the personnel required to be TL 9000 certified? – Not directly, but QuEST Forum membership provides access to SMEs and best practices. Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 24
  24. 24. TL 9000 CERTIFICATION QUESTIONS Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 25
  25. 25. TL 9000 Certification – Audit Days Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 26 I recognize that TL 9000 certification audit is conducted in conjunction with ISO 9001: • How long is the additional audit time for TL 9000 specific part?
  26. 26. Is there a case where a service provider requires its suppliers to require their suppliers to meet TL 9000? • Yes! - See Examples – “The Supplier shall determine which of its subcontractors and vendors (Quality Parties) should be registered to TL 9000 and ensure that each of those timely registers to TL 9000”. – “In the event Seller uses a contract manufacturer or subcontractor to fulfill the terms of the Agreement, Seller shall….. Commit to contract manufacturer or subcontractor registration to TL 9000 Quality Management System Standard” • 7.4.1.C.2 Supplier Performance Management – – requires identified “key suppliers” to align towards TL 9000 OR other appropriate QMS – But does stops short of mandating TL 9000 for suppliers Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 27
  27. 27. TL 9000 Certification - Scope Questions • The clause 3.2 b) in the Requirement Handbook requires in the Scope Statement that if not all products/product lines are included in the registration, then either the included or excluded products/product lines shall be listed. I would like to suggest for the list to be included in the Registration Profile. – Suggestion has been discussed but becomes difficult for large certifications • Is it possible to limit the scope of certification to the products for a specific customer who requires us to be TL 9000 registered? – No, but you are required to submit only data for customers where you have confidence in the data. • Once an organization gets TL 9000 certified, must the organization supply all products that meet TL 9000 requirement to customers? Or, can the organization decide which products should meet TL 9000 requirement s or not. (For an example, is it possible for product A to meet TL 9000 and Product B to meet only ISO 9001) – Yes-you can limit the scope of your products or organizations and activities to either TL or ISO.Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 28
  28. 28. TL 9000 Certification - Other Questions • An organization wants to limit its scope of registration to products for one customer (a service provider). But, because the customer is new, it has not yet supplied any products to the customer. In this case, is it possible to submit preceding three month data as all "0"? The organization has been supplying similar products to the other customer. But the customer is out of scope, so the organization thinks it cannot use the latter customer's data because it's out of scope. - Data must be submitted before you can be TL 9000 certified • Can we participate in the global telecommunications industry if we are not TL 9000 certified? – Yes of course, but… – Customer requirement’s for TL 9000 must also be considered Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 29
  29. 29. APPLICATION OF TL 9000 ADDERS QUESTIONS Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 30
  30. 30. Inputs to TL 9000 from Other Standards • What part of the items in TL 9000 Requirements and Measurements reflect the standards in CMMI and Bellcore? Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 31
  31. 31. Copyright 2012 QuEST Forum. All rights reserved. 32 TL 9000 is based on ISO 9001 ISO 9001 Telcordia (Bellcore) ISO/IEC 12207 (Software) ISO 9004 TL 9000 Later Versions Incorporated Concepts From CMMI, Automotive and ITIL
  32. 32. 33 CMMI and TL 9000 Full TL 9000 Certification Approximates CMMI Level 3. Standards Comparisons including CMMI can be found here .
  33. 33. 7.1.C.2 Disaster Recovery • We have Business Continuity Plan for disaster recovery planning. Could it be applied for this requirement? The head office is responsible for the Plan as a whole. Should the responsible function in the head office be audited for investigation? The head office will be out of scope for the certification at this time. So, we would like to exclude this requirement from the certification. Is it possible? • Answer: – BPC is a major part of disaster recovery – Yes it should be audited – This requirement can’t be excluded Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 34
  34. 34. 7.1.C.3 Product Security • As we are merely producing and supplying the products based on the specification which the Telecom Service Provider requires, the responsibility for the product security should be on the Provider. So we would like to exclude this requirement. Is it possible? • Answer: – In this case, if the supplier is doing any form of development work, even if to the service provider specification, this requirement can’t be excluded. – If there is no design work, and for example providing contract manufacturing (product category 8 ) – it could be excluded Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 35
  35. 35. 7.3.1HS.1 Migration Planning • As we do not intentionally change the OS of the product and just follow the customer requested specification, and the network operation is completely under control of the customer (Telecom Service Provider), so we do not have relevant activities of migration. • Answer: – If your company does any part of the design and the development, then migration will be required, and this requirement can’t be excluded. Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 36
  36. 36. 7.5.5.S.1 Software Virus Protection • As we merely produce/supply customer (service provider) specified products, we take it for granted that the responsibility for virus protection is on the customer. Therefore, we would like to exclude this requirement if possible. • Answer: – This requirement is difficult to exclude if there is any type of SW in the product. For example if you install any of the SW or have to replicate it, this requirement would apply. In these cases, procedures to prevent infections would be required. Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 37
  37. 37. 8.2.4.H.1 Periodic Retesting 8.2.4.H.2 Content of Testing 8.2.4.H.3 Frequency of Testing • Although our production plant is out of the certification scope, our quality control department takes care of the product quality. So we recognize that we should apply these requirements into our QMS. (This is just a confirmation.) • Answer: – Correct! Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 38
  38. 38. MEASUREMENT QUESTIONS Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 39
  39. 39. • What if the "units shipped" period does not actually encompass the past full 12 months, because of a newly introduced product which has become general availability only within a few months ? And also if the BRR basis shipping period does not reach 18 months for a new product? • Answer: – In both cases, report only the data for the months available up to and including the 12 month period Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 40 TL Measurements Questions – Partial Shipping Period
  40. 40. • As per the product category 6.2.1.2.4 Wireless Terminal Software Applications, is it not necessary to register this product category for a company who designs and manufactures complex wireless user terminals and also designs & installs basic control software for the terminals when shipping the terminals, but does not design nor supply application software after shipment? • Answer: – Based on Product Category 5.2, it is correct that a registration would be required, but based on the question it appears that the proper category would be 6.2.1.2.2 (Smart Phone) • MHB3.5.2n) says that the organization shall provide its suppliers all necessary information it possesses to allow those organizations to generate their TL 9000 measurements. In this sentence, do "those organizations" stand for "the organization's suppliers? I believe YES, otherwise, it will cause a confusion. • Answer: – Yes, "those organizations" means "those suppliers". Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 41 TL Measurements – Other Questions:
  41. 41. Measurements Questions – Linked Certifications • My company and its subsidiary company are now on the way to get TL 9000 certifications in parallel, that means to get different certifications. And they have already got different TL 9000 ID's for RMS registration. The main function of my company is design and development of wireless user terminals and the subsidiary company is the production function of the terminals. Some of the measurement data of these two companies are overlapped. So, the subsidiary company would like to submit Proxy-xxxx's for those measurement data using the TL ID of its parent company(my company), for the preceding data submissions(including the three month data submission) before successful completion of the certification. The certification process of the parent company is still on-going(not completed). Will it be allowed for the subsidiary to use Proxies during both company are in on-going certification process? If it is not allowed, there would occur a discrepancy of TL 9000 measurement statistics because of the duplicated data submissions. Please give me advice or opinion to this situation. • Answer: - If they are the same company, registrations can be linked. If there are multiple registrations where one registration wants to use data from the other registration(s), they can be linked. See section 4.2.9. The other registration(s) would need to provide the receiving registration with DSRs that can be viewed during an audit. This is a useful with common data systems. Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 42
  42. 42. DATA SUBMISSION QUESTIONS Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 43
  43. 43. Data Submission Questions - Initial Certification • Do the three consecutive months of data imply measurement data as defined in the Measurement Handbook must be submitted to the TL 9000 Measurement Administrator? – Yes – Data must be submitted to achieve certification • As for the three consecutive months data, should the data be submitted month by month?, or submit at once for three consecutive month? – Either way is acceptable initially for certification. Going forward must be monthly • Shall an organization submit the Data Submission Receipts for three consecutive month to the certification body before the initiation of the certification audit? – CB has access to them once set up in the registration profile Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 44
  44. 44. Data Submission Questions - Customer Data • There is a concern about that we cannot submit data without the permission of customer because some of the field data are customer's own. – There should be no concern. All data submissions are anonymous. Reports also don’t show submissions. • If a customer does not supply data for a specific measurement, can it be exempted? – You are obligated to report data for all customers where you have confidence in their data. . If you do not have confidence in the data from one or more customers you exclude those customers. If you don't have confidence in data from all customers for a particular month, you would report 0/0. • How does an organization, who supply OEM products to the specific customers, submit the measurement data? – Report data in the organization’s product categories Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 45
  45. 45. Copyright 2012 QuEST Forum. All rights reserved. 46 Repository Systems - Identities of Companies - Contact Information - Certification Body - Scope of Registration and Registration Status - No identities (anonymous reporting) - Data segmented by product category and reporting month RMS MRS Registration Management System (RMS) PUBLIC information used to manage TL 9000 Registrations Measurement Repository System (MRS) PRIVATE - Database of TL 9000 Registered Organizations DATA
  46. 46. 47 Secure Data Submission Process Secure Network Internet Switch Front-end Web Server Registration Management System (RMS) Measurements Repository System (MRS) Network Firewall Encrypted Data Processed by Software with No Human Intervention Raw Data Stored in Encryption Format Company Data Encrypted Submission HTTPS / Encrypted Tunnel Connection 5/6/2015 Copyright 2015 QuEST Forum. All Rights Reserved 47
  47. 47. Data Submission Questions - Other • As for the product delivery, the delivery to the end customer is done through our sales company which is out of the certification site. In this case, can we regard the sales company as customer, because we control delivery time for the sales company? – Yes • If, for example, we have data for three models of a product family, shall we submit the data to sum up all?, or separately? – Either way. In the end all data is aggregated anyway • How do you annualize data for the product that does not reach 12 months after the first shipment? – Monthly and smoothed data is annualized by the Afactor. Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 48
  48. 48. Data Submission Questions - Other • What is the reason why the field data and delivery data for the products submission is required for the products before certification is completed? The purpose of measurement data submission under TL 9000 QMS system should be to monitor the field data of the product and to improve the QMS by PDCA cycle. So, this should be after completion of certification. Why data submission needed before certification? • Answer: – The external assessor must verify that processes are in place to collect, validate and submit the required data. Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 49
  49. 49. Data Submission Questions - Advisories • What is the purpose of advisories? • What shall an organization do if an advisory is found in the DSR? Does it need to prepare evidences for each Advisory to prove if any problem exists or not? • How are the Advisories defined and how are the conditions set to issue Advisories? Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 50
  50. 50. Copyright © 2014 QuEST Forum Measurements Advisories (page 1) QuEST Forum strives to continually improve the value of the data provided through TL 9000 Performance Data Reports, which include industry statistic trends (Industry Average, Monthly Average, Best-in-Class, Worst-in-Class) as applicable for each product category and TL 9000 measurement. The more accurate and complete the data, the more useful it is to TL 9000 certified organizations and QuEST Forum members. A QuEST Forum initiative to further enhance data entry checking to identify and correct potential data errors before inclusion in the Measurements Repository System maintained by The University of Texas at Dallas, was completed in 2009. The result of this initiative was an “Advisor” software application that notifies TL 9000 data submitters with “advisories” indicating a potential data integrity issue that should be investigated. The existence of an advisory does not prevent the data from being successfully submitted. However, Organizations are expected to review any advisories identified and verify integrity of the submitted data. If errors are found, the Organization’s data is required to be corrected and resubmitted. Also, during registration and surveillance audits, the Certification Body will be expected to follow-up with the organization to insure data integrity, as appropriate, for any measurement flagged with an advisory.
  51. 51. Copyright © 2014 QuEST Forum Measurements Advisories (page 2) The “Advisor” checks look at data across measurements within submissions and across submissions over time to better validate the integrity of the organization’s submission. Advisories will be highlighted on the summary status of Data Submission Receipts (DSRs). Each advisory is assigned an integer identifier, and is briefly described at the end of the DSR. Some advisories may be common to all measurements , while others may apply to certain measurements. After investigation of a potential problem indicated by an advisory, it is very possible that the organization may find the data is indeed correct and no resubmission is required. For example, when multiple measurements in the same submission have the same Normalization Unit (NU) defined in Table A-3 of the Product Category Table, if the measurement submission includes differing values for the denominators (i.e. NU’s) for those measurements, an advisory will be generated. However, this difference may be legitimate (e.g. for NPR and SONE where in a particular product category customers don’t report outage data, but do report problems). In cases where advisory investigation does identify a problem, the organization will correct it, resubmit and maintain records of this process for review in registration and surveillance audits. More detailed information on the Advisories can be found in the paper referenced on page: http://tl9000.org/alerts/data_submissions.html
  52. 52. Copyright © 2014 QuEST Forum Advisory # 1 - The calculated Measurement over the smoothed period is perfect – no longer applied as of August 2011 Advisory # 2 - This submission is the same as month, year Advisory # 3 - Normalization Units differ across measurements that should be the same per Table A-2 – no longer applied as of July 2014 Advisory # 4 - Normalization Units changed greater than 25% from the prior month Advisory # 5 - Data Element >150% of the highest value reported over the previous 11 months Advisory # 6 - Data Element <50% of the lowest value reported over the previous 11 months Advisory # 7 - Data Element >125% of the highest value reported over the previous 11 months Advisory # 8 - Data Element <75% of the lowest value reported over the previous 11 months Advisory # 9 - Not yet Assigned Advisory # 10 - Data Element >120% of the highest value reported over the previous 11 months Advisory # 11 - Data Element <80% of the lowest value reported over the previous 11 months Advisory # 12 - Data Element >120% of the highest value reported over the previous 11 months and is 3x the average value in the same 11 months – reduced from 4x as of August 2011 Advisory # 13 - Data Element >125% of the highest value reported over the previous 11 months and is 3x the average value in the same 11 months – reduced from 4x as of August 2011 Advisory # 14 - Normalization Units are the same across measurements that should be different per Table A-2 Advisory # 15 - “Exempt” has been submitted for a measurement not listed as exempt on the organization’s TL 9000 profile - added August 2011 Measurements Advisories This list of advisories is current as of July 28, 2014 For the current list see DSR Advisories referenced on page: http://tl9000.org/alerts/data_submissions.html
  53. 53. Copyright © 2014 QuEST Forum Advisory # 16 – Downtime reported less than minimum expected Advisory # 17 – Downtime reported less than minimum expected Advisory # 18 - Outage frequency reported is less than minimum expected Advisory # 19 - Outage frequency reported is less than minimum expected Advisory # 20 - NPRs should normally be greater than NEOs Advisory # 21 - NEOs and SOs should normally be equal Advisory # 22 - Downtown reported less than minimum expected Advisory # 23 - Outage frequency reported is less than minimum expected Advisory # 24 - Input value greater than 95% of maximum value expected Measurements Advisories – Added July 2014 This list of advisories is current as of July 28, 2014 For the current list see DSR Advisories referenced on page: http://tl9000.org/alerts/data_submissions.html
  54. 54. Copyright © 2014 QuEST Forum What is expected of CB Auditors?  Adhere to the QuEST Forum Code of Practice for TL 9000 Registrars  Assure there is a documented system in place to collect, validate and report TL 9000 measurements  Investigate and validate that the system in place is implemented and effective  Ensure that the rational for all measurements exceptions are valid  Verify that Scope and Product Categories are appropriate for the Registration  Ensure that measurements submissions are made for all product categories under the scope of registration  Ensure that measurements submissions are made on time  Understand measurements advisories  Ensure that all measurements advisories are thoroughly investigated
  55. 55. PDR Improvement Executive Board Strategic Initiative
  56. 56. Improve Member Satisfaction with Performance Data And Performance Data Reports (PDRs). EB SI #5 - PDR Improvement Project OverviewTasks • Data Anomaly Investigations • Advisors/Error Checks • Product Category Verification Process • Industry Reports • Normalization Floors • “How To” White Paper Updates • Continued Workshops At BPC’s • Regional Webinar Delivery • Agreement with EB #3 For Goal/Path Forward •Identify Holes •Identify Potential Registrants • Identify “Similar” PC’s • PC Aggregation Initiative #1 Improve PDR Accuracy, Create Industry Reports Initiative #2 Maximize PDR Usage Value Initiative #3 “Fill The Pipe”, i.e. Expand PDR Scope Survey Feedback Initiatives
  57. 57. TL 9000 Industry Reports Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 58 • Network Field Maintenance Study • Wireless Infrastructure Product Family Trends • Then and Now: On-Time Delivery Measurement Study 2009 vs. 2013 • Repair Service Trends Study • Network Operator Quality Improvement Trends Study • Quality Improvements in Simple and Complex Wireless Devices Study • Edge Router Product Category Trends Study • Wireless Return Rate Study • Problem Report Fix Response Time Study • On-Time Delivery Measurement Study All Ten Industry Reports can be found on the QuEST Forum Web Site
  58. 58. Network Field Maintenance Study Copyright 2009 QuEST Forum. All Rights Reserved Copyright 2009 QuEST Forum. All Rights Reserved. 59 7.3.1 Product Category Measurements studied: • NPR4 (Problem Reports) • FRT4 (Fix Response Time) • OFR4 (Overdue Fix Response Time) • OTI (On Time Service) • SQ (Service Quality) • SSO (Support Service Caused Outages)
  59. 59. 60 Problem Report Industry Average Industry Average has Strong Performance showing an improvement trend 0 0.0005 0.001 0.0015 0.002 0.0025 0.003 NumberProblemReportsperService Month 7.3.1 NPR4 Industry Average 7.3.1-IA Linear (7.3.1-IA)
  60. 60. 61 Fix Response Time Industry Average Better than 95% in 2014 Showing an improvement trend 50 55 60 65 70 75 80 85 90 95 100 %FixedOnTime Month 7.3.1 FRT4 Industry Average 7.3.1-IA Linear (7.3.1-IA)
  61. 61. 62 OFR4 Industry Average Consistent Performance 50 55 60 65 70 75 80 85 90 95 100 %OverdueFixResponseTime Month 7.3.1 OFR4 Industry Average 7.3.1-IA Linear (7.3.1-IA)
  62. 62. 63 On Time Service Industry Average Approaching 100% in 2014 50 55 60 65 70 75 80 85 90 95 100 %OnTimeService Month 7.3.1 On Time Service Industry Average 7.3.1-IA Linear (7.3.1-IA)
  63. 63. 64 Service Quality Industry Average Improving to less than 0.5% 0 0.5 1 1.5 2 2.5 %DefectiveServiceTransactions Month 7.3.1 Service Quality Industry Average 7.3.1-IA Linear (7.3.1-IA)
  64. 64. 65 Support Service Caused Outages Industry Average SSO Improving in 2014 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 %SupportServiceCausedOutages Month 7.3.1. Support Service Caused Outages Industry Average 7.3.1-IA Linear (7.3.1-IA)
  65. 65. Wireless Study Copyright 2009 QuEST Forum. All Rights Reserved Copyright 2009 QuEST Forum. All Rights Reserved. 66 Three Wireless Product Categories • 3.3.2.1-Basic (2G) Base Transceiver Systems • 3.3.2.2-Advanced (3G) Base Transceiver Systems • 3.3.2.3-(4G) LTE Base Transceiver Systems Measurements studied: • YRR (One Year Return Rate) • SPR2 (Major Software Problem Reports) • FRT2 (Major Fix Response Time) • OTI (On Time Item Delivery)
  66. 66. 67 Wireless Study 50% Decrease in 3G Return Rate GOOD 0 0.5 1 1.5 2 2.5 %ReturnRate Month YRR Industry Average 3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G) YRR Industry Average
  67. 67. 68 Wireless Study 2G and 3G at a mature level. 4G Trend Improving 0 0.0001 0.0002 0.0003 0.0004 0.0005 0.0006 0.0007 0.0008 MajorSoftwareProblemReports PerNetworkElementperYear Month SPR2 Industry Average 3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G) Major Software Problem Report Industry Average
  68. 68. 69 Wireless Study 4G SPR2 IA Improving; 2G and 3G have mature IA 0 0.0001 0.0002 0.0003 0.0004 0.0005 0.0006 0.0007 0.0008 0.0009 MajorSoftwareProblemReports PerNetworkElementperYear Month SPR2 Industry Average Trends Linear (3.3.2.1 (2G)) Linear (3.3.2.2 (3G)) Linear (3.3.2.3 (4G)) Major Software Problem Report Industry Average
  69. 69. 70 Wireless Study 4G Approaching 2G and 3G Major Fix Response Time GOOD 0 10 20 30 40 50 60 70 80 90 100 %FixResponseTime Month FRT2 Industry Average 3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G) Major Software Fix Response Time Industry Average
  70. 70. 71 Wireless Study OTI Trend lines at about 70%. 3G & 4G Degrading 0 10 20 30 40 50 60 70 80 90 100 %OnTimeItems Month OTI Industry Average Trends Linear (3.3.2.1 (2G)) Linear (3.3.2.2 (3G)) Linear (3.3.2.3 (4G)) On Time Item Delivery Industry Average
  71. 71. 72 Wireless Study 4G YRR, SPR2, FRT2 maturing rivaling 2G and 3G; OTI degrading for 3G and 4G and not in line with customer expectations • 2G – Shows most mature performance as expected • 3G and 4G: •SPR2 improving •FRT2 improving •OTI degrading
  72. 72. Investigation Letter Trial Example (PC 1.2.2) 73 Dear TL 9000 Registration Administrator, It has been brought to the attention of the QuEST Forum Sub-team charged with monitoring and improving the TL 9000 data reports that there is a problem with the return rate reporting for Product Category 1.2.2 Access Multi-Service. The specific issue is there is at least one organization reporting return rates that are extremely high. This is evident in the Worst-In-Class values for all four of the return rates, but is most evident for ERI where the value is running over 200% per year. This would mean that on average each new unit is returned twice within six months of shipment. Since it is highly unlikely this could be true, we are asking all organizations with TL 9000 Certification in this product category to verify the FR data that they are submitting before your next data submittal. Specifically, please verify the following: 1. Only returns received during the month being reported are being counted 2. Each return is included in one and only one basis population 3. The is no factor applied to the number of returns, the actual count is being reported 4. The denominator includes all units shipped that belong to the basis population If you find there has been an error in your data reporting, you are required to correct and resubmit your data for up to the last 24 months. This letter is being sent to all organizations certified in Product Category 1.2.2 and to your Certification Bodies. The Certification Bodies are asked to confirm that the organization has verified their TL 9000 return rate data during their next audit. In the interim, the organization shall confirm to QuEST Forum that this verification has been performed within 30 days of receipt of this e-mail. Failure to do so will result in the suspension of your TL 9000 Certification until your confirmation has been received. If you have any questions, please use the “Contact Us” function on the tl9000.org web site. Thank you for your co-operation in this matter. Correcting this issue is essential in order to provide useful comparable return rate data for the organizations in this product category. QuEST Forum TL 9000 Administrator
  73. 73. 1.2.2 Example Continued – Investigation Letters Resulted in ERI Rates Returning to Typical Levels Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 74 24 letters sent, plus reminders as needed. A few acknowledged issues and resubmitted data. 0 20 40 60 80 100 120 5/2011 8/2011 11/2011 2/2012 5/2012 8/2012 11/2012 1.2.2 ERI ERI Moved to typical levels after resubmissions
  74. 74. 2014 Investigations Conducted 75 Product Category Name Issue # of Certs. # Responded Suspended Dropped Cert Still Suspended Reported Corrected Data 3.2.5 Fiber to the User FR - WIC 18 17 0 1 0 1 7.6.1 Procurement Services OTI - WIC 16 15 0 1 0 1 3.1.1.1.1 Metallic Conductor Cable NPRs 19 17 4 0 2 4 3.1.1.2.1 Fiber Optic Cable NPRs 28 26 6 0 2 7 Tracking of Investigation Responses is Currently Done Manually. The Team Is Considering Mechanized Methods Going Forward. No Response Has Resulted in Suspensions!
  75. 75. Asking Companies to Check Their Data When Anomalies Occur, Can Improve Data Accuracy 76 0 10 20 30 40 50 60 70 80 90 100 4/2012 5/2012 6/2012 7/2012 8/2012 9/2012 10/2012 11/2012 12/2012 1/2013 2/2013 3/2013 4/2013 5/2013 6/2013 7/2013 8/2013 9/2013 10/2013 11/2013 12/2013 1/2014 2/2014 3/2014 Change in 7.6.1 OTI Worst in Class NEW WorstInClass OLD WorstInClass While Anomalies can occur for valid reasons, if one seems completely unreasonable, report it via “Contact Us” and the PDR Team will investigate.
  76. 76. 77 Correct? Category added to profile QF approval noted in profile Pop-up asks for info Info Complete? No Yes Verification team notified Info Sufficient? No Yes Request for more info No Yes Feedback to organization Info received 1) Description 2) Product name and/or model number (hardware/software products – Families 1-6 and 8) 3) Primary function (hardware/software products – Families 1-6 and 8) 4) Competitors products name or model numbers (hardware/software products– Families 1-6 and 8) 5) Link(s) to material on web 6) Attach brochure(s) A New Product Category Approval Process Has Been Approved to Improve Product Category Selection Accuracy –
  77. 77. Useful PDR Whitepapers • PDR Usage • PDR Measurements Outputs and Calculations • Product Category Selection • TL 9000 Data Submission Methods Copyright 2009 QuEST Forum. All Rights ReservedCopyright 2009 QuEST Forum. All Rights Reserved. 78
  78. 78. QuEST FORUM Member Value Study 2014 Results
  79. 79. © 2014 Walker Information - proprietary and confidential 80 2014 MEMBER VALUE SURVEY RESULTS Response Rate Invitations Sent1871 Completed Surveys273 Response Rate15% Companies Identified205 Companies Participated125 Company Response61% 2014 INDIVIDUAL RESPONSE 2014 COMPANY RESPONSE 1720 Invitations Sent 258 Completed Surveys 15% Response Rate 187 Companies Identified 112 Companies Participated 60% Company Response 2013 COMPARISON 2013 COMPARISON
  80. 80. © 2014 Walker Information - proprietary and confidential 81 2014 MEMBER VALUE SURVEY RESULTS Overall Membership Satisfaction The higher the impact number, the more impact that measure has on the overall question shown at the top of the graph.
  81. 81. © 2014 Walker Information - proprietary and confidential 82 2014 MEMBER VALUE SURVEY RESULTS Performance Data Reports The higher the impact number, the more impact that measure has on the overall question shown at the top of the graph.
  82. 82. © 2014 Walker Information - proprietary and confidential 83 2014 MEMBER VALUE SURVEY RESULTS Performance Data Reports
  83. 83. QuEST Forum Member Value Survey PDR Specific Data Trends 2008 2009 2010 2011 2012 2013 2014 Overall PDR Satisfaction 49 60 63 70 67 71 70 Useful/Relevant PDR Data 48 54 62 68 66 69 69 Identifying Improvement Areas 45 52 56 58 55 60 62 Easy To Use 54 58 60 71 63 68 69 Providing Benchmarking Data 53 56 64 73 67 67 65 Providing Accurate Data 47 52 61 67 67 64 71 Timely Data 65 66 73 84 79 79 82 Supporting Continual Improvement 70 Aiding in Supplier Management 51 Top 2 Box Results 2014 n=152 2013 n=161 2012 n=156 2011 n=172 Significant Improvement in “Providing Accurate Data” while the 2014 PDR Overall Satisfaction Score holds steady near all time highs. Efforts are paying off - but more needs to be done
  84. 84. Contact Information: Ken Koffman JDSU +1 (240) 404-1249 ken.koffman@jdsu.com John Wronka Alcatel-Lucent +1 (727) 204-9508 john.wronka@alcatel-lucent.com

×