9. • Multiple Data Formats -> Uniform Data
• Difficult to compare and correlate
• No automation of data collection
• Problems not discovered when they happen
• Early intervention gets impossible
• Globally Distributed Data Sources
• Networking topologies, router config, firewalls
• Limited or no data from Subcontractors
• Providing customers with access to data
• Analysis Paralysis
• Analysis addressing the wrong problems
• Do you know your True First Pass Yield?
Homegrown solutions
• Poor performance, high maintenance cost
• Non-core activities
• Diverging impact on stress levels
Challenges Using Test Data
10. Building Training Sets
60% 19%
Mining Data
for Patterns
Other
Cost of Analytics
Refining Algorithms
Cleaning and
Organizing Data
Collecting
Data Sets
Source: forbes.com
57% 21% Collecting
Data Sets
Cleaning and
Organizing Data
What data scientists spend the most time doing
Least enjoyable part of their work
11. Statistical Process Control
SPC Assumption: Elimination of common
cause variation in measurements
Highly dynamic data!
components, 3rd party vendors,
number of steps, operators, test machines,
instrumentation, fixtures, processes etc
100s of test sequences
1000s of test steps
Aidon Oy Example
Batch size 10000 units
× 357 – Total components
× 137 – Different types of components
× 37 – Component changes -> every 280th unit
× Processes, operators, instrumentation changes…
A change every 10th product, or less
12. WECO - Western Electric Rules
Traditional SPC analysis
• Proactive detection of ‘out of control’ processes
Source: Wikipedia
13. Origin of Fault? KPI Monitoring
PCB
Test
In-
Circuit
Test
Module
Test
System
Test
Deployment
Traditional methods
Selecting KPIs
14. Deployment
Profit Loss from Failures Found
1x 10x 100x 1.000x >10.000x
PCB
Test
In-
Circuit
Test
Module
Test
System
Test
10X Cost Rule
15. Limited dimensions/meta data
due to capabilities and cost
Assumptions of root cause
Only known-unknown
Limited by Data available
Analysis Paralysis by Low Level Approach
Lack of Real-Time visibility
Lack of Data Availability
Wrong Improvements
Long Cycle Time
Not Lean - Significant Waste
What?
Why?
Improvement Contstraints
16. Following Best-Practices, 5M&E
Automated Collection
Integrated Repair Data
Initiatives based on insight
Don’t actively search in data
True 1st Pass Yield
Yield, Pareto, CPK
Numerical Analysis
8D Root-Cause Module
Global Data Acquisition
Real-Time Dashboards
Trigger-Automation Alarms
High-Impact Improvement
with good cost/benefit ratio
27. Repair/RMA Module
Why?
• A failed test is a
symptom of a
problem
• Repair Data adds
important context
• Historical
knowledge base to
guide repair
operator
28. Repair/RMA Module
• A failed test is a
symptom of a
problem
• Repair Data adds
important context
• Historical
knowledge base to
guide repair
operator
Why?
29. Gage R&R
• How much of the variation comes from the measurement system?
Why?
30. Other Capabilities
Manual Inspection Sequences
Overall Equipment Effectiveness
Connection and Execution Time
Execution Time Pareto
Rolled Throughput
Unit Verification
Box Build Configuration
Software/Firmware Distribution
MAC Address Distribution
Why?Why?What?
In this video we’ll look at a practical and specific approach OEMs and contract manufacturers can take to proactively make use of test data, for the purpose of driving improved business results.
We’ll start by looking at some of the broader issues around Digitalization.
We’ll then discuss how traditional approaches to electronics-manufacturing-quality-management falls short of meeting these objectives, and how a proven high level approach to test data management looks like. We’ll also show some specific examples on relevant key performance indicators.
If we consider the manufacturing industry, the purpose of trends such as Digitalization is to enable companies to act on data. The aim is to improve their profitability by doing things better than they did before.
<click> If your company excels in responding to insight from data you can expect to experience increased profit margins in the short run.
<click> This is largely due to operational improvements you do, improvements adding to your competitive advantage.
<click> In the longer run however, more and more competitors will go through a similar journey of improvements. The market pressure will then reduce your profit margins, as more competitors are able to supply products at a competing price and cost compared to you.
<click> If we assume that the time after today is where your opportunity for improvement and profits are found
<click> we also have to be mindful of the investment needed to get you there.
<click> Equally relevant is the fact that there is a delay between this investment and the return.
<click> This initial investment of time and or money can occur at any point in the future. If it happens too late the opportunity for improved profits will have passed, and you are potentially scrambling to avoid bankruptcy.
There are several dimensions that affects unit cost and profitability. In a broader sense, the continuum where the optimum is found is determined by the quality of your products, the efficiency and the cost of your operations. If metrics associated with these are hidden, or invisible to your organization, the outlook for short term profit gain and long term competitive advantage are not good, and you will need to look at how these barriers can be removed.
At the end of the day, what you would like to be able to achieve using your insight, is to initiate improvements that has the highest possible return compared to the efforts needed to complete them.
There are always going to be improvements available, and it is critical that you are able to distinguish the high value improvements from the trivial ones.
When we consider using the data from automated manufacturing testers, there are several challenges potentially keeping you from succeeding
<click> First, the data you have is likely stored in multiple different formats, making aggregated analytics and correlation very difficult.
<click> Data collection is not fully automated
<click> and there may be IT challenges in getting the data transferred from all test stations
Getting data from your subcontractors are often restricted by IT and bureaucracy,
<click> issues your customers also would experience if attempting to get this data from you.
Another significant problem is analysis paralysis, by not being able to assess the relevance of data trends.
<click> Since problems are always going to be present, this is likely to have you focus on issues that should have been given a low priority, or that might not even be the actual problem facing you.
These challenges are often battled from home-grown data management solutions. Solutions that adds low relative value, has high internal cost, and takes focus away from your core-activities. In addition, the inability to be proactive often carries a heavy stress burden on both managers and engineers, who might even have to travel across the world on short notices to engage in fire-fighting.
An article published on forbes.com
<click>has investigated how much time data scientists spent on different activities.
<click> As shown here, about 80% of the time is spent on collecting and organizing data. The problem is that this is an activity that has absolutely no contribution to competitive advantage for your company.
<click> In addition, about 80% report that this is the least enjoyable part of their job - making it a Human Resource issue as well.
https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#17530a7c6f63
Many Quality Assurance solutions used in the electronics manufacturing industry, both off-the-shelf and homegrown, are based on Statistical Process Control, a technique dating back to the 1920s.
<click>
One of the fundamental assumptions of this to work is that you are able to remove, or distinguish what is called common cause variations from the manufacturing process.
<click>
In a modern electronical product you have a massive amount of variables compared to the 20s. Things such as firmware, test operations, component changes, revisions and so forth.
<click>
An example we have from Aidon - designer and manufacturer of household smart-power-meters, shows that for one of their batches containing roughly 10.000 units they have a change in their process parameters
<click>
every 10th product or so. This is representative to a change in these common cause variation factors that SPC assumes are stable.
Going further, in SPC you have rules such as the Western Electric Rules for Proactive detection of issues. Put simply, it defines alarms for out-of-control processes, and originated back in the 1950s. One problem here is the dynamics of manufacturing itself, as we looked at, and your ability to understand the relevance of received alarms. Another is that on average you’ll one false alarm per 91,75 observation.
<click>
If you manufacture 10.000 products per year, each tested over 5 different processes - and each with an average of 25 different measurements per process, this will flood you with alarms, totally undermining the purpose of an alarming system.
So what you typically do is that you makes assumptions on what measurements are extra important, and monitor a limited set of Key Performance Indicators,
<click>
Very often these are well downstream in your manufacturing process, like at system level testing. The origin of a failure however could just as easily be located upstream of where the KPIs are recorded
<click>
So a poor batch of PCBs that was allowed to pass, all of the sudden means that you need to move the full system to repair and deconstruction, as opposed to just fixing the PCB.
The 10-X-cost-rule tells us that for every step in the process a defect is carried forward, the cost of fixing it increases by a factor of 10.
In this example a failure from the PCB Process identified at “System Test” would cost 100 times what it could have, had the issue been detected where it originated.
Takata Airbags is a good example from recent time, having to file for bankrupcy because faulty products was shipped out to the market.
If we map these issues onto Continuous Improvement Initiatives - we’ve illustrated it here with the Six Sigma DMAIC cycle,
<click> but it can be anything intended to drive improvements
traditional approaches contains limitations in every step when it comes to understanding
<click> what is really going on, and why this is happening.
<click> Because of the inherent limitations of traditional approaches as seen, you are forced to assume that your problems are found and originates where you actively are looking. At best these are able to inform you only of your known-unknowns, likely leaving blind-spots with serious issues unaddressed.
The limitations from this further affects the entire improvement cycle. And at the end of it, you probably don’t even have enough data, or real-time access to it, keeping you from evaluating even your weak improvements.
These constraints are what we remove with our skyWATS cloud solution and the WATS on-premise equivalent. And how our customers are able to continuously progress towards operational excellence.
<Click>
And you need the complete story to get there. Our philosophy is that actively looking for problems in your data is not sustainable, and as a rule of thumb you need to be presented the indicators of problems as they happen. And you absolutely need to be able to drill down from your relevant high level KPIs to any of the influencing categories, and compare the performance of the associated elements against each other. Such as comparing the performance of individual test stations, individual product revisions or different test sequence versions.
This approach to quality management is well proven,
<click> something our customer database is a strong testament to. Here, skyWATS and WATS are key elements in their internal value chains. Some of them reportedly even unable to understand how they could function before adopting it.
One of our customers gaining significantly from using WATS as part of a Lean Six Sigma program is Eltek, part of Delta Group. Before adopting WATS they operated with a First Pass Yield around 93%, and field failures within their customers warranty period of around 3%.
<click>
Elteks strategy moving forward became improving field failures through active management and optimization of their True First Pass Yield.
<click>
This again had the potential to impact their financial performance, both hard cash from reducing the cost of field failures, and soft cash from the necessary internal improvements.
<click>
By organizing themselves around ownership to the data they were able to increase their True First Pass Yield to 97% over a 5 year period. They were also able to reduce the field failures in warranty to less than one percentage, and estimated annual saving for year 5 to exceed one million dollars, with accumulated savings far exceeding that.
<click>
Today, Eltek operates with a First Pass Yield of around 98%, and a Fail Rate within warranty of less than 0,1%.
The skyWATS and WATS solution collects data by having clients installed where the test data is produced. This can either be on the test machine itself, or a local database.
<click> The client can be deployed to any location, making global data aggregation seamless. It supports buffering of test reports, avoiding loss of data due to network issues.
<Click>
We have both plug and play support and existing APIs that can pull data directly if you use TestStand, LabVIEW or .NET. The user can also modify their file output to a standard, well documented, XML or Text Format. Alternatively a file-converter can be made that reads your existing data formats. That way, no changes are needed to your software architecture. This option also allows you to import historical data.
<click>
The data is then in real-time moved to a Microsoft Azure hosted cloud or an on-premise server through TCP/IP
<click> that the browser-based reporting and analytics application connects to. Since it runs in the browser, no local installation is required.
<click>There is even a Rest API that you can pull data from the server with, to connect to existing Enterprise Software. Or read back data to the test stations if needed. We use this API ourselves in our mobile app, available for iOS and Android, giving you instant access to the status of manufacturing.
As mentioned, our process flow is a top-down one, and most often start with real-time dashboards, configured based on your needs. These will typically be either globally shared dashboard for specific functional units, or private ones that gives you the specific views you need.
The more detailed views, what we call the reporting section, often starts with yield statistics.
<click>You can slice these KPIs across multiple dimensions such as product, revision, factory, station and so forth, to look for categories performing worse than their compares, therefore negatively impacting the aggregated yield.
One level deeper we would typically look for the most frequent failures. This don’t have to be on one specific product, it can also be the most frequent failures at a product family, or factory for that matter. Applying the pareto principle to these, you’ll start to get the ability to weight effort vs impact, and get improvement initiatives with higher returns than before.
One of the reports also lets you see in what test run the products actually passed. In this example - using real data from an undisclosed Contract Manufacturer - we see that about 600 products passed the second test. The red flag though, is that there were actually some products that did not pass before the 21st attempt. Not necessarily something you want to ship to your customers just because you have a passed test in the end.
The Process Capability Analysis reports helps to understand how well your test limits are, compared to the process variations from the individual test measurements.
<click> The easiest way to get a good yield number is to have extremely wide limits, although this is likely not helping your quality go to where you want it to be. The values found here are also then useful in assessing how valid your yield results are and can be fed back to make sure that you have good test coverage.
<click> In this example you clearly see outliers in the data that should have been detected during test, but the limits were to wide.
Digging further down we can also do numerical analysis, like looking at correlations between different measurements, plot the measurements in different view-modes, and other more advanced analytics.
Part of continuous improvement is also how stakeholders collaborate to fix problems. skyWATS and WATS contains a root-cause feature, helping these people to collaborate and solve issues in a structured manner.
We consider capturing repair data, in the same system as test data, instrumental for a good quality management solution. One reason is this will provide valuable context data to why a test has failed. Most manufacturers captures some repair data, but it often lives only in their MES system or a stand alone application. We can either capture it directly through our web application used by the repair technicians, or we can interface the MES system to copy the data automatically from there.
One of the benefits of having repair data in WATS is that we can link the repair to a specific test report, and give you full traceability of the product history.
Our Gage R&R feature lets you further investigate how much of the variation in your measurements comes from the test system, fixture, or operator.
Other features available includes Manual Inspection Routines, analysis of Overall Equipment Efficiency, Test Connection and Execution Time, Time Pareto of individual test steps, Rolled Throughput, Process throughput and more.
There is also functionality to distribute test software and unit firmware automatically, to ensure that the latest software is always being used. Also the option to use skyWATS to make sure you keep track of the distribution of MAC addresses.
And in case you need to work with some of the data differently from what is available here you can always export it from our Export Wizard.
<click> where we have multiple options for formats.
Unless you have already done so, you can visit skyWATS.com/register to sign up for a free trial. You can then upload your own test data to see what lies hidden underneath. Or you can request that we load it with dummy data that you can play around with.
For a product demonstration, you can visit our contact page to request a web-based, or an in person demonstration.
Make sure to also check out our other videos, where we go more into technical details on specific functionality.
