Recommended
Recommended
More Related Content
Similar to Platinum Inspection Case Study Using Image Processing
Similar to Platinum Inspection Case Study Using Image Processing (20)
Recently uploaded
Recently uploaded (20)
Platinum Inspection Case Study Using Image Processing
- 1. CASE STUDY A PLATINUM INSPECTION SYSTEM USING IMAGE PROCESSING Darien Pardinas Diaz
- 2. 1. System concept and setup • Scanning setup • Defect types to detect 2. Training and validation dataset development • Manual annotations with product manager 3. Algorithm development • Prototyping • Development tools used 4. Lessons learnt and recommendations 5. Quick Demo Page 2 Outline
- 3. • Tray of up to 15 platinum strips 480mm x 23mm • Line scan camera with 100pixels/mm resolution (600MB) • 3 Lighting arrangements targeting different defect types • Scanning time less than 5 minutes Page 3 System concept and scan setup
- 4. Page 4 Scanning the tray…
- 5. Page 5 Sample scan image Lighting B Lighting ALighting C (Never used)
- 6. 1. Scratches 2. Squiggles 3. Blisters (burst/unburst) 4. Dents 5. Stains Page 6 Defect types to find
- 7. Page 7 Training set development 23 failed strips with a range of defect types and severities
- 8. Page 8 Algorithm development A Dark blob detection Hysteresis threshold Coarse blob selection Classification (MLP) Line Detection Line gauss operator with scaled image Contour selection Defects (Squiggles, some scratches) Defects (Mainly scratches) Classification (MLP) B Detect Bright Blobs • Thresholding • Coarse blob selection Paired blobs Defects (Pits and dents) Detect Dark Blobs • Thresholding • Coarse blob selection Classification (MLP) Defects (Pits, Deep scratches) Defects (Squiggles, stains) Classification (MLP) Classification (MLP) Image normalisation
- 9. Page 9 Dark blob detection - example hysteresis_threshold
- 10. Page 10 Lines detection - scratches line_gauss
- 11. Page 11 Intensity normalisation subtract mean image
- 12. Page 12 Paired blob detection – dents, pits Blob pairing algorithm
- 13. Morphological features (blob contour) Texture features (pixel intensity) area contlength circularity compactness dist_sigma roundness entropy anisotropy anisometry bulkiness structure_factor phi length convexity mean_gray std_gray energy correlation homogeneity mean_gray std_gray energy correlation homogeneity contrast entropy anisotropy Invetech confidential file: File name/number (use macro : alt+f8 > Run ‘footer’) Features for classification MLP training – Overfitting problem - Adjusted the number of neurons in the hidden layer (30 neurons) - Used PCA to reduce the number of neurons in the input layer - Multiple scans of the same annotated strip – evaluate repeatability
- 14. Page 14 DevelopmentTools - HDevelop Great REPL experience Detailed feature inspection Interactive region filtering 100+ real life examples out of the box Export to C/C++/C# $7000 Developer license $2000 Runtime license Not all operators have equivalent in OpenCV
- 15. Page 15 Algorithm prototyping with HDevelop
- 16. 1. Early prototyping is critical to mitigate project risk 2. Invest as much effort as possible on developing a large, high-quality validation set of images and ideal results • Helps to understand the problem space • Provide confidence in the robustness of the algorithms 3. Involve the client in the annotation process of a validation set (if the project contains supervised training) • Manage client expectation on what the system will be capable of 4. Get the client to agree on objective measurements of system performance • Transparency on the outcome of the project • Non technical stakeholders usually have high expectation of what ComputerVision can do Page 16 Lessons learnt and recommendations
- 17. Page 17 Thank you! DEMO time!
Editor's Notes
- So, what I’ll do today is to run through one of the machine inspection systems I developed while working at BE This project was for a client who manufactures platinum to be used mainly in medical implants, such as cochlear implants The problem they were facing was that they had inspect manually the platinum strips they produce and assess the quality of the surface, so they wanted to automate this process and hired BE to look at the problem So I joined BE about 2 years after this project started, but at that time the image processing algorithms that were designed didn’t provide the results the client requested. So, my first assignment was to start looking at the existing algorithms and come up with new approaches that would solve this problem.
- 2
- -So the system consisted of a tray where you could lay up to 15 platinum strips of approximately 480mm by 23mm and image them with a line scan camera. -The image resolution was about 100pixels/mm, producing grayscale images that were 600MB each uncompressed. -Each strip is imaged progressively under 3 different lighting arrangement with the aim of highlighting a range of defect types with each one. -This system is capable of imaging and processing the tray in less than 5min and show the defects found. -As a note, each of this strips costs over $1000, so you can understand now why I’m not bringing one of them as sample
- In this video you can see how the strips are scanned with an X/Y robot. The line scan camera is coupled to the Y motor encoder to trigger the acquisition of lines as the scan goes along.
- So this is what the acquired scan image looks like. On every scan, 3 different strips are imaged at once, each illuminated with a different light arrangement. Then the camera is moved to the right, one strip position at the time. For a 15 strip scan, you end up with 17 scan images. There is a pre-processing step to segment each scan image into individual strip images so that we end up with 3 strip images referring to the same strip. In the final setup, we ended up just using lighting A and B, as defects showing up on C would always show better in either A or B light.
- In this slide you can see the most common defect types that the system had to be able to detect. They include scratches, squiggles, blisters (both burst and unburst blisters), dents, stains, pits, inclusions, etc.
- Given the range of defect types and severities in which they could appear it becomes really important to develop a dataset as large as possible that contains a representative sample of each of the defect types to find So for this project I developed an annotation tool that would allow us to annotate the defects we were trying to find as well as things that would look just like a defect, but weren’t, for example lint dust that remained on top of the strip after cleaning them with a “lint-free” cloth. This was really important in order to provide the machine learning algorithm good quality input. So I went onsite and spent a couple of days with the production manager imaging and annotating as many failed strips as we could (23 in total) This is a really tedious work, but the effort you put into it really have an impact in the success or failure of a project For example, while going through the annotation process we found that some defects showed well the image, but were hardly visible on the strips by the naked eye and vice-versa – and this is because the platinum is highly reflective and the defect visibility is extremely sensitive to the illumination angle and the eye location The fact that the production manager was involved in this annotation process helped initiate discussions that ultimately ended up adjusting the client’s expectation on the system and gave them the proper view that this project had a huge research component that translated in project risk. In order to keep things as transparent as possible, I regularly provided algorithm performance measures such as sensitivity/specificity and accuracy across the range of defects and severities – those are well known metrics that apply to most detection and classification systems While it was hard for them to understand what these numbers actually mean, at least gave them a sense that we were making progress while the algorithms were refined and tuned.
- So to cover the broad range of defect types I designed several algorithms to extract potential defective blobs from the strip images For the strip image under lighting A, I simply applied a hysteresis thresholding algorithm to extract dark blobs In this lighting, defects such as scratches were visible but really challenging to detect, so I used a line detection operator that is really good at joining weak and broken lines together For the strips image under lighting B, things were more complicated, in this case there was a number of defects that would either show as dark regions or bright regions depending on the surface relief of the strip. In this case, I had to normalise the image intensity range and they apply traditional thresholding techniques to segment potential defective blobs. Once the potential candidates are found the next step is to extract a feature vector for each of them and pass them to the classification model, which will evaluate each input and provide a score on how likely this blob is a defect based on the learning from the training stage.
- In this image you can see for example a well defined squiggle somewhat overlapping with a short scratch, in this case you can see how a simple hysteresis thresholding algorithm can perfectly segment the defect. Hysteresis thresholding is similar to normal thresholding but has the advantage of adding less defined pixels to the blob as long as they are over a minimum threshold intensity and certain distance of a well defined pixel (upper threshold)
- In this image you can see how the scratch detection algorithm works. The line detector operator I used works by estimating for each pixel, if they were part of a line, in what direction that line would run and how likely is that that pixel is part of a line. Then, depending on how selective the algorithm is configured, it will output a collection of contours which you can filter out, for example, these short lines on the left are immediately ignored because their length is too short.
- Here you can see the normalisation algorithm in action. On the left you get the original image taken under lighting B, which shows a surface illuminated unevenly and in the right side you can see the normalised image, which makes easer to parameterise a segmenting algorithm.
- This is one of my favourite contributions to this project In order to detect defects like dents, I came up with a simple and effective algorithm: I found that if I pair dark and bright blobs that satisfied certain Area / Intensity and maximum separation distance, I could detect defects like dents with really good accuracy.
- Once the potential blobs are detected you need to classify them into defect or non defect using some classification model For this project, I chose a MLPs networks (that’s a type of ANN) with a single hidden layer of neurons One of the biggest problem with ANN and machine learning in general is that is hard provide a meaningful input and at the same time avoid the overfitting problem. The overfitting problem happens when the size of the network too large for the training set. In this case, the algorithm learns extremely well to classify correctly your training data, but performs poorly on other validation data of your problem domain. Because the amount of training data was short, I had to limit both the number of neurons in the input and hidden layer. For the input layer, I used PCA, which reduces the dimension of the input feature vector. For the hidden layer I just used the rule of thumb that recommends the double of the number of neurons in the input layer. To evaluate algorithm repeatability – which proved to be an issue in some cases – I scanned the same strip multiple times and validate classification against them to find out how consistent the results were.
- So to prototype all these algorithms I mainly used a machine vision software package developed by a German company called MVTec This machine vision kit is called HALCON (Falcon) and includes an IDE called HDevelop with a simple yet powerful scripting language To the best of my knowledge there isn’t anything as helpful as this tool for prototyping image processing algorithms: It has a great REPL – you can set the executing cursor in whatever line you want to execute next, you can overlay images and regions to see intermediate results without having to write any code whatsoever You can do detailed feature inspection, create custom selection filters interactively to filter regions of interest and so much more. It also comes with more than 100 different real life examples sorted by industry, method, etc. that you can study and adapt to your specific problem with little effort The downside is that as good as it sounds it is about $7000 per developer license and $2000 for run-time license – only useful for prototyping or if developing some really expensive instrument or system (e.g. MRI, CAT scanner) Also not all operators have a direct equivalent in OpenCV – but usually enough documentation is provided to understand how the algorithm works so that it could be implemented if needed (reference to research papers)
- This screenshot for example shows the algorithm prototyped to detect blobs in lighting A On the top left you have a graphics window where you can see the results on each execution step if run step by step Below you get the list of iconic variables in your executing procedure and you can clear the window and overlay the results as you like A bit below are the control tuples – that is, the control variables you use to configure your operators (list of integers, doubles and strings) On the right hand side you can see the algorithm code you are developing and testing. The green arrow is the next executing instruction, which you can change to point anywhere in the procedure or outside of it.
- Firstly, Early prototyping is critical to mitigate project risk – this is no news for anyone here, but where there is image processing involved, there is always uncertainty and this is because a camera never produces the same image twice. I have worked in projects that you get to a solution that works fine 99.9% of the time, but from there to get to 100% of the requirement could take 2 or 3 times more. Secondly, spend as much effort as possible in developing a large, high-quality validation set of images and ideal results Helps to understand the problem space which is really important when prototyping Provide confidence in the robustness of the algorithms – you know you are covering a big chunk of the problem space Thirdly, Involve the client in the annotation process of a validation set (if the project contains supervised training) Finally, get the client to agree on objective measurements of system performance - Non technical people usually have high expectations on what computer vision can do – I blame Hollywood for this