Download as PDF, PPTX
Uploaded byAnviLewis
QR Code Reading Benchmark and Comparison
The document provides a comprehensive benchmark of various QR code reading SDKs, comparing performance based on reading rates and runtime across 536 images with 1232 QR codes in different categories. It evaluates five libraries, highlighting the Dynamsoft Barcode Reader as the top performer in accuracy, while also detailing modifications to runtime settings that can enhance readings, especially in challenging conditions. The benchmarks indicate that despite lower reading rates, some SDKs like commercial SDK B achieve faster runtimes, underlining the importance of optimizing settings for specific usage scenarios.
More Related Content
Similar to QR Code Reading Benchmark and Comparison
QR Code Reading Benchmark and Comparison
- 1. Best QR CodeReading SDKs
- 2. What is QRcode? § QR Code is a type of two-dimensional barcode § Consists of black squares arranged in a square grid on a white background § Stores more data than a 1D barcodes § QR Code has full (360 degrees) reading features § QR code has a finder pattern located in three of its corners
- 3. QR Code ReadingBenchmark and Comparison In the following slides we are going to run a benchmark based on an image data set to find out the best QR Code reading SDKs.
- 4. Dataset A comprehensive benchmarkwas made by the author of boofcv to compare boofcv with other 4 libraries § Performance test is made on the dataset to evaluate 5 open-source libraries § Dataset contains 536 images containing 1232 QR codes and has 16 categories § blurred § bright_spots § brightness § close § curved § damaged § glare § high_version § lots § monitor § nominal § noncompliane § pathological § perspective § rotations § shadows
- 5. Performance test tool Inorder to run the benchmark, a performance test tool is written § The tool uses Python to run the test and provides a web interface. The Flask web framework is used. § Code can be accessed here § It is going to run on a PC device with an Intel i5-10400 CPU and 16GB memory.
- 6. Evaluated libraries • DynamsoftBarcode Reader (DBR, version: 8.6, commercial) • Commercial SDK A (version: Java 13.5, commercial) • Commercial SDK B (version: Windows 5.19.3.10, commercial) • BoofCV (version: pyboof 0.36.1, open-source) • Zxing (version: 3.4.1, open-source) • Zbar (version: pyzbar 0.1.8, open-source) • WeChat QR code detector in OpenCV (version: OpenCV 4.5.3, open-source)
- 7. Evaluation metrics Performance isevaluated by reading rate and runtime. Reading rate can be achieved by dividing detected QR codes by total QR codes in images If the value of IoU (Intersection over Union) between the detected QR code’s polygon and the ground truth’s is over 0, then the QR code is considered detected. Reading rate = Detected QR codes / Total QR codes *More details about evaluation metrics can be found here
- 8. Detection results Reading ratein percent Categories Dynamsoft Commercial SDK A Commercial SDK B Zxing Zbar BoofCV OpenCV Wechat blurred 69.23 38.46 26.15 23.08 38.46 40.00 55.38 bright_spots 38.14 30.93 6.19 21.65 19.59 23.71 30.93 brightness 81.18 52.94 32.94 54.12 51.76 78.82 68.24 close 92.50 25.00 35.00 5.00 12.50 77.50 65.00 curved 73.33 38.33 33.33 30.00 35.00 50.00 43.33 damaged 53.49 27.91 23.26 20.93 25.58 16.28 48.84 glare 86.79 20.75 22.64 20.75 35.85 32.08 64.15 high_version 97.30 35.14 59.46 5.41 27.03 40.54 18.92
- 9. Categories Dynamsoft Commercial SDK A Commercial SDKB Zxing Zbar BoofCV OpenCV Wechat lots 60.48 97.86 1.90 82.86 18.10 99.76 0.00 monitor 100 5.88 5.88 0.00 0.00 64.71 94.12 nominal 91.03 62.82 51.28 51.28 66.67 89.74 80.77 noncompliant 88.46 3.85 15.38 19.23 50.00 3.85 92.31 pathological 100 78.26 78.26 34.78 65.22 43.48 91.30 perspective 68.57 34.29 51.43 37.14 42.86 82.86 42.86 rotations 99.25 67.67 63.91 42.86 49.62 96.99 82.71 shadows 100 95.00 80.00 65.00 90.00 85.00 85.00 Total average 81.23 44.69 36.69 32.13 39.27 57.83 60.24 Detection results Reading rate in percent
- 10. Detection results Dynamsoft BarcodeReader ranks 1st in most of the categories. It ranks 2nd in the noncompliant and perspective categories and 4th in the lots category. Reading rate in percent
- 11. Detection results Runtime perimage (in milliseconds): Runtime Results Engine Result DBR 141.10 Commercial SDK A 314.04 Commercial SDK B 48.31 Zxing 181.68 Zbar 199.26 BoofCV 205.48 OpenCV Wechat 175.06
- 12. Detection results Runtime perimage by categories (in milliseconds): Categories Dynamsoft Commercial SDK A Commercial SDK B Zxing Zbar BoofCV OpenCV Wechat blurred 298.00 335.16 39.67 206.89 177.58 138.16 125.31 bright_spots 252.84 625.47 47.56 311.59 319.81 221.81 112.25 brightness 192.75 600.14 47.86 320.32 349.79 273.43 134.79 close 169.40 607.27 80.08 295.18 381.65 150.70 676.00 curved 188.76 300.82 43.40 173.44 220.28 103.42 129.54 damaged 200.03 205.49 42.05 84.41 96.76 46.14 138.35 glare 104.66 217.60 45.40 130.32 157.30 70.80 95.38 high_version 106.06 349.21 79.88 294.97 141.70 268.36 566.03
- 13. Categories Dynamsoft Commercial SDK A Commercial SDKB Zxing Zbar BoofCV OpenCV Wechat lots 275.43 851.14 110.14 1073.00 1477.57 6149.00 139.57 monitor 176.59 812.53 105.88 233.29 515.47 142.47 422.76 nominal 74.05 203.65 32.12 141.49 111.68 109.49 54.54 noncompliant 58.88 81.63 34.44 47.81 57.06 31.94 43.31 pathological 8.52 4.09 65.87 3.00 1.83 14.61 19.39 perspective 25.74 46.34 14.91 27.66 24.31 26.74 24.60 rotations 72.11 173.84 35.80 106.84 134.66 178.45 86.91 shadows 102.21 243.50 42.29 150.00 163.29 117.50 86.36 total average 144.13 353.62 54.21 225.01 270.67 502.69 178.44 Detection results Runtime per image by categories (in milliseconds):
- 14. Detection results Although CommercialSDK B has low reading rate on the dataset, it is the fastest. The Dynamsoft Barcode Reader ranks second on average.
- 15. Update Runtime Settingsto Improve the Results § The benchmark is done without updating the SDKs’ default settings, which can be updated to improve the reading accuracy or speed § Dynamsoft Barcode Reader provides rich parameters that users can customize and optimize for different usage scenarios for the best scanning performance § The runtime settings of DBR can be represented and modified using a JSON template *Learn more about parameters of DBR here
- 16. Modify the Templateto Decode the Lots Category § Reason - QR codes are relatively small and the image has a large resolution (3024x4032) while in DBR’s template, there is a ScaleDownThreshold parameter which is set to 2300 so that the image will be scaled down for a better decoding speed. § If we set the ScaleDownThreshold to 99999, we can decode all the QR codes § The reading rate for the lots category can increase from 60.48 to 100 using the modified template There are several files in the lots category that DBR cannot detect all the QR codes
- 17. Modify the Templateto Decode the Blurred Category § For blurred QR codes as below, GRAY_SMOOTH image processing method can be used before deblurring to decode it § If the QR codes are blurred too much, not much can be done § DBR can only decode one extra image after using the modified template in the blurred category. However, SDKs like the Dynamsoft Camera Enhancer can help filter the blurred image DBR comes with many image processing algorithms
- 18. Conclusion § The DynamsoftBarcode Reader has the best QR code reading accuracy on the test set § DBR speed is good and has rich runtime settings to adapt to different scenarios § If conditions allow, it is better to choose DBR as the barcode reading SDK
- 19. Links Download Offline SDK Get30-day FREE Trial License Links related to Dynamsoft Barcode Reader