Implementation of robust algorithms for dense-image matching and post-processing of machine vision data

Recent advances in computer vision allow complex objects to be surveyed and reconstructed using a large set of images, recorded with passive or active sensors, such as laser scanners, improved structured light systems, hyperspectral imaging systems, photogrammetry systems, etc. able to capture millions of points in real-time. Due to the availability of several low-cost and open-source software systems, automated 3D reconstruction methods are becoming very popular. Nevertheless, the metrological and reliability aspects of the resulting 3D scanning and modelling should not be ignored, particularly if the community wishes to adopt such solutions, not only for quick 3D modelling and visualisation but also for accurate scanning purposes related to tangible industrial applications (required uncertainties are less than 100 µm). The processing is one pillar of the full metrology chain that includes 3 main components: (1) Traceable Machine Vision Systems, (2) reference thermo-invariant material standards and (3) reference software and reference data. The addressed and implemented reference processing tools could be validated on numerous reference data to ensure 3D reconstruction, segmentation, classification, evaluation, etc., with very low uncertainties (calculation error <1 nm), aiming at least a tenth below the achieved scanning uncertainty. Even if the establishment of a good reference dataset (ground truth) with an accuracy which is two to three times better than the expected matching results is not trivial, it remains an essential and unavoidable task for the validation of dense-image matching software.

Although image matching has a history of more than 40 years (it initially appeared for digital surface model and digital terrain model generation from large blocks of near-nadir aerial images), the recent integration of computer vision algorithms and photogrammetric methods has increasingly accelerated automation of entire image-based 3D modelling process. Most of them are dedicated to the visualization of scenes without any traceability or connection to digital industry objectives. Here, image matching is one of the key steps in 3D reconstruction and mapping, usually solved using a stereo pair (stereo matching) (also pairwise) or identifying correspondences in multiple images (multi-view stereo-Machine Vision Systems) (also Groupwise). Pairwise feature-based methods, aiming to align two point clouds, are comprised of four main categories: Iterative Closest Point (ICP) [15], graph matchings [16,17], Gaussian mixture models [18], and Machine Learning (ML) techniques [19]. Groupwise methods, to align more than two point clouds, include graph optimizations, information theoretic measures, probability functions, and ML techniques [20], and are often based on multiple pairwise registrations. Usually such processing is performed into two steps: coarse matching and fine matching. The success of the matching lies in the accuracy of the implemented algorithms and also the evaluation metrics (or selected loss functions).

It is to be noted that software validation procedures for reference algorithms are not the same as for operating (commercial) ones. Despite its importance, performance in terms of execution time is not the first characteristic sought for reference algorithms. However, the algorithm must be stable and robust. Stability means that the underlying numerical operations are numerically stable. Robustness refers to the software ability to handle extreme cases. Unlike reference software, performance is the key for commercial ones. The algorithm is supposed to perform well only for representative data.

Once the objects are reconstructed, two approaches will be targeted: (i) accurate classification of defects as to take acceptance or rejection decision, and (ii) identification of metrological specifications with enhanced uncertainties.

Specific objectives of the thesis

1. To investigate and implement a robust generator of reference data for validating reference, commercial and open-source dense-image matching software, also called “softgauges”, defined by sets of data points for which the exact value of deviation is known with an associated uncertainty. The method for reference data generation enables the control of a number of parameters in the generated data sets including but not limited to, the density and distribution of the points, the value of the error/deviation, the number of data sets, the initial poses, etc. The aim of reference data use is to make sure that the matching algorithms return correct values with an uncertainty below the nanometre level (<1 nm).
2. To investigate and implement robust reference dense-image matching algorithms. MVSs provide dense-images in a short timeframe. These will be merged into one coordinate system to quickly visualise the scene or surface, while classifying identified defects or evaluating geometric characteristics. This calculation should be carried out in two steps: coarse matching and fine matching. Hence, one robust specific approach for coarse dense-image matching that may exploit intrinsic features of the surface and external information retuned by the industrial robot combined with the traceable 4 contactless tracking systems (distributed multi-head telemeter system based on self-calibrated principle and associated with automated multilateration algorithm) could be implemented. The target theoretical calculation error on reference data will be below the sub-micrometre, but the target coarse matching uncertainty on real data will be below 20 µm for translation and 100 µrad for rotation (including the positioning uncertainties of both the robot and the multi-head telemeter system). In addition, one reference algorithm for fine matching of multiple dense-images based on a variant of ICP (point to surface) should be implemented for 3D reconstruction of recorded dense-images by the traceable stereovision system, in relation to one proposed use case in the automotive on the defection of detects on painted surfaces​. The target theoretical calculation error will be < 10^-5 mm.
3. Toinvestigate and develop advanced reference algorithms for surface quality evaluation. As the enhancement of metrology capabilities requires the enhancement of each component in the MV full metrology loop, then, reference algorithms for the evaluation of geometric characteristics need to be addressed. The target algorithms will be dedicated to partition, association, rigid registration and areal surface texture. For example, minimum zone (MZ) fitting is still a major challenge for complex shapes and robust and deterministic MZ fitting algorithms with nanometric accuracy are essential for form metrology. Some attempts have been made using Least Squares (LS) fitting instead of MZ but the resulting from error is overestimated. The development of fitting algorithms also requires the development of appropriate tools for the validation of the underlying output. Reference data must be generated with a defined logic, expected values of the geometrical and statistical quantities. In the IND10-FORM and 15SIB01-FreeFORM projects, only reference LS and MZ fitting algorithms were developed for aspherical surfaces. Nevertheless, those reference algorithms should be extended to more complex shapes usually available in industry, which will be done in this task. Furthermore, validated algorithms for (i) rigid registration and (ii) identification of areal surface texture parameters on complex industrial shapes represent two additional challenges that should be addressed here. The robustness of all those algorithms will be validated on several reference datasets.
4. To investigate one case study in the automotive industry on the quality control of painted surfaces, considered as a crucial aspect of vehicle production. Paint plays a vital role not only in a car's aesthetics but also in protecting it from external elements such as corrosion, UV rays, and weather conditions. A well-executed paint job enhances wear resistance and adds aesthetic value, while inadequate quality control can lead to unsightly defects, reduced durability, and customer dissatisfaction.
5. to evaluate the performance of state of the art industrially available software for dense-image feature matching. Industrially available software will be compared against the reference data (softgauges), with regards to the requirements of ISO 5436-2 2012 “Type F1”. The comparison will enable the evaluation of feature matching contributions to MVSs applications.

Reference

[1] “Machine Vision Market Size, Share & Trends Analysis Report By Offering (Hardware, Software, Services), By Product, By Application, By End-use Industry, By Region, And Segment Forecasts, 2022 – 2030,” Report ID: GVR-1-68038-842-8.

[2] Z. Ren, et al., “State of the Art in Defect Detection Based on Machine Vision,” Int. J. Precis. Eng. Manuf. - Green Technol., 9, 2022, 661-91.

[3] A. Charan, et al., “The Future of Machine Vision in Industries- A systematic review,” IOP Conf. Ser.: Mater. Sci., 1224, 2022.

[4] “Directive 2008/98/EC on waste and repealing certain directives”.

[5] “COM(2019) 640 final, Brussels, 11.12.2019, The European Green Deal”.

[6] “COM(2020) 98 final, Brussels, 2020, A new Circular Economy Action Plan For a cleaner and more competitive Europe”.

[7] M. L. Smith, et al., “The quiet revolution in machine vision-a state-of-the-art survey paper, including historical review, perspectives, and future directions,” Computers in Industry, 130, 2021, 103472.

[8] “Section A: Global Car & Automobile Manufacturing Industry - Market Research Report,” Updated: July 12, 2023.

[9] I. Bianchi, et al., “Environmental impact assessment of zero waste approach for carbon fiber prepreg scraps,” Sustainable Materials and Technologies, 29, 2021, 308.

[10] K. K. Patel, et al., “Machine vision system: “…”  of food and agricultural products,” J. Food Sci. Technol., 49, 2012, 123-41.

[11] P. Wang, et al., “Application of machine vision image feature recognition in 3D map construction,”AEJ, 64, 2023, 731–739.

[12] F. Remondino, et al., “State of the art in high density image matching”, The Photogrammetric Record, 29, 2014, 144-66.

[13] J. Bartelsen, et al., “Orientation and dense reconstruction from unordered wide baseline image sets,” PFG Photogrammetrie, Fernerkundung, Geoinformation, 4, 2012, 421-32.

[14] N. Haala, “The landscape of dense-image matching algorithms,” Wichmann, Berlin/Offenbach, Germany, 2013.

[15] X. Jiang, M. Liu, Y. Huang, and R. Luo, ‘Review on improved algorithms based on ICP algorithm’, in 2020 International Conference on Computer Engineering and Intelligent Control (ICCEIC), Nov. 2020, pp. 185–189. doi: 10.1109/ICCEIC51584.2020.00043.

[16] P. J. Besl and N. D. McKay, ‘A method for registration of 3-D shapes’, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239–256, Feb. 1992, doi: 10.1109/34.121791.

[17] Y. Chen and G. Medioni, ‘Object modeling by registration of multiple range images’, in 1991 IEEE International Conference on Robotics and Automation Proceedings, Apr. 1991, pp. 2724–2729 vol.3. doi: 10.1109/ROBOT.1991.132043.

[18] X. Huang, G. Mei, J. Zhang, and R. Abbas, ‘A comprehensive survey on point cloud registration’. arXiv, Mar. 05, 2021. doi: 10.48550/arXiv.2103.02690.

[19] V. Villena-Martinez, S. Oprea, M. Saval-Calvo, J. Azorin-Lopez, A. Fuster-Guillo, and R. B. Fisher, ‘When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)’, Applied Sciences, vol. 10, no. 21, Art. no. 21, Jan. 2020, doi: 10.3390/app10217524.

[20] H. Zhu et al., ‘A Review of Point Set Registration: From Pairwise Registration to Groupwise Registration’, Sensors, vol. 19, no. 5, Art. no. 5, Jan. 2019, doi: 10.3390/s19051191.

La thèse se déroulera essentiellement au LNE de Paris et LURPA de l’ENS Paris-Saclay. Le cadre du projet européen (23IND08-DI-Vision) permettra des échanges scientifiques avec le consortium du projet ainsi que des déplacements en Europe sur la période de thèse.

Computer vision, industrial machine vision, imaging, image matching, optimisation and numerical calculation