"Method for large-scale structured-light system calibration," Optics Express (2021)

A. G. Marrugo, R. Vargas, L. A. Romero, and S. Zhang,  "Method for large-scale structured-light system calibration," Optics Express, 29(11), 17316-17329 (2021); doi:10.1364/OE.422327

Abstract

We propose a multi-stage calibration method for increasing the overall accuracy of a large-scale structured light system by leveraging the conventional stereo calibration approach using a pinhole model. We first calibrate the intrinsic parameters at a near distance and then the extrinsic parameters with a low-cost large- calibration target at the designed measurement distance. Finally, we estimate pixel-wise errors from standard stereo 3D reconstructions and determine the pixel-wise phase-to-coordinate relationships using low-order polynomials. The calibrated pixel-wise polynomial functions can be used for 3D reconstruction for a given pixel phase value. We experimentally demonstrated that our proposed method achieves high accuracy for a large volume: sub-millimeter within 1200(H) \times 800 (V) x 1000(D) mm^3.

"Flexible and high-accuracy method for uni-directional structured light system calibration," Optics and Lasers in Engineering (2021)

S. Zhang,  "Flexible and high-accuracy method for uni-directional structured light system calibration" Optics and Lasers in Engineering, 143, 106637 (2021)

Abstract

This paper presents a novel method that can accurately calibrate structured light system requiring only one directional structured patterns. This method leverages the existing flexible camera calibration to extract each calibration pose for three-dimensional coordinate determination for each pixel, and then establish pixel-wise relationship between each coordinate and phase. The phase artifacts caused by high-contrast calibration target was addressed to eliminate its impact on the calibration. Comparing with the traditional structured light system calibration method that requires two directional fringe patterns, our method requires only one single directional fringe patterns yet achieves 6.6 times higher accuracy for flat surface measurements.

“Portable high-resolution automated 3D imaging for footwear and tire impression capture,” Journal of Forensic Sciences (2021)

Y.-H. Liao, J.-S.-Hyun, M. Feller, T. Bell, I. Bortins, J. Wolfe, D. Baldwin, and S. Zhang, “Portable high-resolution automated 3D imaging for footwear and tire impression capture,” Journal of Forensic Sciences 66(1), 112-128 (2021) doi: 10.1111/1556-4029.14594

Abstract

This paper presents a high-resolution 3D imaging technology that we developed specifically for footwear and tire impression capture. We developed fully automated software algorithms and graphical user interface (GUI) that allow anyone without training being able to operate this system for high-quality 3D data capture. Comparing with the high-end commercially available 3D scanner, our technology achieves a similar level accuracy and resolution, our system has the merits of 1) being more affordable (a fraction cost of the commercial system); 2) being much easier to operate; and 3) being more robust. Comparing with the current practice of casting, our technology demonstrates its superiority because 1) it is non-destructive, 2) it collects more evidence than casts especially when an impression is fragile (e.g. in dry fine sand), and 3) it costs less time and money to collect each evidence. This paper describes the principle of the proposed technology, compares its performance with a commercially available 3D scanner, and presents our research findings on capturing various impressions with conventional practices and our technology.

"Comparative study on 3D optical sensors for short range applications", Optics and Laser Technology (2021)

Abstract

The increasing availability of commercial 3D optical sensors drastically benefits the mechatronics community by providing affordable sensing means for perception and control. Yet, to our knowledge, there are no comparable study to the state-of-the-art 3D optical sensors, making it difficult for users to select for their specific applications. This paper evaluates the performance of each sensor for short range applications (i.e., $\le$  1 m ). Specifically, we present our findings on the measurement accuracy of each sensor under ``ideal'' situations, compare the influence of various lighting conditions, object surface properties (e.g., transparency, shininess, contrast), and object locations. In addition, we developed software APIs and user instructions that are available for the community to easily use each of the evaluated commercially available 3D optical sensor.

Vide associated with submitted manuscript is below

"Large depth-of-field three-dimensional shape measurement with focal sweep technique," (2020)

X. Hu, S. Zhang, Y. Zhang, Y. Liu, G. Wang, "Large depth-of-field three-dimensional shape measurement with focal sweep technique," Optics Express  28(21), 31197-31208 (2020) 

Abstract

Three-dimensional (3D) shape measurement based on fringe projection technique has been extensively used for scientific discoveries and industrial practices. Yet, one of the most challenging issues is its limited depth of field (DOF). This paper presents a method to drastically increase DOF of 3D shape measurement technique by employing the focal sweep method. The proposed method employs an electrically tunable lens (ETL) to rapidly sweep focal planes during image integration, and the post deconvolution algorithm to reconstruct focused image for 3D reconstruction. Experimental results demonstrated that our proposed method can achieve high-resolution and high accuracy 3D shape measurement with greatly improved DOF with a potential to achieve real-time performance.

“State-of-the-art active optical techniques for three-dimensional surface metrology: a review,” (2020)

A. G. Marrugo, F. Gao, and S. Zhang, “State-of-the-art active optical techniques for three-dimensional surface metrology: a review,” Journal of Optical Society of America A 37(9) B60-B77 (2020) [Open Access]

Abstract

This paper reviews recent developments of non-contact 3D three-dimensional (3D) surface metrology using an active structured optical probe. We primarily focus on those active non-contact 3D surface mea- surement techniques that could be applicable to the manufacturing industry. We discuss principles of each technology, and its advantageous characteristics as well as limitations. Towards the end, we will discuss our perspectives on the current technological challenges in designing and implementing these methods in practical applications. 

"Status, challenges, and future perspectives of fringe projection profilometry," (2020)

J. Xu and S. Zhang, “Status, challenges, and future perspectives of fringe projection profilometry.” Optics and Lasers in Engineering, 135, 106193 (2020); doi: j.optlaseng.2020.106193

Abstract

As one of the most popular techniques for non-contact three-dimensional (3D) sensing/imaging, fringe projection profilometry (FPP) has been growing rapidly over the past decades partially because of the improved speed of computing devices and reduced cost of hardware.  3D optical sensing has started being an integral part of  our daily lives such as Face ID enabled by 3D sensors on smart phones. The impact of such techniques can be even greater with the ever-growing artificial intelligent (AI), machine learning, smart manufacturing, robotics as well as other fields.  However,  there are still fundamental challenges to be tackled to make such advanced optical sensing techniques ubiquitous.  This paper presents the current status of FPP techniques, the major challenges still facing in the field, and our perspectives on the future of FPP techniques.

"Autofocusing method for digital fringe projection system with dual projectors," (2020)

M. Zhong, X. Hu, F. Chen, C. Xiao, D. Peng, and S. Zhang, "Autofocusing method for digital fringe projection system with dual projectors," Optics Express 28(9), 12609-12620 (2020)

Abstract

This paper presents a novel technique to achieve autofocusing for a three-dimensional (3D) profilometry system with dual projectors. The proposed system uses a camera that is attached with an electronically focus-tunable lens (ETL) that allows dynamic change of camera's focal plane such that the camera can focus on the object; the camera captures fringe patterns projected by each projector to establish corresponding points between two projectors; two pre-calibrated projectors form triangulation for 3D reconstruction. We pre-calibrate the relationship between the depth and the current being used for each focal plane, perform a 3D shape measurement with an unknown focus level, and calculate the desired current value based on the initial 3D result. We developed a prototype system that can automatically focus on an object positioned between 450 mm to 850 mm.

"Influence of projector pixel shape on ultrahigh-resolution 3D shape measurement", (2020)

J.-S. Hyun and S. Zhang, “Influence of projector pixel shape on ultrahigh-resolution 3D shape measurement”, Optics Express 28(7), 9510-9520 (2020); do:10.1364/OE.389331

Abstract

The state-of-art three-dimensional (3D) shape measurement with digital fringe projection (DFP) techniques assume that the influence of projector pixel shape is negligible. However, our research reveals that when the camera pixel size is much smaller than the projector pixel size in object space (e.g., 1/5), the shape of projector pixel can play a critical role on ultimate measurement quality. This paper evaluates the performance of two shapes of projector pixels: rectangular and diamond shaped. Both simulation and experimental results demonstrated that when, the camera pixel size is significantly smaller than the projector pixel size, it is advantageous for ultrahigh resolution 3D shape measurement system to use a projector with rectangular shaped pixels than the projector with diamond-shaped pixels.

"A convenient 3D reconstruction model based on coaxial structure-light system," (2020)

J. Zhang, B. Luo, X. Su, L. Li, B. Li, S. Zhang, and Y. Wang,  "A convenient 3D reconstruction model based on coaxial structure-light system," Optics and Lasers in Engineering, 138, 106366 (2021)

Abstract

Conventional high-accuracy three-dimensional (3D) reconstruction methods usually involve complicated and time- consuming system calibration. This Letter presents a novel and simple 3D reconstruction model based on a coaxial structure-light system (CSLS). Compared to traditional structured light system, the proposed system contains a perspective projection unit and a telecentric imaging unit. Instead of obtaining the complicated structure-light system parameters, the proposed model only requires the retrieval of a product matrix which could be easily calibrated by using an ordinary white plane. For an arbitrary point A on the object surface, it is mathematically proved that the product matrix of its phase map and height distribution is proportional to the distance between A and projector optical center in XY plane, and independent of the depth in Z direction. Experiments were carried out to measure a step- height object and a cylindrical surface, and the results verify that the proposed method can achieve high-accuracy 3D shape measurement.

"Hybrid calibration procedure for fringe projection profilometry based on stereo-vision and polynomial fitting," (2020)

R. Vargas, A. G. Marrugo, S. Zhang, L. A. Romero "Hybrid calibration procedure for fringe projection profilometry based on stereo-vision and polynomial fitting," Applied Optics, 59(13), D163-D169 (2020); doi:10.1364/AO.383602

Abstract

The key to accurate 3D shape measurement in Fringe Projection Profilometry (FPP) is the proper calibration of the measurement system. Current calibration techniques rely on phase-coordinate mapping (PCM) or back-projection stereo-vision (SV) methods. PCM methods are cumbersome to implement as they require precise positioning of the calibration target relative to the FPP system but produce highly accurate measurements within the calibration volume. SV methods generally do not achieve the same accuracy level. However, the calibration is more flexible in that the calibration target can be arbitrarily positioned. In this work, we propose a hybrid calibration method that combines SV and PCM methods. The method has the flexibility of SV methods, is robust to lens distortions, and has a simple relation between the recovered phase and coordinates. Experimental results show that the proposed Hybrid method outperforms the SV method in terms of accuracy, acquisition time (as it requires fewer acquired images), and reconstruction time due to its low computational complexity.

"High-speed three-dimensional absolute shape measurement with three binary patterns," (2020)

J.-S. Hyun and S. Zhang, "High-speed three-dimensional absolute shape measurement with three binary patterns," Optical Engineering, 59(2), 024104 (2020); doi: 10.1117/1.OE.59.2.024104

Abstract

Reducing the number of structured patterns for three-dimensional (3D) reconstruction is of great importance for high-speed 3D shape measurement. This paper presents a method that reconstructs absolute 3D shape using three binary patterns: one DC, one low-frequency and one high-frequency fringe pattern. The procedures are: 1) take the difference between the sinusoidal fringe patterns and the DC pattern;  2) apply Hilbert transform to the difference images to generate two phase maps; 3) employ the geometric constraint based phase unwrapping method to unwrap the low-frequency phase map; 4) unwrap the high-frequency phase map using the unwrapped low-frequency phase map; and 5) reconstruct 3D shape.  We developed a prototype system that can capture 2D images at 6,000 Hz, achieving 2,000 Hz 3D shape measurement speed.

"Multilevel symmetric pattern design and optimization for high-speed and high- accuracy 3D shape measurement," (2020)

Y. Wang, J. S. Hyun, S. Zhang, B. Luo, Z. Liu, C. Jiang, B. Tao, "Multilevel symmetric pattern design and optimization for high-speed and high- accuracy 3D shape measurement," Optics and Laser Technology, 126, 106103 (2020); doi:10.1016/j.optlastec.2020.106103

Abstract

The binary defocusing technique has enabled speed breakthroughs for 3D shape measurement, yet simultaneously achieving high accuracy and high speed remains difficult. To overcome this limitation, we propose to utilize multilevel symmetric pattern for high-speed and high-accuracy 3D measurement. Compared to conventional binary patterns, multilevel pattern could bring more flexibility for eliminating undesired high-frequency harmonics, thus has the potential to greatly enhance the phase quality and measurement accuracy. In this paper, the symmetric pattern design principle and related optimization procedure were presented to find the best multilevel fringe patterns. Both simulation and experiments verify that comparing with conventional methods, the proposed method could consistently generate better fringe patterns for a wide range of fringe periods. Furthermore, we developed an absolute 3D shape measurement system with the speed of 667 Hz, verifying that the proposed method is applicable for high-speed, high-accuracy applications.

"Uniaxial 3D phase-shifting profilometry using a dual-telecentric structured light system in micro-scale devices," (2020)

M. Zhong, J. Cui, J.-S., Hyun, L. Pan, P. Duan, and S. Zhang, "Uniaxial 3D phase-shifting profilometry using a dual-telecentric structured light system in micro-scale devices," Measurement Science and Technology, 31(8), 085003 (2020); doi: 10.1088/1361-6501/ab63b2

Abstract

This paper presents a novel uniaxial microscopic 3D profilometry method using a structured light system with dual-telecentric lenses in micro-scale devices. Specifically, a telecentric lens can produce an orthographic view of an object and provide the exact size of objects in \textit and \textit-direction. An electronically focus-tunable lens attached to the projector rapidly and precisely changes the focal plane of the projected structured patterns, a camera captures the structured patterns from the same perspective, and a computational framework analyzes the relationship between captured structure images and the drive current of the electronically focus-tunable lens to obtain depth information for each pixel. By replacing the normal lens with telecentric lenses, the proposed method dodges perspective error and is easier for precision measurement. We developed a shadow-free coaxial 3D shape measurement system with dual telecentric lenses that achieved approximately 5.86 $\mu$m spatial resolution and 4.18 $\mu$m root-mean-square (RMS) error with a depth range of 1200 $\mu$m.

"Rapid and automatic optimal exposure control for digital fringe projection technique," (2020)

S. Zhang, "Rapid and automatic optimal exposure control for digital fringe projection technique," Optics and Lasers in Engineering 128, 106029 (2020); doi:10.1016/j.optlaseng.2020.106029

Abstract

This paper presents a method that can determine the optimal exposure time for high-quality three-dimensional (3D) shape measurement with digital fringe projection technique. The proposed method only requires capture fringe image(s) with one exposure time to automatically determine the global optimal exposure time for high-quality 3D shape measurement; and we further propose to automatically achieve high-dynamic range (HDR) 3D shape measurement by capture  image(s) with the optimized global single exposure.  Experimental results demonstrated the proposed method can successful work on a static complex scene and can also be applicable for real-time applications.

"Autofocusing method for high-resolution three-dimensional profilometry," (2020)

[121] X. Hu, G. Wang, J.-S. Hyun, Y. Zhang, H. Yang, S. Zhang, "Autofocusing method for high-resolution three-dimensional profilometry," Opt. Lett. 45(2), 375-378 (2020)

Abstract

State-of-the-art high-accuracy three-dimensional (3D) profilometry systems typically use lens with a fixed focal length, making it difficult for them to measure scenes with large depth variations especially dynamically changing ones. To address this need, this Letter proposes a novel autofocusing method for high-resolution 3D profilometry with a digital fringe projection (DFP) technique by 1) developing a novel continuous geometric parameter model for systems using electronically tunable lenses, and 2) employing a focal plane detection algorithm. The validity of the proposed method is confirmed by experiments.

"Calibration method for panoramic 3D shape measurement with plane mirrors," Opt. Express (2019)

[120] W. Yin, S. Feng, T. Tao, L. Huang, S. Zhang, Q. Chen, C. Zuo, “Calibration method for panoramic 3D shape measurement with plane mirrors," Opt. Express, 27(25), 36538-36550 (2019)

Abstract

High-speed panoramic three dimensional (3D) shape measurement can be achieved by introducing plane mirrors into the traditional fringe projection profilometry (FPP) system because such a system simultaneously captures fringe patterns from three different perspectives ( i.e., by a real camera and two virtual cameras in the plane mirrors). However, calibrating such a system is nontrivial due to the complicated setup. This work introduces a flexible new technique to calibrate such a system. We first present the mathematical representation of the plane mirror, and then mathematically prove that it only requires the camera to observe a set of feature point pairs (including real points and virtual points) to generate a solution to the reflection matrix of a plane mirror. By calibrating the virtual and real camera in the same world coordinate system, 3D point cloud data obtained from real and virtual perspectives can be automatically aligned to generate a panoramic 3D model of the object. Finally, we developed a system to verify the performance of the proposed calibration technique for panoramic 3D shape measurement.

"Large depth-of-field 3D shape measurement using an electrically tunable lens," Opt. Express (2019)

X. Hu, G. Wang, Y. Zhang, H. Yang, and S. Zhang, “Large depth-of field 3d shape measurement using an electrically tunable lens," Opt. Express 27(21), 29697-29709 (2019).

Abstract

The state-of-the-art 3D shape measurement system has rather shallow working volume due to the limited depth-of-field (DOF) of conventional lens. In this paper, we propose to use the electrically tunable lens to substantially enlarge the DOF. Specifically, we capture always in-focus phase-shifted fringe patterns by precisely synchronizing the tunable lens attached to the camera with the image acquisition and the pattern projection; we develop a phase unwrapping framework that fully utilizes the geometric constraint from the camera focal length setting; and we pre-calibrate the system under different focal distance to reconstruct 3D shape from unwrapped phase map. To validate the proposed idea, we developed a prototype system that can perform high-quality measurement for the depth range of approximately 1,000 mm (400 mm – 1400 mm) with the measurement error of 0.05%. Furthermore, we demonstrated that such a technique can be used for real-time 3D shape measurement by experimentally measuring moving objects.

"Pixel-by-pixel absolute phase retrieval assisted by an additional three-dimensional scanner," Appl. Opt., (2019)

Y. An and S. Zhang, “Pixel-by-pixel absolute phase retrieval assisted by an additional three-dimensional scanner”, Appl. Opt., 58(8), 2033-2041, 2019, doi:10.1364/AO.58.002033

Abstract

This paper presents a novel absolute phase unwrapping method assisted by a low-cost three-dimensional (3D) scanner. The proposed absolute phase unwrapping method leverages a low-cost 3D scanner to capture rough 3D data of the scene, and transforms the rough 3D data to the world coordinate system to generate an artificial reference phase map Φ_. By referring to Φ_, we can do absolute phase unwrapping directly without projecting any additional patterns, such that the digital fringe projection (DFP) system can achieve higher measurement speed. We develop a multi-resolution system consisting of a DFP system and Kinect V2 to validate our method. Experiments demonstrate that our method works for a large depth range, and the speed of the low-cost 3D scanner is not necessarily the maximum speed of our proposed method. Assisted by Kinect V2 whose maximum speed is only 30Hz, our DFP system achieves 53Hz with a resolution 1600x1000 pixels when we measure dynamic objects that are moving in a large depth range of 400mm.

"Motion induced phase error reduction using a Hilbert transform," Opt. Express (2018)

 [117] Y. Wang, Z. Liu, and C. Jiang, and S. Zhang, "Motion induced phase error reduction using a Hilbert transform," Opt. Express 26(26), 34224-34235 (2018); doi:10.1364/OE.26.034224

Abstract

The motion of object could introduce phase error and thus measurement error for phase-shifting profilometry. This paper proposes a generic motion error compensation method based on our finding that the dominant motion introduced phase error doubles the frequency of the projected fringe frequency, and Hilbert transform shifts the phase of a fringe pattern by $\pi/2$. We apply Hilbert transform to phase-shifted fringe patterns to generate another set of fringe patterns, calculate one phase map using the original fringe patterns and another phase map using Hilbert transformed fringe patterns, and then use the average of these two phase maps for 3D reconstruction. Both simulation and experiments demonstrated the proposed method can substantially reduce motion-introduced measurement error.