"Calibration method for panoramic 3D shape measurement with plane mirrors," Opt. Express (2019)

[120] W. Yin, S. Feng, T. Tao, L. Huang, S. Zhang, Q. Chen, C. Zuo, “Calibration method for panoramic 3D shape measurement with plane mirrors," Opt. Express, 27(25), 36538-36550 (2019)

Abstract

High-speed panoramic three dimensional (3D) shape measurement can be achieved by introducing plane mirrors into the traditional fringe projection profilometry (FPP) system because such a system simultaneously captures fringe patterns from three different perspectives ( i.e., by a real camera and two virtual cameras in the plane mirrors). However, calibrating such a system is nontrivial due to the complicated setup. This work introduces a flexible new technique to calibrate such a system. We first present the mathematical representation of the plane mirror, and then mathematically prove that it only requires the camera to observe a set of feature point pairs (including real points and virtual points) to generate a solution to the reflection matrix of a plane mirror. By calibrating the virtual and real camera in the same world coordinate system, 3D point cloud data obtained from real and virtual perspectives can be automatically aligned to generate a panoramic 3D model of the object. Finally, we developed a system to verify the performance of the proposed calibration technique for panoramic 3D shape measurement.

"Large depth-of-field 3D shape measurement using an electrically tunable lens," Opt. Express (2019)

X. Hu, G. Wang, Y. Zhang, H. Yang, and S. Zhang, “Large depth-of field 3d shape measurement using an electrically tunable lens," Opt. Express 27(21), 29697-29709 (2019).

Abstract

The state-of-the-art 3D shape measurement system has rather shallow working volume due to the limited depth-of-field (DOF) of conventional lens. In this paper, we propose to use the electrically tunable lens to substantially enlarge the DOF. Specifically, we capture always in-focus phase-shifted fringe patterns by precisely synchronizing the tunable lens attached to the camera with the image acquisition and the pattern projection; we develop a phase unwrapping framework that fully utilizes the geometric constraint from the camera focal length setting; and we pre-calibrate the system under different focal distance to reconstruct 3D shape from unwrapped phase map. To validate the proposed idea, we developed a prototype system that can perform high-quality measurement for the depth range of approximately 1,000 mm (400 mm – 1400 mm) with the measurement error of 0.05%. Furthermore, we demonstrated that such a technique can be used for real-time 3D shape measurement by experimentally measuring moving objects.

"Pixel-by-pixel absolute phase retrieval assisted by an additional three-dimensional scanner," Appl. Opt., (2019)

Y. An and S. Zhang, “Pixel-by-pixel absolute phase retrieval assisted by an additional three-dimensional scanner”, Appl. Opt., 58(8), 2033-2041, 2019, doi:10.1364/AO.58.002033

Abstract

This paper presents a novel absolute phase unwrapping method assisted by a low-cost three-dimensional (3D) scanner. The proposed absolute phase unwrapping method leverages a low-cost 3D scanner to capture rough 3D data of the scene, and transforms the rough 3D data to the world coordinate system to generate an artificial reference phase map Φ_. By referring to Φ_, we can do absolute phase unwrapping directly without projecting any additional patterns, such that the digital fringe projection (DFP) system can achieve higher measurement speed. We develop a multi-resolution system consisting of a DFP system and Kinect V2 to validate our method. Experiments demonstrate that our method works for a large depth range, and the speed of the low-cost 3D scanner is not necessarily the maximum speed of our proposed method. Assisted by Kinect V2 whose maximum speed is only 30Hz, our DFP system achieves 53Hz with a resolution 1600x1000 pixels when we measure dynamic objects that are moving in a large depth range of 400mm.

"Motion induced phase error reduction using a Hilbert transform," Opt. Express (2018)

 [117] Y. Wang, Z. Liu, and C. Jiang, and S. Zhang, "Motion induced phase error reduction using a Hilbert transform," Opt. Express 26(26), 34224-34235 (2018); doi:10.1364/OE.26.034224

Abstract

The motion of object could introduce phase error and thus measurement error for phase-shifting profilometry. This paper proposes a generic motion error compensation method based on our finding that the dominant motion introduced phase error doubles the frequency of the projected fringe frequency, and Hilbert transform shifts the phase of a fringe pattern by $\pi/2$. We apply Hilbert transform to phase-shifted fringe patterns to generate another set of fringe patterns, calculate one phase map using the original fringe patterns and another phase map using Hilbert transformed fringe patterns, and then use the average of these two phase maps for 3D reconstruction. Both simulation and experiments demonstrated the proposed method can substantially reduce motion-introduced measurement error.

"Depth-driven variable-frequency sinusoidal fringe pattern for accuracy improvement in fringe projection profilometry," Opt. Express (2018)

 [116] G. Rao, L. Song, and S. Zhang, X. Yang, K. Chen, and J. Xu, "Depth-driven variable-frequency sinusoidal fringe pattern for accuracy improvement in fringe projection profilometry," Opt. Express 26(16), 19986-20008 (2018); doi:10.1364/OE.26.019986

Abstract

Sinusoidal fringe pattern is widely used in optical profilometry; however, the traditional constant-frequency sinusoidal fringe pattern reduces 3D measurement accuracy in the defocus region. To this end, this paper presents a variable-frequency sinusoidal fringe pattern method that is optimized by the measurement depth. The proposed method  improves the pixel matching accuracy and thus increases measurement accuracy. This paper  1) theoretically determines the optimal frequency by analyzing the pixel matching error caused by intense noise in a captured image; 2) presents the online frequency  optimization along abscissa and ordinate axes in the sinusoidal fringe patterns; 3) details the encoding and decoding to use variable-frequency  fringe patterns for 3D profilometry. Simulations and experiments demonstrate that that our proposed method can improve the 3D measurement accuracy and increase measurement robustness.

"Guest Editorial Focused Section on sensing and perception systems for intelligent manufacturing (SPIM)," IEEE/ASME Trans. Mechatronics (2018)

[115] X. Chen, S. Zhang, and J. M.P. Geraedts, "Guest Editorial Focused Section on sensing and perception systems for intelligent manufacturing (SPIM)," IEEE/ASME Trans. Mechatronics 23(3), 983-984 (2018); doi:10.1109/TMECH.2018.2837008

Abstract

This Focused Section provides a state-of-the-art update of research fronts in Field Sensing and Perception as  well as the applications to intelligent manufacturing problems. The topics covered in the collected papers include: optimization of field sensing network and systems, intelligent sensing and perception for robotic manufacturing processes, SLAM of indoor environments, and model guided measurement of various field effects. The problems related to field sensing and perception are challenging and spread out in various areas in practices. To our best knowledge, this Focused Section is the first effort to try to collectively present the research results related sensing and perception of ‘field effects’ which exist in various manufacturing  processes. It is our intention that this Focused Section would serve as an efficient highlight of such challenging and important problems to attract intensive research interests on these problems by arguably claiming an effective research area for field sensing and perception. 

"Motion-induced error compensation for phase shifting profilometry," Opt. Express (2018)

[114] Z. Liu, P. Zibley and S. Zhang, "Motion-induced error compensation for phase shifting profilometry," Opt. Express, 26(10), 12632-12637 (2018); doi:10.1364/OE.26.012632

Abstract

This paper proposes a novel method to substantially reduce motion-introduced phase error in phase-shifting profilometry. We first estimate the motion of an object from the difference between two subsequent 3D frames. After that, by leveraging the projector’s pinhole model, we can determine the motion-induced phase shift error from the estimated motion. A generic phase-shifting algorithm considering phase shift error is then utilized to compute the phase. Experiments demonstrated that proposed algorithm effectively improved the measurement quality by compensating for the phase shift error introduced by rigid and nonrigid motion for a standard single-projector and single-camera digital fringe projection system.

"Three-dimensional shape measurement using a structured light system with dual projectors", Appl. Opt., (2018)

[113] C. Jiang, B. Lim and S. Zhang, "Three-dimensional shape measurement using a structured light system with dual projectors," Appl. Opt.,  57(14), 3983-3990(2018); doi:10.1364/AO.57.003983

This paper introduces a structured light system with two projectors and one camera for three-dimensional (3D) shape measurement to alleviate problems created by a single projector such as the shadow problem. In particular, we developed (1) a system calibration framework that can accurately calibrate each such camera-projector system; (2) a residual error correction method based on the system error function; and (3) a data fusion method utilizing the angle between the projection direction and surface normal. Experimental results demonstrate that the proposed dual-projector structured light system improves the measurement accuracy besides extending the measurement range of a single projector system.

"High-speed 3D shape measurement with structured light methods: a review," Opt. Laser Eng. (2018)

[111] S. Zhang, "High-speed 3D shape measurement with structured light methods: a review," Opt. Laser Eng. 106, 119-131 (2018)

Abstract

High-speed 3D shape measurement (or imaging) has seen tremendous growths over the past decades, especially the past few years due to the improved speed of computing devices and reduced costs of hardware components. 3D shape measurement technologies have started penetrating more into our daily lives than ever before with the recent release of iPhone X that has an built-in 3D sensor for Face ID, along with prior commercial success of inexpensive commercial sensors (e.g., Microsoft Kinect).  This paper overviews the primary state-of-the-art  3D shape measurement techniques based on structured light methods, especially those that could achieve high measurement speed and accuracy. The fundamental principles behind those technologies will be elucidated,  experimental results will be presented to demonstrate capabilities and/or limitations for those popular techniques, and finally present our perspectives on those remaining challenges to be conquered to make advanced 3D shape measurement techniques ubiquitous.

"Absolute phase retrieval methods for digital fringe projection profilometry: A review," Opt. Laser Eng. (2018)

[112] S. Zhang, "Absolute phase retrieval methods for digital fringe projection profilometry: A review," Opt. Laser Eng. 107, 28-37 (2018); doi:10.1016/j.optlaseng.2018.03.003

Abstract

This paper provides a review for absolute phase recovery methods that are applicable for digital fringe projection (DFP) systems. Specifically, we present two conventional absolute phase unwrapping methods: multi-frequency or -wavelength phase-shifting methods, and hybrid  binary coding and phase-shifting methods; and also introduce some non-conventional methods that are specific for DFP systems: multiview geometry methods with additional camera(s) or projector(s), DFP system geometric constraint-based phase unwrapping method, and pre-knowledge (e.g., computer-aided-design, CAD, model) based phase unwrapping method. This paper also briefly overviews hybrid methods including phase coding, composite, and pre-defined markers based absolute phase unwrapping methods. This paper explains the principle behind each individual absolute phase unwrapping method; and finally offers some practical tips to handle common phase unwrapping artifact issues.

"High-speed and high-accuracy 3D surface measurement using a mechanical projector," Opt. Express (2018)

J. -S. Hyun, George T. -C. Chiu and S. Zhang, "High-speed and high-accuracy 3D surface measurement using a mechanical projector," Opt. Express, 26(2), 1474-1487 (2018); doi:10.1364/OE.26.001474

Abstract

This paper presents a method to achieve high-speed and high-accuracy 3D surface measurement using a custom-designed mechanical projector and two high-speed cameras. We developed a computational framework that can achieve absolute shape measurement in sub-pixel accuracy through: 1) capturing precisely phase-shifted fringe patterns by synchronizing the cameras with the projector; 2) generating a rough disparity map between two cameras by employing a standard stereo-vision method using texture images with encoded statistical patterns; and 3) utilizing the wrapped phase as a constraint to refine the disparity map. The projector can project binary patterns at a speed of up to 10,000 Hz, and the camera can capture the required number of phase-shifted fringe patterns with 1/10,000 second, and thus 3D shape measurement can be realized as high as 10,000 Hz regardless the number of phase-shifted fringe patterns required for one 3D reconstruction. Experimental results demonstrated the success of our proposed method.

"Novel method for measuring dense 3D strain map of robotic flapping wings," Measurement Science and Technology (2018)

[109] B. Li and S. Zhang, "Novel method for measuring dense 3D strain map of robotic flapping wings," Measurement Science and Technology, 29(4), 045402 (2018);

Abstract

Measuring dense 3D strain map of inextensible membranous flapping wings of robots is of vital importance to the field of bio-inspired engineering. Conventional high-speed 3D videography method typically reconstructs the wing geometries through measuring sparse points with fiducial markers, and thus cannot obtain full-field mechanics of the wings in details.  In this research, we propose a novel system to measure dense strain map of the  inextensible membranous flapping wings by developing a superfast 3D imaging system and a computational framework for strain analysis. Specifically, first, we developed a 5,000 Hz 3D imaging system based on the digital fringe projection technique using the defocused binary patterns to precisely measure the dynamic 3D geometries of rapidly flapping wings. Then, we developed a geometry-based algorithm to perform point tracking on the precisely measured 3D surface data. Finally, we developed a dense strain computational method using the Kirchhoff-Love shell theory. Experiments demonstrate that our method can effectively perform point tracking and measure highly dense strain map of the wings without many fiducial markers.

"Introduction to the focused section on sensing and perception for autonomous and networked robotics" (2017)

X. Chen, M. Grossard, N. Kubota, D. Wollherr, S. X. Yang, and S. Zhang, "Introduction to the focused section on sensing and perception for autonomous and networked robotics," Int. J. Intell. Robot. Applic. 1 (4), 369–371 (2017); (Focused Section Guest Editorial) doi:10.1007/s41315-017-0040-8

"Double-pattern triangular pulse width modulation technique for high-accuracy high-speed 3D shape measurement," Opt. Express (2017)

Y. Wang, C. Jiang, and S. Zhang, "Double-pattern triangular pulse width modulation technique for high-accuracy high-speed 3D shape measurement," Opt. Express 25(24), 30177-30188 (2017); doi:10.1364/OE.25.03177

Abstract

Using 1-bit binary patterns for 3D shape measurement has been demonstrated advantageous over using  8-bit sinusoidal patterns in terms of achievable speeds. However, the phase quality generated by binary pattern(s) typically  is not  high if only a small number of phase-shifted patterns is used. This paper proposes a method to improve the phase quality  by representing each pattern with the difference of two binary patterns:  the first binary pattern is generated by triangular pulse width modulation (TPWM) technique, and the second being $\pi$ shifted from the first pattern is also generated by TPWM technique. The phase is retrieved by applying a three-step phase-shifting algorithm to the difference patterns. Through optimizing the modulation frequency of the  triangular carrier signal, we demonstrate that high-quality phase can be generated for a wide range of fringe periods (e.g., from 18 to 1140 pixels) with only six binary patterns. Since only 1-bit binary patterns are required for 3D shape measurement, this paper will present a real-time 3D shape measurement system that can achieve 30 Hz.

"Three-dimensional range geometry compression via phase encoding," Appl. Opt. (2017)

[106] T. Bell, B. Vlahov, J.P. Allebach, and S. Zhang, "Three-dimensional range geometry compression via phase encoding," Appl. Opt., 56(33), 9285-9292, (2017); doi: 10.1364/AO.56.009285

Abstract

One of the state-of-the-art methods for three-dimensional (3D) range geometry compression is to encode 3D data within a regular 24-bit 2D color image. However, most existing methods use all three color channels to solely encode 3D data, leaving no room to store other information (e.g., texture) within the same image. This paper presents a novel method which utilizes geometric constraints, inherent to the structured light 3D scanning device, to reduce the amount of data which need be stored within the output image. The proposed method thus only requires two color channels to represent 3D data, leaving one channel free to store additional information (such as a texture image). Experimental results verify the overall robustness of the proposed method. For example, a compression ratio of 3038:1 can be achieved, versus the STL format, with a root-mean-square (RMS) error of 0.47% if the output image is compressed with JPEG 80%.

Technical Paper

“Superfast, high-resolution absolute 3D recovery of a stabilized flapping flight process,” Opt. Express, (2017)

B. Li and S. Zhang, “Superfast, high-resolution absolute 3D recovery of a stabilized flapping flight process,” Opt. Express, 25(22), 27270-27282 (2017); doi:10.1364/OE.25.027270

Abstract

Scientific research of a stabilized flapping flight process (e.g. hovering) has been of great interest to a variety of fields including biology, aerodynamics and bio-inspired robotics. Different from the current passive photogrammetry based methods, the digital fringe projection (DFP) technique has the capability of performing dense superfast (e.g. kHz) 3D topological reconstruction with the projection of defocused binary patterns, yet it is still a challenge to measure a flapping flight process with the presence of rapid flapping wings. This paper presents a novel absolute 3D reconstruction method for a stabilized flapping flight process. Essentially, the slow motion parts (e.g. body) and the fast-motion parts (e.g. wings) are segmented and separately reconstructed with phase shifting techniques and Fourier transform, respectively. The topological relations between the wings and the body are utilized to ensure absolute 3D reconstruction. Experiments demonstrate the success of our computational framework by testing a flapping wing robot at different flapping speeds.

“Absolute three-dimensional shape measurement with two-frequency square binary patterns,” Appl. Opt., (2017)

C. Jiang and S. Zhang, “Absolute three-dimensional shape measurement with two-frequency square binary patterns,” Appl. Opt., 56(31), 8710-8718 (2017); doi:10.1364/AO.56.008710

Abstract

This paper presents a novel method to achieve absolute three-dimensional (3D) shape measurement solely using square binary patterns. This method uses six patterns: three low-frequency phase-shifted patterns and three phase-shifted high-frequency patterns. The phase obtained from low-frequency phase temporally unwraps the phase obtained from high-frequency patterns. The projector is defocused such that the high-frequency patterns produce high-quality phase, but the phase retrieved from low-frequency patterns has large harmonic error that fails two-frequency temporal phase unwrapping process. In this paper, we develop a computational framework to address the challenge. The proposed computational framework includes four major approaches to alleviate the harmonic error problem: i) use more than one period of low-frequency patterns enabled by geometric constraint-based phase unwrapping method; ii) artificially apply a large Gaussian filter to low frequency patterns before phase computation; iii) create an error lookup table (LUT) to compensate for harmonic error; and iv) develop a boundary error correction method to alleviate problems associated with filtering. Both simulation and experimental results demonstrated the success of the proposed method.

"Three dimensional range geometry and texture data compression with space-filling curves," Opt. Express (2017)

X. Chen and S. Zhang, "Three dimensional range geometry and texture data compression with space-filling curves," Opt. Express 25(21), 26148-26159 (2017); doi:10.1364/OE.25.026103

Abstract

This paper presents a novel method to effectively store three-dimensional (3D) data and 2D texture data into a regular 24-bit image. The proposed method uses the Hilbert space-filling curve to map the normalized unwrapped phase map to two 8-bit color channels, and saves the third color channel for 2D texture storage. By further leveraging existing 2D image and video compression techniques, the proposed method can achieve high compression ratios while effectively preserving data quality.  Since the encoding and decoding processes can be applied to most of the current 2D media platforms, this proposed compression method can make 3D data storage and transmission available for many electrical devices without requiring special hardware changes. Experiments demonstrate that if a lossless 2D image/video format is used, both original 3D geometry and 2D color texture can be accurately recovered; if lossy image/video compression is used, only black-and-white or grayscale texture can be properly recovered, but much higher compression ratios (e.g., 1543:1 against the ASCII OBJ format) are achieved with slight loss of 3D geometry quality.

"Dynamic projection theory for fringe projection profilometry," Appl. Opt., (2017)

[102] H. Sheng, J. Xu, and S. Zhang, "Dynamic projection theory for fringe projection profilometry," Appl. Opt., 56(30), 8452-8460 (2017); doi: 10.1364/AO.56.008452

Abstract

 Fringe projection profilometry (FPP)  has been widely used for 3D reconstruction, surface measurement and reverse engineering. However, fringe projection profilometry is prone to overexposure if objects have a wide range of reflectance. In this paper, we propose a dynamic projection theory based on fringe projection profilometry to rapidly measure the overexposed region with an attempt to conquer this challenge. This theory modifies the projected fringe image to the next better measurement based on the feedback provided by the previously captured image intensity. Experiments demonstrated that the number of overexposed points can be drastically reduced after one or two iterations. Compared with the state-of-the-art methods, our proposed dynamic projection theory measures the overexposed region quickly and effectively, and thus broadens the applications of fringe projection profilometry.

“Absolute phase unwrapping for dual-camera system without embedding statistical features,” Opt. Eng., (2017)

C. Jiang and S. Zhang, “Absolute phase unwrapping for dual-camera system without embedding statistical features,” Opt. Eng. 56(9), 094114 (2017), doi: 10.1117/1.OE.56.9.094114.

Abstract

This paper proposes an absolute phase unwrapping method for 3D measurement that uses two cameras and one projector. On the left camera image, each pixel has one wrapped phase value which corresponds to multiple projector candidates with different absolute phase values. We use geometric relationship of the system to map projector candidates into right camera candidates. By applying a series of candidate rejection criteria, a unique correspondence pair between two camera images can be determined. Then the absolute phase is obtained by tracing the correspondence point back to projector space. Experimental results demonstrate that the proposed absolute phase unwrapping algorithm can successfully work on both complex geometry and multiple isolated objects measurement.