"High-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques," Journal of Visualized Experiments, (2013)

L. Ekstrand*, N. Karpinsky*, Y. Wang*, and S. Zhang, "High-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques," Journal of Visualized Experiments (JoVE), (82), e50421, 2013. (Associated with Video Illustrations) (invited); doi: 10.3791/50421

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces. Like the human eyes and brain, DFP uses triangulation between matching points in two views of the same scene at different angles to compute depth. However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras. The projector rapidly projects a known sinusoidal pattern onto the subject, and the surface of the subject distorts these patterns in the camera’s field of view. Three distorted patterns (fringe images) from the camera can be used to compute the depth using triangulation.

Unlike other 3D measurement methods, DFP techniques lead to systems that tend to be faster, lower in equipment cost, more flexible, and easier to develop. DFP systems can also achieve the same measurement resolution as the camera. For this reason, DFP and other digital structured light techniques have recently been the focus of intense research (as summarized in1-5). Taking advantage of DFP, the graphics processing unit, and optimized algorithms, we have developed a system capable of 30 Hz 3D video data acquisition, reconstruction, and display for over 300,000 measurement points per frame. Binary defocusing DFP methods can achieve even greater speeds.

Diverse applications can benefit from DFP techniques. Our collaborators have used our systems for facial function analysis9, facial animation10, cardiac mechanics studies11, and fluid surface measurements, but many other potential applications exist. This video will teach the fundamentals of DFP techniques and illustrate the design and operation of a binary defocusing DFP system.

"Optimal fringe angle selection for digital fringe projection technique," Appl. Opt., (2013)

[55] Y. Wang*, and S. Zhang, "Optimal fringe angle selection for digital fringe projection technique," Appl. Opt. 52(29),  7094-7098, 2013; doi: 10.1364/AO.52.007094

Abstract

Existing digital fringe projection (DFP) systems mainly use either horizontal or vertical fringe patterns for three-dimensional shape measurement. This paper reveals that these two fringe directions are usually not optimal where the phase change is the largest to a given depth variation. We propose a novel and efficient method to determine the optimal fringe angle by projecting a set of horizontal and vertical fringe patterns onto a step-height object and by further analyzing two resultant phase maps. Experiments demonstrate the existence of the optimal angle and the success of the proposed optimal angle determination method.

"Intensity-optimized dithering technique for high-quality 3d shape measurement," Opt. Laser Eng., 2014

J. Dai, B. Li*, and S. Zhang, "Intensity-optimized dithering technique for high-quality 3d shape measurement," Opt. Laser Eng. 53, 79-85, 2014; doi: 10.1016/j.optlaseng.2013.08.015

Our previously proposed phase-based optimization method [1] has proven successful in improving the measurement quality when a dithering technique is used. This paper presents an intensity-based optimization method for 3D shape measurement with binary dithering techniques. Both simulations and experiments find that the phase-based optimization method can generate high-quality phase under a given condition, but it is sensitive to the amount of defocusing. In contrast, the proposed intensity-based optimization method can consistently generate high-quality phase with various amounts of defocusing.

"High-quality fringe pattern generation using binary pattern optimization through symmetry and periodicity," Opt. Laser Eng., 2014

J. Dai, B. Li*, and S. Zhang, "High-quality fringe pattern generation using binary pattern optimization through symmetry and periodicity," Opt. Laser Eng., 52, 195-200, 2014; doi: 10.1016/j.optlaseng.2013.06.010

This paper presents a novel method to construct binary patterns for high-quality 3D shape measurement. The algorithm generates small patches using symmetry and periodicity, randomly initializes each pixels, optimizes the small patches through mutations, and finally tiles the optimized patches into full size patterns using again symmetry and periodicity. We will demonstrate that the proposed method can achieve substantial phase quality improvements over the dithering techniques for different amounts of defocusing.

"Flexible real-time natural 2D color and 3D shape measurement," Opt. Express, 2013;

P. Ou, B. Li*, Y. Wang*, and S. Zhang, "Flexible real-time natural 2D color and 3D shape measurement," Opt. Express,21(14), 16736-16741, 2013; doi: 10.1364/OE.21.016736

The majority of existing real-time 3D shape measurement systems only generate non-nature texture (i.e., having illumination other than ambient lights) that induces shadow related issues. This paper presents a method that can simultaneously capture natural 2D color texture and 3D shape in real time. Specifically, we use an infrared fringe projection system to acquire 3D shapes, and a secondary color camera to simultaneously capture 2D color images of the object. Finally, we develop a flexible and simple calibration technique to determine the mapping between the 2D color image and the 3D geometry. Experimental results demonstrate the success of the proposed technique.  

"Three bit representation of three-dimensional range data," Appl. Opt. , (2013)

[54] N. Karpinsky*, Y. Wang*, and S. Zhang, "Three bit representation of three-dimensional range data," Appl. Opt. 52(11), 2286-2293, 2013; doi: 10.1364/AO.52.002286

Abstract

Our previous research has shown that 3D range data sizes can be substantially reduced if they are converted into regular 2D images using the Holoimage technique. Yet, this technique requires all 24 bits of a standard image to represent one 3D point, making it impossible for a regular 2D image to carry 2D texture information as well. This paper proposes an approach to represent 3D range data with 3 bits, further reducing the data size. We demonstrate that more than an 8.2∶1 compression ratio can be achieved with compression root-mean-square error of only 0.34%. Moreover, we can use another bit to represent a black-and-white 2D texture, and thus both 3D data and 2D texture images can be stored into an 8 bit grayscale image. Both simulation and experiments are presented to verify the performance of the proposed technique.

"Natural method for three-dimensional range data compression," Appl. Opt. , (2013)

[53] P. Ou and S. Zhang, "Natural method for three-dimensional range data compression," Appl. Opt. 52(9), 1857-1863, 2013;doi: 10.1364/AO.51.004058

Abstract

Prior studies on converting three-dimensional (3D) range data into regular two-dimensional (2D) color images using virtual fringe projection techniques showed great promise for 3D range data compression, yet they require resampling the raw scanned data. Due to this resampling, the natural 3D range data are altered and sampling error may be introduced. This paper presents a method that compresses the raw sampling points without modifications. Instead of directly utilizing the 3D recovered shape, this method compresses the s map, the scale factor of a perspective projection from a 3D space to a 2D space. The s map is then converted to 2D color image for further compression with existing 2D image compression techniques. By this means, 3D data obtained by 3D range scanners can be compressed into 2D images without any resampling, providing a natural and more accurate method of compressing 3D range data. Experimental results verified the success of the proposed method.

"3D absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique," Opt. Express , (2013)

[52] Y. Wang*, J. I. Laughner, I. R. Efimov, and S. Zhang, "3D absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique," Opt. Express 21(5), 5822-5832, 2013 (Cover feature)  (Selected for May 22, 2013 issue of The Virtual Journal for Biomedical Optics); doi: 10.1364/OE.21.005822

Abstract

This paper presents a two-frequency binary phase-shifting technique to measure three-dimensional (3D) absolute shape of beating rabbit hearts. Due to the low contrast of the cardiac surface, the projector and the camera must remain focused, which poses challenges for any existing binary method where the measurement accuracy is low. To conquer this challenge, this paper proposes to utilize the optimal pulse width modulation (OPWM) technique to generate high-frequency fringe patterns, and the error-diffusion dithering technique to produce low-frequency fringe patterns. Furthermore, this paper will show that fringe patterns produced with blue light provide the best quality measurements compared to fringe patterns generated with red or green light; and the minimum data acquisition speed for high quality measurements is around 800 Hz for a rabbit heart beating at 180 beats per minute.

"Phase-optimized dithering technique for high-quality 3D shape measurement," Opt. Laser Eng., (2013)

[51] J. Dai and S. Zhang, "Phase-optimized dithering technique for high-quality 3D shape measurement," Opt. Laser Eng. 51(6), 790-795, 2013; doi: 10.1016/j.optlaseng.2013.02.003

Abstract

Our recent study showed that the Bayer-dithering technique could substantially improve 3D measurement quality for the binary defocusing method. Yet, the dithering technique was developed to optimize the appearance or intensity representation, rather than the phase, of an image. This paper presents a framework to optimize the Bayer-dithering technique in phase domain by iteratively mutating the status (0 or 1) of a binary pixel. We will demonstrate that the proposed optimization technique can drastically reduce the phase error when the projector is nearly focused.

"Genetic method to optimize binary dithering technique for high-quality fringe generation," Opt. Lett. ,(2013)

[50] W. Lohry* and S. Zhang, "Genetic method to optimize binary dithering technique for high-quality fringe generation," Opt. Lett. 38(4), 540-542, 2013; doi: 10.1117/1.OE.51.11.113602

Abstract

The recently proposed dithering techniques could substantially improve measurement quality when fringes are wide, but offer limited improvement when fringes are narrow. This Letter presents a genetic algorithm to optimize the dithering technique for sinusoidal structured pattern representation. We believe both simulation and experimental results show that this proposed algorithm can substantially improve fringe quality for both narrow and wide fringe patterns. 

"3D range geometry video compression with the H.264 codec," Opt. Laser Eng. , (2013)

[49] N. Karpinsky* and S. Zhang, "3D range geometry video compression with the H.264 codec," Opt. Laser Eng. 51(5), 620-625, 2013; doi: 10.1016/j.optlaseng.2012.12.021

Abstract

Advances in three-dimensional (3D) scanning have enabled the real-time capture of high-resolution 3D videos. With these advances brings the challenge of streaming and storing 3D videos in a manner that can be quickly and effectively used. This research addresses this challenge by generalizing the Holovideo technique to video codecs that use the YUV color space such as the H.264 codec. With the H.264 codec, we have achieved a compression ratio of over 6086:1 (Holovideo to OBJ) with a reasonably high quality; utilizing an NVIDIA GeForce 9400 m GPU, we have realized 17 frames per second encoding, and 28 frames per second decoding speed, making it a viable solution for real-time 3D video compression.

"Accurate calibration for 3D shape measurement system using a binary defocusing technique," Opt. Laser Eng., (2013)

L. Merner*, Y. Wang*, and S. Zhang, "Accurate calibration for 3D shape measurement system using a binary defocusing technique," Opt. Laser Eng. 51(5), 514-519, 2013; doi: 10.1016/j.optlaseng.2012.10.015

This paper introduces a novel method to calibrate 3D shape measurement systems that use the binary defocusing technique. Specifically, this method calibrates the pixelwise z as low-order polynomial functions of absolute phase; (x, y) coordinates are calculated from camera calibration with known z value; and the camera is calibrated using the standard flat checkerboard method. Because this method does not require estimation of the projector’s parameters, it can be adopted for any phase measurement system including those employing out-of-focus projectors. Our experiment found that the root-mean squared (rms) error for the depth measurement is less than 70 mm when the measurement depth range is about 100 mm, which is at the same level of the calibration stage +-50 mm.

"Fourier transform profilometry using a binary area modulation technique," Opt. Eng., (2012)

W. Lohry* and S. Zhang, "Fourier transform profilometry using a binary area modulation technique," Opt. Eng. 51(11),113602, 2012; doi: 10.1117/1.OE.51.11.113602

A recent study found that it is very difficult to use the squared binary defocusing technique to eliminate the influence of third-order harmonics without compromising fringe quality, and thus it is challenging to utilize Fourier transform profilometry to achieve high-quality three-dimensional measurement. A novel approach is presented to effectively eliminate the third-order harmonics by modulating the squared binary structured patterns. Both simulation and experiments are presented to verify the performance of the proposed technique.

"Three-dimensional shape measurement with binary dithered patterns," Appl. Opt.,(2012)

Y. Wang* and S. Zhang, "Three-dimensional shape measurement with binary dithered patterns," Appl. Opt. 51(27), 6631-6636, 2012; doi: 10.1364/AO.51.006631

The previously proposed binary defocusing technique and its variations have proven successful for high-quality three-dimensional (3D) shape measurement when fringe stripes are relatively narrow, but they suffer if fringe stripes are wide. This paper proposes to utilize the binary dithering technique to conquer this challenge. Both simulation and experimental results show the phase error is always less than 0.6% even when the fringe stripes are wide and the projector is nearly focused.

"Mapping cardiac surface mechanics with structured light imaging," American Journal of Physiology: Heart and Circular Physiology, (2012)

J. I. Laughner, S. Zhang, H. Li, C. C. Shao, and I. R. Efimov, "Mapping cardiac surface mechanics with structured light imaging," American Journal of Physiology: Heart and Circular Physiology 303(6), H712-H720, 2012 (Image of the week of October 1, 2012, American Journal of Physiology); doi: 10.1152/ajpheart.00269.2012

Cardiovascular disease often manifests as a combination of pathological electrical and structural heart remodeling. The relationship between mechanics and electrophysiology is crucial to our understanding of mechanisms of cardiac arrhythmias and the treatment of cardiac disease. While several technologies exist for describing whole heart electrophysiology, studies of cardiac mechanics are often limited to rhythmic patterns or small sections of tissue. Here, we present a comprehensive system based on ultrafast three-dimensional (3-D) structured light imaging to map surface dynamics of whole heart cardiac motion. Additionally, we introduce a novel nonrigid motion-tracking algorithm based on an isometry-maximizing optimization framework that forms correspondences between consecutive 3-D frames without the use of any fiducial markers. By combining our 3-D imaging system with nonrigid surface registration, we are able to measure cardiac surface mechanics at unprecedented spatial and temporal resolution. In conclusion, we demonstrate accurate cardiac deformation at over 200,000 surface points of a rabbit heart recorded at 200 frames/s and validate our results on highly contrasting heart motions during normal sinus rhythm, ventricular pacing, and ventricular fibrillation.

"Three-dimensional range data compression using computer graphics rendering pipeline," Appl. Opt., (2012)

S. Zhang, "Three-dimensional range data compression using computer graphics rendering pipeline," Appl. Opt. 51(18), 4058-4064, 2012 (Cover feature); doi: 10.1364/AO.51.004058

This paper presents the idea of naturally encoding three-dimensional (3D) range data into regular two dimensional (2D) images utilizing computer graphics rendering pipeline. The computer graphics pipeline provides a means to sample 3D geometry data into regular 2D images, and also to retrieve the depth information for each sampled pixel. The depth information for each pixel is further encoded into red, green, and blue color channels of regular 2D images. The 2D images can further be compressed with existing 2D image compression techniques. By this novel means, 3D geometry data obtained by 3D range scanners can be instantaneously compressed into 2D images, providing a novel way of storing 3D range data into its 2D counterparts. We will present experimental results to verify the performance of this proposed technique.

"Novel phase coding method for absolute phase retrieval," Opt. Lett., (2012)

Y. Wang* and S. Zhang, "Novel phase coding method for absolute phase retrieval," Opt. Lett. 37(11), 2067-2069, 2012;doi: 10.1364/OL.37.002067

This Letter presents a novel absolute phase recovery technique with phase coding. Unlike the conventional graycoding method, the codeword is embedded into the phase and then used to determine the fringe order for absolute phase retrieval. This technique is robust because it uses phase instead of intensity to determine codewords, and it could achieve a faster measurement speed, since three additional images can represent more than 8
 unique codewords for phase unwrapping. Experimental results will be presented to verify the performance of the proposed technique. 

"3D shape measurement with 2D area modulated binary patterns," Opt. Laser Eng.,(2012)

W. Lohry* and S. Zhang, "3D shape measurement with 2D area modulated binary patterns," Opt. Laser Eng. 50(7), 917-921, 2012; doi: 10.1016/j.optlaseng.2012.03.002

This paper presents a novel area-modulation technique for three-dimensional (3D) shape measurement with binary defocusing. Specifically, this technique modulates local 2*2 pixels to create five grayscale values to enhance fringe quality when the projector is not perfectly in focus. With this novel technique, we will show that the phase error is approximately 1/3 of the square binary method when fringe pattern is dense and the projector is nearly focused. 

"Uniaxial three-dimensional shape measurement with projector defocusing," Opt. Eng., (2012)

Y. Xu* and S. Zhang, "Uniaxial three-dimensional shape measurement with projector defocusing," Opt. Eng. 51(2) 023604, 2012; doi: 10.1117/1.OE.51.2.023604

Our study shows that the phase error caused by improperly defocused binary structured patterns correlates to the depth z. This finding leads to a novel uniaxial three-dimensional shape measurement technique without triangulation. Since the measurement can be performed from the same viewing angle, this proposed method overcomes some limitations of the triangulation-based techniques, such as the problem of measuring deep holes. Our study explains the principle of the technique and presents some experimental results to verify its feasibility.

"Autoexposure for three-dimensional shape measurement with a digital-light-processing projector," Opt. Eng. (2011)

L. Ekstrand* and S. Zhang, "Autoexposure for three-dimensional shape measurement with a digital-light-processing projector," Opt. Eng. 50(12) 123603, 2011; doi: 10.1117/1.3662387

Automatically adapting the camera exposure time is crucial for industrial applications where minimum human intervention is usually desirable. However, it is very challenging to realize such a capability for a conventional fringe projection system where only a finite increment of the exposure time is allowed due to its digital fringe generation nature. We study the generation of sinusoidal fringe patterns by properly defocusing binary ones, which permits the use of an arbitrary exposure time. This provides the potential to adapt the exposure time automatically. We present the principle of an automatic exposure technique and show some experimental results.