"Flexible 3-D shape measurement using projector defocusing," Opt. Lett., (2009)

[20] S Lei* and S Zhang, "Flexible 3-D shape measurement using projector defocusing," Opt. Lett. 34(20),3080-3082, 2009;doi: 10.1364/OL.34.003080

We present a 3-D shape-measurement technique using a defocused projector. The ideal sinusoidal fringe patterns are generated by defocusing binary structured patterns, and the phase shift is realized by shifting the binary patterns spatially. Because this technique does not require calibration of the gamma of the projector, it is easy to implement and thus is promising for developing flexible 3-D shape measurement systems using digital video projectors. 

"High dynamic range scanning technique," Opt. Eng., (2009)

[18] S Zhang and S-T Yau, "High dynamic range scanning technique," Opt. Eng. 48(3), 033604, 2009; doi: 10.1117/1.3099720

Measuring objects with a high variation range of surface reflectivity is challenging for any optical method: This paper addresses a high dynamic range scanning technique that can measure this type of object. It takes advantage of one merit of a phase-shifting algorithm: pixel-by-pixel phase retrieval. For each measurement, a sequence of fringe images with different exposures are taken: the brightest ones have good fringe quality in the darkest areas while the darkest ones have good fringe quality in the brightest areas. They are arranged from brighter to darker
i.e., from higher exposure to lower exposure. The final fringe images, used for phase retrieval, are produced pixel-by-pixel by choosing the brightest but unsaturated corresponding pixel from one exposure. A phase-shifting algorithm is employed to compute the phase, which can be further converted to coordinates. Our experiments demonstrate that the proposed technique can successfully measure objects with high dynamic range of surface reflectivity variation. 
 

"Digital sinusoidal fringe generation: defocusing binary patterns VS focusing sinusoidal patterns," Opt. Laser, (2010)

[22] S. Lei* and S. Zhang, "Digital sinusoidal fringe generation: defocusing binary patterns VS focusing sinusoidal patterns,"Opt. Laser Eng. 48, 561-569, 2010; doi: 10.1016/j.optlaseng.2009.12.002

There are two approaches to generate sinusoidal fringe patterns with a digital-light-processing (DLP) projector: defocusing binary patterns (DBP), and focusing sinusoidal patterns (FSP). This paper will compare the potential errors for both methods induced by the following factors: (1) degree of defocusing, (2) exposure time, (3) synchronization, and (4) projector’s nonlinear gamma. Experiments show that in most scenarios, the error for the DBP method is smaller than that of the FSP method. Therefore, generating a sinusoidal fringe image using a DBP method seems to be appealing.

"Simultaneous three-dimensional geometry and color texture acquisition using a single-chip color camera," Opt. Eng., (2008)

[17] S Zhang and S-T Yau, "Simultaneous three-dimensional geometry and color texture acquisition using a single-chip color camera," Opt. Eng. 47(12), 123604, 2008; doi: 10.1117/1.3046715

A novel technique that uses a single color camera to capture high-resolution three-dimensional
3-D geometry and the perfectly aligned color texture simultaneously is discussed. A projector projects three phase-shifted black-and-white fringe patterns onto the object, and a color camera captures the fringe images reflected by the object. From these three fringe images, both 3-D shape and the color texture are obtained. Moreover, since only three fringe images are required, this proposed technique permits real-time 3-D shape measurement. Experiments are presented to demonstrate the success of this technique. 

"Absolute phase assisted three-dimensional data registration for a dual-camera structured light system," Appl. Opt., (2008)

[16] S Zhang and S-T Yau, "Absolute phase assisted three-dimensional data registration for a dual-camera structured light system,"  Appl. Opt., 47(17), 3134-3142, 2008 (Cover Feature); doi: 10.1364/AO.47.003134

For a three-dimensional shape measurement system with a single projector and multiple cameras, registering patches from different cameras is crucial. Registration usually involves a complicated and timeconsuming procedure. We propose a new method that can robustly match different patches via absolute phase without significantly increasing its cost. For y and z coordinates, the transformations from one camera to the other are approximated as third-order polynomial functions of the absolute phase. The x coordinates involve only translations and scalings. These functions are calibrated and only need to be determined once. Experiments demonstrated that the alignment error is within RMS 0:7 mm. © 2008 Optical Society of America

"Novel 3-D video for quantification of facial movement," Otolaryngol Head Neck Surg., (2008)

[15] RP Mehta, S Zhang, and TA Hadlock , "Novel 3-D video for quantification of facial movement,"  Otolaryngol Head Neck Surg., 138(4), 468-472, 2008; doi: 10.1016/j.otohns.2007.12.017

OBJECTIVES: To evaluate a novel 3-D geometry video acquisition system (3-D VAS) for quantifying facial movement. METHODS: Ten normal subjects and four patients with facial paralysis underwent recording of facial movement during three facial expressions. Facial feature points of interest were digitally marked on selected images. The maximum 3-D displacement of the brow and oral commissure and palpebral fissure width were determined. RESULTS: Test-retest reliability using the 3-D VAS in normal subjects was high (Spearman correlation coefficients 0.661-0.830, P
0.05). The predicted ratio of left to right facial measurements was 1.00 in normal subjects, and measured mean ratios were 1.05 (brow excursion), 1.01 (palpebral fissure width), and 1.03 (oral commissure excursion), respectively, matching the prediction. The 3-D VAS reliably quantified facial movement on both sides in patients with facial paralysis. CONCLUSIONS: The novel 3-D VAS can accurately and reliably quantify facial movement in healthy subjects. It is promising as a clinical tool to quantify facial movement in patients with facial paralysis.

"High resolution tracking of non-rigid 3D motion of densely sampled data using harmonic maps," Intl J. Comp. Vis., (2008)

[14] Y Wang, M Gupta, S Zhang, S Wang, X Gu, D Samaras, and P Huang, "High resolution tracking of non-rigid 3D motion of densely sampled data using harmonic maps," Intl J. Comp. Vis., 76(3), 283-300, 2008; doi: 10.1007/s11263-007-0063-y

We present a novel automatic method for high resolution, non-rigid dense 3D point tracking. High quality dense point clouds of non-rigid geometry moving at video speeds are acquired using a phase-shifting structured light ranging technique. To use such data for the temporal study of subtle motions such as those seen in facial expressions, an efficient non-rigid 3D motion tracking algorithm is needed to establish inter-frame correspondences. The novelty of this paper is the development of an algorithmic framework for 3D tracking that unifies tracking of intensity and geometric features, using harmonic maps with added feature correspondence constraints. While the previous uses of harmonic maps provided only global alignment, the proposed introduction of interior feature constraints allows to track non-rigid deformations accurately as well. The harmonic map between two topological disks is a diffeomorphism with minimal stretching energy and bounded angle distortion. The map is stable, insensitive to resolution changes and is robust to noise. Due to the strong implicit and explicit smoothness constraints imposed by the algorithm and the high-resolution data, the resulting registration/deformation field is smooth, continuous and gives dense one-to-one interframe correspondences. Our method is validated through a series of experiments demonstrating its accuracy and efficiency.

"Three-dimensional data merging using Holoimage," Opt. Eng., (2008)

[13] S Zhang and S-T Yau, "Three-dimensional data merging using Holoimage," Opt. Eng., 47(3), 033608, 2008 (Cover Feature); doi: 10.1117/1.2898902

Three-dimensional data merging is vital for full-field threedimensional
3D shape measurement. All 3D range data patches, acquired from either different sensors or the same sensor in different viewing angles, have to be merged into a single piece to facilitate future data analysis. A novel method for 3D data merging using Holoimage is proposed. Similar to the 3D shape measurement system using a phaseshifting method, Holoimage is a phase-shifting–based computer synthesized fringe image. The 3D information is retrieved from Holoimage using a phase-shifting method. If two patches of 3D data with overlapping areas are rendered by OpenGL, the overlapping areas are resolved by the graphics pipeline, that is, only the front geometry can be visualized. Therefore, the merging is performed if the front geometry information can be obtained. Holoimage is to obtain the front geometry by projecting the fringe patterns onto the rendered scene. We also demonstrated that each point of the geometry in the overlapping area can be obtained by averaging the corresponding point of the geometries reconstructed from Holoimage for each patch. Moreover, using Holoimage, the texture can also be obtained. Both simulation and experiments demonstrated the success of the proposed method. 

"Three-dimensional shape measurement using a structured light system with dual cameras," Opt. Eng., (2008)

[12] S Zhang and S-T Yau, "Three-dimensional shape measurement using a structured light system with dual cameras," Opt. Eng., 47(1), 013604, 2008; doi: 10.1117/1.2835686

A structured light system for three-dimensional shape measurement with single camera has the shortcoming of camera occlusion. To alleviate this problem, this paper introduces a structured light system with dual cameras for three-dimensional shape measurement. We discuss
1 system description,
2 system calibration,
3 three-dimensional data registration using the iterative closest-point
ICP algorithm, and
4 three-dimensional data merging using holoimage. The principle of the system is introduced, and experiments are presented to verify its performance.

"High-speed three-dimensional shape measurement system using a modified two-plus-one phase-shifting algorithm," Opt. Eng., (2007)

[11] S Zhang and S-T Yau, "High-speed three-dimensional shape measurement system using a modified two-plus-one phase-shifting algorithm," Opt. Eng., 46(11), 113603, 2007; doi:10.1117/1.2802546

This paper describes a high-resolution, real-time, threedimensional shape measurement system using the modified two-plusone phase-shifting algorithm. The data acquisition speed is as high as 60 frames/ s with an image resolution of 640
480 pixels per frame. Experiments demonstrated that the system was able to acquire the dynamic changing objects such as facial geometric shape changes when the subject is speaking, and the modified two-plus-one phase-shifting algorithm can further alleviate the error due to motion. Applications of this system include manufacturing, online inspection, medical imaging, compute vision, and computer graphics. 

"Phase error compensation for a 3-D shape measurement system based on the phase shifting method," Opt. Eng., (2007)

[10] S Zhang and PS Huang, "Phase error compensation for a 3-D shape measurement system based on the phase shifting method," Opt. Eng., 46(6), 063601, 2007; doi:10.1117/1.2746814

This paper describes a novel phase error compensation method for reducing the measurement error caused by nonsinusoidal waveforms in phase-shifting methods. For 3-D shape measurement systems using commercial video projectors, the nonsinusoidal waveform of the projected fringe patterns as a result of the nonlinear gamma of projectors causes significant phase measurement error and therefore shape measurement error. The proposed phase error compensation method is based on our finding that the phase error due to the nonsinusoidal waveform depends only on the nonlinearity of the projector’s gamma. Therefore, if the projector’s gamma is calibrated and the phase error due to the nonlinearity of the gamma is calculated, a lookup table that stores the phase error can be constructed for error compensation. Our experimental results demonstrate that by using the proposed method, the measurement error can be reduced by 10 times. In addition to phase error compensation, a similar method is also proposed to correct the nonsinusoidality of the fringe patterns for the purpose of generating a more accurate flat image of the object for texture mapping. While not relevant to applications in metrology, texture mapping is important for applications in computer vision and computer graphics.

"Generic nonsinusoidal phase error correction for three-dimensional shape measurement using a digital video projector," Appl. Opt., (2007)

[9] S Zhang and S-T Yau, "Generic nonsinusoidal phase error correction for three-dimensional shape measurement using a digital video projector," Appl. Opt., 46(1), 36-43, 2007; doi: 10.1364/AO.46.000036

A structured light system using a digital video projector is widely used for 3D shape measurement. However, the nonlinear
of the projector causes the projected fringe patterns to be nonsinusoidal, which results in phase error and therefore measurement error. It has been shown that, by using a small look-up table (LUT), this type of phase error can be reduced significantly for a three-step phase-shifting algorithm. We prove that this algorithm is generic for any phase-shifting algorithm. Moreover, we propose a new LUT generation method by analyzing the captured fringe image of a flat board directly. Experiments show that this error compensation algorithm can reduce the phase error to at least 13 times smaller.

"Multilevel quality-guided phase unwrapping algorithm for real-time 3-D shape reconstruction,"Appl. Opt., (2007)

[8] S Zhang, X Li and S-T Yau, "Multilevel quality-guided phase unwrapping algorithm for real-time 3-D shape reconstruction,"Appl. Opt. 46(1), 50-57, 2007 (Selected for February 5, 2007 issue of The Virtual Journal for Biomedical Optics); doi: 10.1364/AO.46.000050

A multilevel quality-guided phase unwrapping algorithm for real-time 3D shape measurement is presented. The quality map is generated from the gradient of the phase map. Multilevel thresholds are used to unwrap the phase level by level. Within the data points in each level, a fast scan-line algorithm is employed. The processing time of this algorithm is approximately 18.3 ms for an image size of 640
480 pixels in an ordinary computer. We demonstrate that this algorithm can be implemented into our real-time 3D shape measurement system for real-time 3D reconstruction. Experiments show that this algorithm improves the previous scan-line phase unwrapping algorithm significantly although it reduces its processing speed slightly

"High-resolution real-time three-dimensional shape measurement," Opt. Eng., (2006)

[7] S Zhang and PS Huang, "High-resolution real-time three-dimensional shape measurement," Opt. Eng., 45(12), 123601, 2006 ; doi: 10.1117/1.2402128 

We describe a high-resolution, real-time 3-D shape measurement system based on a digital fringe projection and phase-shifting technique. It utilizes a single-chip digital light processing projector to project computer-generated fringe patterns onto the object, and a high-speed CCD camera synchronized with the projector to acquire the fringe images at a frame rate of 120 frames/ s. A color CCD camera is also used to capture images for texture mapping. Based on a three-step phaseshifting technique, each frame of the 3-D shape is reconstructed using three consecutive fringe images. Therefore the 3-D data acquisition speed of the system is 40 frames/ s. With this system, together with the fast three-step phase-shifting algorithm and parallel processing software we developed, high-resolution, real-time 3-D shape measurement is realized at a frame rate of up to 40 frames/ s and a resolution of 532
500 points per frame

"GPU-assisted high-resolution, real-time 3-D shape measurement," Opt. Express, (2006)

[6] S Zhang, D Royer and S-T Yau, "GPU-assisted high-resolution, real-time 3-D shape measurement," Opt. Express, 14, 9120-9129, 2006 (Selected for November 13, 2006 issue of The Virtual Journal for Biomedical Optics); doi: 10.1364/OE.14.009120

This paper describes a Graphics Processing Unit (GPU)-assisted real-time three-dimensional shape measurement system. Our experiments demonstrated that the absolute coordinates calculation and rendering speed of a GPU is more than four times faster than that of a dual CPU workstation with the same graphics card. By implementing the GPU into our system, we realized simultaneous absolute coordinate acquisition, reconstruction and display at 30 frames per second with a resolution of approximately 266K points per frame. Moreover, a 2+1 phase-shifting algorithm was employed to alleviate the measurement error caused by motion. Applications of the system include medical imaging, manufacturing, entertainment, and security.

"Novel method for structured light system calibration," Opt. Eng., (2006)

[4] S Zhang and PS Huang, "Novel method for structured light system calibration," Opt. Eng., 45(8), 083601, 2006; doi:10.1117/1.2336196

System calibration, which usually involves complicated and time-consuming procedures, is crucial for any 3-D shape measurement system. In this work, a novel systematic method is proposed for accurate and quick calibration of a 3-D shape measurement system we developed based on a structured light technique. The key concept is to enable the projector to “capture” images like a camera, thus making the calibration of a projector the same as that of a camera. With this new concept, the calibration of structured light systems becomes essentially the same as the calibration of traditional stereovision systems, which is well established. The calibration method is fast, robust, and accurate. It signifi- cantly simplifies the calibration and recalibration procedures of structured light systems. This work describes the principle of the proposed method and presents some experimental results that demonstrate its performance.

"Fast three-step phase-shifting algorithm," Appl. Opt., (2006)

[3] PS Huang and S Zhang, "Fast three-step phase-shifting algorithm," Appl. Opt., 45(21), 5086-5091, 2006 (Cover Feature); doi: 10.1364/AO.45.005086

We propose a new three-step phase-shifting algorithm, which is much faster than the traditional three-step algorithm. We achieve the speed advantage by using a simple intensity ratio function to replace the arctangent function in the traditional algorithm. The phase error caused by this new algorithm is compensated for by use of a lookup table. Our experimental results show that both the new algorithm and the traditional algorithm generate similar results, but the new algorithm is 3.4 times faster. By implementing this new algorithm in a high-resolution, real-time three-dimensional shape measurement system, we were able to achieve a measurement speed of 40 frames per second at a resolution of 532
500 pixels, all with an ordinary personal computer.

"High-resolution, real-time 3-D absolute coordinate measurement based on a phase-shifting method,"Opt. Express, (2006)

[5] S Zhang and S-T Yau, "High-resolution, real-time 3-D absolute coordinate measurement based on a phase-shifting method,"Opt. Express, 14, 2644-2649, 2006 ; doi: 10.1364/OE.14.002644

We describe a high-resolution, real-time 3D absolute coordinate measurement system based on a phase-shifting method. It acquires 3D shape at 30 frames per second (fps), with 266K points per frame. A tiny marker is encoded in the projected fringe pattern, and detected by software from the texture image and the gamma map. Absolute 3D coordinates are obtained from the detected marker position and the calibrated system parameters. To demonstrate the performance of the system, we measure a hand moving over a depth distance of approximately 700 mm, and human faces with expressions. Applications of such a system include manufacturing, inspection, entertainment, security, medical imaging.

"Trapezoidal phase-shifting method for three-dimensional shape measurement," Opt. Eng., (2005)

[2] PS Huang, S Zhang, and F-P Chiang, "Trapezoidal phase-shifting method for three-dimensional shape measurement," Opt. Eng. 44(12), 123601, 2005; doi:10.1117/1.2147311

We propose a novel structured light method, namely a trapezoidal phase-shifting method, for 3-D shape measurement. This method uses three patterns coded with phase-shifted, trapezoidalshaped gray levels. The 3-D information of the object is extracted by direct calculation of an intensity ratio. Compared to traditional intensityratio-based methods, the vertical or depth resolution is six times better. Also, this new method is significantly less sensitive to the defocusing effect of the captured images, which makes large-depth 3-D shape measurement possible. If compared to sinusoidal phase-shifting methods, the resolution is similar, but the data processing speed is at least 4.5 times faster. The feasibility of this method is demonstrated in a previously developed real-time 3-D shape measurement system. The reconstructed 3-D results show similar quality to those obtained by the sinusoidal phase-shifting method. However, since the data processing speed is much faster
4.6 ms per frame, both image acquisition and 3-D reconstruction can be done in real time at a frame rate of 40 fps and a resolution of 532
500 points. This real-time capability allows us to measure dynamically changing objects, such as human faces. The potential applications of this new method include industrial inspection, reverse engineering, robotic vision, computer graphics, medical diagnosis, etc. 

"High-resolution acquisition, learning and transfer dynamic 3D facial expressions," Computer Graphics Forum, (2004)

[1] Y Wang, X Huang, C-S Lee, S Zhang, Z Li, D Samaras, D Metaxas, A Elgammal, and P Huang, "High-resolution acquisition, learning and transfer dynamic 3D facial expressions," Computer Graphics Forum, 23(3), 2004; doi: 10.1111/j.1467-8659.2004.00800.x

Synthesis and re-targeting of facial expressions is central to facial animation and often involves significant manual work in order to achieve realistic expressions, due to the difficulty of capturing high quality dynamic expression data. In this paper we address fundamental issues regarding the use of high quality dense 3-D data samples undergoing motions at video speeds, e.g. human facial expressions. In order to utilize such data for motion analysis and re-targeting, correspondences must be established between data in different frames of the same faces as well as between different faces. We present a data driven approach that consists of four parts: 1) High speed, high accuracy capture of moving faces without the use of markers, 2) Very precise tracking of facial motion using a multi-resolution deformable mesh, 3) A unified low dimensional mapping of dynamic facial motion that can separate expression style, and 4) Synthesis of novel expressions as a combination of expression styles. The accuracy and resolution of our method allows us to capture and track subtle expression details. The low dimensional representation of motion data in a unified embedding for all the subjects in the database allows for learning the most discriminating characteristics of each individual’s expressions as that person’s “expression style”. Thus new expressions can be synthesized, either as dynamic morphing between individuals, or as expression transfer from a source face to a target face, as demonstrated in a series of experiments.