"High-resolution, real-time three-dimensional shape measurement on graphics processing unit," Opt. Eng., (2014)

[64] N. Karpinsky*, M. Hoke*, V. Chen*, and S.  Zhang, "High-resolution, real-time three-dimensional shape measurement on graphics processing unit," Opt. Eng. 53(2), 024105, 2014;  doi: 10.1117/1.OE.53.2.024105

Abstract

A three-dimensional (3-D) shape measurement system that can simultaneously achieve 3-D shape acquisition, reconstruction, and display at 30 frames per second (fps) with 480,000 measurement points per frame is presented. The entire processing pipeline was realized on a graphics processing unit (GPU) without the need of substantial central processing unit (CPU) power, making it achievable on a portable device, namely a laptop computer. Furthermore, the system is extremely inexpensive compared with similar state-of-art systems, making it possible to be accessed by the general public. Specifically, advanced GPU techniques such as multipass rendering and offscreen rendering were used in conjunction with direct memory access to achieve the aforementioned performance. The developed system, implementation details, and experimental results to verify the performance of the proposed technique are presented.

"High-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques," Journal of Visualized Experiments, (2013)

L. Ekstrand*, N. Karpinsky*, Y. Wang*, and S. Zhang, "High-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques," Journal of Visualized Experiments (JoVE), (82), e50421, 2013. (Associated with Video Illustrations) (invited); doi: 10.3791/50421

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces. Like the human eyes and brain, DFP uses triangulation between matching points in two views of the same scene at different angles to compute depth. However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras. The projector rapidly projects a known sinusoidal pattern onto the subject, and the surface of the subject distorts these patterns in the camera’s field of view. Three distorted patterns (fringe images) from the camera can be used to compute the depth using triangulation.

Unlike other 3D measurement methods, DFP techniques lead to systems that tend to be faster, lower in equipment cost, more flexible, and easier to develop. DFP systems can also achieve the same measurement resolution as the camera. For this reason, DFP and other digital structured light techniques have recently been the focus of intense research (as summarized in1-5). Taking advantage of DFP, the graphics processing unit, and optimized algorithms, we have developed a system capable of 30 Hz 3D video data acquisition, reconstruction, and display for over 300,000 measurement points per frame. Binary defocusing DFP methods can achieve even greater speeds.

Diverse applications can benefit from DFP techniques. Our collaborators have used our systems for facial function analysis9, facial animation10, cardiac mechanics studies11, and fluid surface measurements, but many other potential applications exist. This video will teach the fundamentals of DFP techniques and illustrate the design and operation of a binary defocusing DFP system.

"Three bit representation of three-dimensional range data," Appl. Opt. , (2013)

[54] N. Karpinsky*, Y. Wang*, and S. Zhang, "Three bit representation of three-dimensional range data," Appl. Opt. 52(11), 2286-2293, 2013; doi: 10.1364/AO.52.002286

Abstract

Our previous research has shown that 3D range data sizes can be substantially reduced if they are converted into regular 2D images using the Holoimage technique. Yet, this technique requires all 24 bits of a standard image to represent one 3D point, making it impossible for a regular 2D image to carry 2D texture information as well. This paper proposes an approach to represent 3D range data with 3 bits, further reducing the data size. We demonstrate that more than an 8.2∶1 compression ratio can be achieved with compression root-mean-square error of only 0.34%. Moreover, we can use another bit to represent a black-and-white 2D texture, and thus both 3D data and 2D texture images can be stored into an 8 bit grayscale image. Both simulation and experiments are presented to verify the performance of the proposed technique.

"3D range geometry video compression with the H.264 codec," Opt. Laser Eng. , (2013)

[49] N. Karpinsky* and S. Zhang, "3D range geometry video compression with the H.264 codec," Opt. Laser Eng. 51(5), 620-625, 2013; doi: 10.1016/j.optlaseng.2012.12.021

Abstract

Advances in three-dimensional (3D) scanning have enabled the real-time capture of high-resolution 3D videos. With these advances brings the challenge of streaming and storing 3D videos in a manner that can be quickly and effectively used. This research addresses this challenge by generalizing the Holovideo technique to video codecs that use the YUV color space such as the H.264 codec. With the H.264 codec, we have achieved a compression ratio of over 6086:1 (Holovideo to OBJ) with a reasonably high quality; utilizing an NVIDIA GeForce 9400 m GPU, we have realized 17 frames per second encoding, and 28 frames per second decoding speed, making it a viable solution for real-time 3D video compression.

"Holovideo: Real-time 3D range video encoding and decoding on GPU," Opt. Laser Eng. (2012)

N. Karpinsky* and S. Zhang, "Holovideo: Real-time 3D range video encoding and decoding on GPU," Opt. Laser Eng.50(2), 280-286, 2012; doi: 10.1016/j.optlaseng.2011.08.002

We present a 3D video-encoding technique called Holovideo that is capable of encoding high-resolution 3D videos into standard 2D videos, and then decoding the 2D videos back into 3D rapidly without significant loss of quality. Due to the nature of the algorithm, 2D video compression such as JPEG encoding with QuickTime Run Length Encoding (QTRLE) can be applied with little quality loss, resulting in an effective way to store 3D video at very small file sizes. We found that under a compression ratio of 134:1, Holovideo to OBJ file format, the 3D geometry quality drops at a negligible level. Several sets of 3D videos were captured using a structured light scanner, compressed using the Holovideo codec, and then uncompressed and displayed to demonstrate the effectiveness of the codec. With the use of OpenGL Shaders (GLSL), the 3D video codec can encode and decode in realtime. We demonstrated that for a video size of 512 512, the decoding speed is 28 frames per second (FPS) with a laptop computer using an embedded NVIDIA GeForce 9400 m graphics processing unit (GPU). Encoding can be done with this same setup at 18 FPS, making this technology suitable for applications such as interactive 3D video games and 3D video conferencing. 

"High-resolution, real-time 3-D imaging with fringe analysis," Real Time Image Processing, (2012)

[25] N. Karpinsky* and S. Zhang, "High-resolution, real-time 3-D imaging with fringe analysis," Real Time Image Processing,7(1) 55-66, 2012; doi: 10.1007/s11554-010-0167-4

Real-time 3D imaging is becoming increasingly important in areas such as medical science, entertainment, homeland security, and manufacturing. Numerous 3D imaging techniques have been developed, but only a few of them have the potential to achieve realtime. Of these few, fringe analysis based techniques stand out, having many advantages over the rest. This paper will explain the principles behind fringe analysis based techniques, and will provide experimental results from systems using these techniques.

"Composite phase-shifting algorithm for 3-D shape compression," Opt. Eng.. (2010)

[24] N Karpinsky* and S Zhang, "Composite phase-shifting algorithm for 3-D shape compression," Opt. Eng. 49(6), 063604, 2010; doi:10.1117/1.3456632 

With recent advancements in 3-D imaging and computational technologies, acquiring 3-D data is unprecedentedly simple. However, the use of 3-D data is still limited due to the size of 3-D data, especially 3-D video data. Therefore, the study of how to store and transmit the 3-D data in real time is vital. We address a technique that encodes a 3-D surface shape into a single 24-bit color image. In particular, this image is generated by advanced computer graphics tools with two primary color channels encoded as sine and cosine fringe images, and the third channel encoded as a stair image to unwrap the phase obtained from the two fringe images. An arbitrary 3-D shape can then be recovered from a single image. We test 3-D shapes with differing levels of complexity along with various image formats. Experiments demonstrate that, without significantly losing the shape quality, the compression ratio can go up to 1:36.86, compared with the native smallest possible 3-D data representation method.

"Flexible 3-D shape measurement using projector defocusing: extended measurement range," Opt. Lett., (2010)

[23] S. Zhang, "Flexible 3-D shape measurement using projector defocusing: extended measurement range," Opt. Lett. 35(7), 931-933, 2010 (Cover Feature); doi: 10.1364/OL.35.000934

A recently proposed flexible 3D shape measurement technique using a defocused projector [Opt. Lett. 34, 3080 (2009)] shows great potential because of its elimination of projector’s gamma calibration. However, it cannot handle step-height surfaces. I present here a technique to extend its measurement range to an arbitrary shape by integrating a binary coding method. A computational framework is also proposed to tackle the problems related to the defocusing.