Simplified Integral Imaging Pickup Method for Real Objects Using a Depth Camera

  • cc icon
  • ABSTRACT

    In this paper, we present a novel integral imaging pickup method. We extract each pixel’s actual depth data from a real object’s surface using a depth camera, then generate elemental images based on the depth map. Since the proposed method generates elemental images without a lens array, it has simplified the pickup process and overcome some disadvantages caused by a conventional optical pickup process using a lens array. As a result, we can display a three-dimensional (3D) image in integral imaging. To show the usefulness of the proposed method, an experiment is presented. Though the pickup process has been simplified in the proposed method, the experimental results reveal that it can also display a full motion parallax image the same as the image reconstructed by the conventional method. In addition, if we improve calculation speed, it will be useful in a real-time integral imaging display system.


  • KEYWORD

    Integral imaging , Pickup , Depth camera , Elemental image

  • I. INTRODUCTION

    Although many researchers have investigated 3D image display and visualization, most of the 3D display technologies are not effective or suitable for commercial applications. This is due to the fact that most of these technologies require supplementary glasses or tracking devices in order to visualize a 3D scene. Some other 3D technologies have been developed, for example stereoscopy [1], autostereoscopy [2], holography [3-4]. Amongst these techniques, integral imaging is an attractive autostereoscopic 3D display technology using a lens array [5-18]. An integral imaging system consists of a pickup and display sections. In the pickup section, an object is imaged through a lens array. Each elemental lens forms a corresponding image of the object, and the elemental images are captured by a charge-coupled device (CCD) camera. However, when we obtain elemental images from real objects in a conventional optical pickup method, there are some disadvantages in using a lens array [5-16]. The setup of a lens array and a CCD camera should be placed exactly, and it should prevent unnecessary beams to come into the lens array. Also, aberrations of the lenses should be considered, generated elemental images sometimes need post-processing, and so on. Furthermore, generating elemental images from an object as big as a person, requires a complex optical structure, because of limitations of the camera lens and lens array.

    In order to overcome these drawbacks, in this paper, we present a new pickup method to extract 3D information from a real object using a depth camera, then to generate elemental images based on the depth and color data of object pixels. The configuration of the proposed method is shown in Fig. 1. The proposed pickup method just needs a depth camera, PC, and LCD monitor. As shown in Fig. 1, we can generate elemental images from people without a lens array. Hence it has simplified the optical setup of the pickup process, and the problems in conventional pickup process caused by using a lens array have been solved. Finally, a 3D image would then be integrated through a lens array and the image can be observed as being a natural 3D image. The proposed pickup method computes the elemental image pixel coordinates, as if they focused through a lens

    array. The proposed method consists of three stages: input, calculation and display. Fig. 2 shows the architecture of elemental images generation using the proposed method. Object information is entered in the input stage. Then elemental image generation uses pixel mapping based on the object information in the calculation stage. Finally, we generate the elemental image set.

    II. INPUT STAGE

    In the input stage, the parameters of the lens array and display panel should be stored in the program. The depth data of every pixel on the 3D object surface has been extracted by the depth camera. An example of extracted depth data distribution is shown in Fig. 3. The depth data of the object are real distance from object surface points to the depth-camera. Note that the obtained object depth information should be converted. There are two reasons why we should convert it. First, when we reconstruct the 3D image depending on our proposed method, it needs to be converted from a pseudoscopic image to orthoscopic image [12,13]. As shown in Fig. 4, if we use the original depth data to reconstruct integral images directly, the recorded elemental images are back-projected through a lens array. As shown in Fig. 4(a), the observed image is reversed.

    However our ultimate goal is to display the 3D image as shown in Fig. 4(b). Therefore, the real depth data Z(i,j) must be in inverse proportion to the converted distance L(i,j). Another reason is that the depth range of an integral imaging technique has inherent limitations [17,18]. Typically, the real depth data extracted by a depth camera will exceed the expressible depth range of the integral imaging method. The definition of depth range Δd of the integral imaging method is expressed as

    image

    where PL is the pitch of the elemental lens, PI is the pixel size of the object image, and d is the central depth of the integral imaging system. The variables d and PI can be expressed by

    image
    image

    where f is the focal length of the elemental lens, g is the gap between the LCD display and the lens array, and PD is the pixel pitch of the LCD monitor. As shown in Fig. 4, in order to display all pixels clearly, the reconstructed pixels must locate in the range from d-Δd/ 2 to d+Δd/ . It means that all of the converted depth data should be located in this range. Finally, the converted distance L(i,j) is expressed as

    image

    where Z(i, j) is the real depth of (i, j)-th pixel.

    III. ELEMENTAL IMAGES CALCULATION AND GENERATION

    For the calculation stage, we need to create three buffers. The information for every object pixel is stored in the first buffer. The second buffer is reserved for the center coordinates of the elemental lenses. And the third buffer is used for storing the calculated elemental image pixel set. Fig. 5 shows the geometry of pixel mapping from object pixels to the elemental image plane through an elemental lens. The coordinates of the elemental lenses center are computed based on lens pitch and elemental lens indices. As shown in Fig. 5, after the ray of the object pixel A(i, j) passes through the center of the lens, it will be located at the positions of elemental image plane A'(u, v). The pixel coordinate (u, v) is given by

    image
    image

    where i and j are the object pixel indices in the x and y axes, respectively; iL and jL are the indices of the lenses on the x and y axes. From Eqs. (5) and (6), we can verify that every pixel position in the elemental image plane is calculated by its own depth data L(i,j).

    Figure 6 shows the color images of the object and their corresponding depth map. In order to extract the region of interest in the depth map, we separated the background from the person, as shown in Figs. 6(b) and (d). It is necessary to prevent unwanted beams to come into the lens array, in the conventional optical pickup process. While we capture object and separate the background from the object to extract the region of interest, then generate elemental images. So it eased the setup conditions, and it avoided the aberrations caused by lens array, which means we can generate an elemental image from an object almost anywhere. Fig. 7 elemental images plotted by the proposed method.

    IV. EXPERIMENT

    To demonstrate the proposed method, we have carried out an experiment. The implemented experimental setup is shown in Fig. 8. It is composed of a depth camera, a PC, and a high resolution LCD monitor. And the lens array is used in the reconstruction process to integrate elemental image generated by the proposed pickup method. Compared with the conventional optical pickup method, it has significantly simplified the pickup process.

    System parameters are listed in Table 1. In our experiment, we used a KINECT sensor as the depth camera. Recently Microsoft released a non-commercial KINECT software development kit (SDK) for Window 7. We use C# program to run the KINECT SDK, and extract each pixel’s real depth data. The depth map resolution is same with the color image, which is 320 × 240. The detectable depth range of the KINECT sensor is from 800 mm to 4000 mm. The extracted real depth range from the object which shown in Fig. 3 is from 994.9 mm to 1207.7 mm, and the real depth data of every pixel is converted by Eq. (4). The lens array used in the experiment consisted of 30(H) × 30(V) rectangular elemental lenses that are 5 mm × 5 mm with a 10 mm focal length. Furthermore the gap between the lens array and the LCD monitor is set to 11mm. In order to verify that the image generated by the proposed pickup method is also effective as a conventional optical pickup method, we captured the reconstruction image obtained by

    the proposed method in five different viewing positions. Fig. 9 shows the experimental result. As shown in Figs. 9(a), (c), and (e), the vertical distance y between marked pixels are 77.19 mm, 89.6 mm, and 82.1 mm, respectively. It can be confirmed that the proposed method successfully creates pixels disparity in the vertical direction. In the same way, the horizontal distance x between the marked pixels in Figs. 9(b), (c), and (d) is 30.8 mm, 47.8 mm, and 34.4 mm, respectively. It can be also confirmed that the proposed method successfully creates pixels disparity in the horizontal direction. From the experimental result, in different viewing positions we can observe the correct parallax image.

    V. CONCLUSION

    In conclusion, a novel pickup method that generates elemental images from real objects based on integral imaging is proposed. The proposed method extracts 3D information from the object using a depth camera. Since our proposed pickup method generates elemental images without a lens array, the optical pickup process of conventional integral imaging has been simplified. Some disadvantages caused by using a lens array have been solved. If the calculation speed improved, the proposed method can be applied to a

    real-time integral imaging display [19], because of the simplified pickup process. Furthermore, it easy to obtain elemental images even from moving objects.

  • 1. Fergason J. L., Robinson S. D., McLaughlin C. W., Brown B., Abileah A., Baker T. E., Green P. J. (2005) “An innovative beamsplitter-based stereoscopic/3D display design” [Proc. SPIE] Vol.5664 P.488-494 google
  • 2. Tao Y.-H., Wang Q.-H., Gu J., Zhao W.-X., Li D.-H. (2009) “Autostereoscopic three-dimensional projector based on two parallax barriers” [Opt. Lett.] Vol.34 P.3220-3222 google doi
  • 3. Choi K., Kim J., Lim Y., Lee B. (2005) “Full parallax viewingangle enhanced computer-generated holographic 3D display system using integral lens array” [Opt. Express] Vol.13 P.10494-10502 google doi
  • 4. Zhao J.-L., Jiang H., Di J. (2008) “Recording and reconstruction of a color holographic image by using digital lensless Fourier transform holography” [Opt. Express] Vol.16 P.2514-2519 google doi
  • 5. Lippmann G. (1908) “La photographie integrale” [C. R. Acad. Sci.] Vol.146 P.446-451 google
  • 6. Kim Y., Hong K., Lee B. (2010) “Recent researches based on integral imaging display method” [3D Res.] Vol.1 P.17-27 google
  • 7. Baasantseren G., Park J.-H., Kwon K.-C., Kim N. (2009) “View angle enhanced integral imaging display using two elemental image masks” [Opt. Express] Vol.17 P.14405-14417 google doi
  • 8. Park S.-g., Song B.-S., Min S.-W. (2010) “Analysis of image visibility in projection-type integral imaging system without diffuser” [J. Opt. Soc. Korea] Vol.14 P.121-126 google doi
  • 9. Yeom S., Woo Y.-H., Beak W.-W. (2011) “Distance extraction by means of photon-counting passive sensing combined with integral imaging” [J. Opt. Soc. Korea] Vol.15 P.357-361 google doi
  • 10. Park J.-H., Kim J., Kim Y., Lee B. (2005) “Resolution-enhanced three-dimension/ two-dimension convertible display based on integral imaging” [Opt. Express] Vol.13 P.1875-1884 google doi
  • 11. Kim Y., Kim J., Kang J.-M., Jung J.-H., Choi H., Lee B. (2007) “Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array” [Opt. Express] Vol.15 P.18253-18267 google doi
  • 12. Okano F., Hoshino H., Arai J., Yuyama I. (1997) “Real-time pickup method for a three-dimensional image based on integral photography” [Appl. Opt.] Vol.36 P.1598-1603 google doi
  • 13. Navarro H., Martinez-Cuenca R., Saavedra G., Martinez-Corral M., Javidi B. (2010) “3D integral imaging display by smart pseudoscopic-to-orthoscopic conversion (SPOC)” [Opt. Express] Vol.18 P.25573-25583 google doi
  • 14. Tolosa A., Martinez-Cuenca R., Pons A., Saavedra G., Martinez-Corral, M., Javidi B. (2010) “Optical implementation of micro-zoom arrays for parallel focusing in integral imaging” [J. Opt. Soc. Am.] Vol.27 P.495-500 google doi
  • 15. Jang J.-S., Javidi B. (2002) “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics” [Opt. Lett.] Vol.27 P.324-326 google doi
  • 16. Hahn J., Kim Y., Kim E.-H., Lee B. (2008) “Undistorted pickup method of both virtual and real objects for integral imaging” [Opt. Express] Vol.16 P.13969-13978 google doi
  • 17. Pham D.-Q., Kim N., Kwon K.-C., Jung J.-H., Hong K., Lee B., Park J.-H. (2010) “Depth enhancement of integral imaging by using polymer-dispersed liquid-crystal films and a dual-depth configuration” [Opt. Lett.] Vol.35 P.3135-3137 google doi
  • 18. Kim Y., Choi H., Kim J., Cho S.-W., Kim Y., Park G., Lee B. (2007) “Depth-enhanced integral imaging display system with electrically variable image planes using polymerdispersed liquid-crystal layers” [Appl. Opt.] Vol.46 P.3766-3773 google doi
  • 19. Kwon K.-C., Park C., Erdenebat M.-U., Jeong J.-S., Choi J.-H., Kim N., Park J.-H., Lim Y.-T., Yoo K.-H. (2012) “High speed image space parrael processing for computergenerated integral imaging system” [Opt. Express] Vol.20 P.732-740 google doi
  • [FIG. 1.] Concept of the proposed method.
    Concept of the proposed method.
  • [FIG. 2.] Architecture of elemental images generation using proposed method.
    Architecture of elemental images generation using proposed method.
  • [FIG. 3.] An example of real depth data distribution from a 3D object using the depth camera.
    An example of real depth data distribution from a 3D object using the depth camera.
  • [FIG. 4.] Geometry of integral image reconstruction: (a) using original depth data, (b) using converted depth data.
    Geometry of integral image reconstruction: (a) using original depth data, (b) using converted depth data.
  • [FIG. 5.] Geometry of pixels mapping from object image plane to elemental image plane for generating elemental image pixels.
    Geometry of pixels mapping from object image plane to elemental image plane for generating elemental image pixels.
  • [FIG. 6.] Object color images and their corresponding depth map images; (a) full color image and (c) its depth map, (b) region of interest and (d) its depth map.
    Object color images and their corresponding depth map images; (a) full color image and (c) its depth map, (b) region of interest and (d) its depth map.
  • [FIG. 7.] Elemental image generated by proposed method.
    Elemental image generated by proposed method.
  • [FIG. 8.] Experimental setup.
    Experimental setup.
  • [TABLE 1.] Specifications of experimental setup
    Specifications of experimental setup
  • [FIG. 9.] Reconstruction image using proposed pickup method and captured at five different viewing positions.
    Reconstruction image using proposed pickup method and captured at five different viewing positions.