ProjectionType Integral Imaging Using a Picoprojector
 Author: Yang Yucheol, Min SungWook
 Publish: Journal of the Optical Society of Korea Volume 18, Issue6, p714~719, 25 Dec 2014

ABSTRACT
A picoprojector is a compact and mobile projector that has an infinite focus. We apply the picoprojector to a projectiontype integral imaging system, which can expand the image depth to form multiple central depth planes. In a projectiontype integral imaging system, the image flipping problem arises because the expanded elemental images pass through a lens array. To solve this problem, we propose the ray tracing of a picoprojector at a central depth plane and compensate the elemental image using a pixelmapping process. Experiments to verify the proposed method are performed, and the results are presented.

KEYWORD
Projectiontype integral imaging system , Picoprojector , Image flipping

I. INTRODUCTION
It has been demonstrated that integral imaging is an autostereoscopic technique that shows a fullcolor, fullparallax, real threedimensional (3D) image [1]. Also, realtime 3D pictures can be provided with relatively low eye fatigue by an integral imaging display. However, integral imaging systems have many challenges, such as a narrow viewing angle, the spatial limitation of resolution, and the expressible depth range. It has been reported that these parameters exhibit a tradeoff relationship [2]. Since the first proposal by Lippmann, integral imaging systems have been further developed and improved by many research groups [35].
A projector can be used as the display device for the elemental images in an integral imaging system, which is called projectiontype integral imaging. Employing a diffuser screen in the elemental image plane, the projectiontype integral imaging system uses the same method as an ordinary integral imaging system [611], but, due to the diffuser, the resolution and brightness of the elemental image are restricted in projectiontype integral imaging. To overcome these weaknesses, previous studies have proposed some kinds of projectiontype systems without the diffuser and reported a method for evaluating the visibility of the reconstructed images in terms of the fill factor [12]. This system has the strength that spatial and/or temporal multiplexing is feasible. Also, multiple elemental image layers can be used within the depth of field of the projector [13, 14].
In this paper, we apply a picoprojector to projectiontype integral imaging, without a diffuser, in realdisplay mode. Due to the infinite focus of the picoprojector, the location of central depth plane (CDP) can be controlled in a projectiontype integral imaging system. Using matrix analysis, ray tracing is performed at the CDP, and the compensation of elemental images for this system is proposed. Some experiments and their results are presented to prove the feasibility of the proposed method.
II. PROJECTIONTYPE INTEGRAL IMAGING SYSTEM USING A PICOPROJECTOR
The picoprojector is a compact and mobile projector that has an infinite focus. As this technology is expected to be embedded in other devices such as smart phones and digital cameras, picoprojectors are drawing much attention. According to its operating method, a picoprojector is categorized according to three types: digital light processing (DLP), liquid crystal on silicon (LCoS), or laser beam steering (LBS). Figure 1 (a) shows the basic layout of a LBStype picoprojector [15]. As shown, there is no projection lens set. The projected beam with low numerical aperture (NA) leaves the picoprojector directly and creates an image that is always in focus. The projected image grows as its distance from the picoprojector increases, as shown in Fig. 1(b).
In an integral imaging system, the reconstructed image can be formed near the CDP. The location of the CDP depends on the gap between the lens array and the elemental image, according to the Gauss lens law. In a projectiontype integral imaging system without a diffuser, multiple elemental images can be projected within the depth of field, which generates multiple central depth planes [16]. Because the depth of field of a picoprojector is almost infinite, the region of expressible CDP can be expanded by using multiple elemental image planes. As shown in Fig. 2, elemental images with different gaps can be projected and integrated into reconstructed images with different CDPs. This means that the location of the CDP can be controlled solely by image processing, without changing the distance between projector and lens array. In this paper, we divide projectiontype integral imaging systems into two cases, to analyze and propose the processing for the compensated elemental images.
First, as in a conventional integral imaging system, the elemental image is aligned with the gap
g between the lens array and the displayed image. In this case, when the rate of image expansion is considered, the projected elemental image with gap is equivalent to the expanded image at the lens array. As shown in Fig. 3, the sizeX of the elemental image at the lens array is given bywhere
z is the distance between the picoprojector and the lens array,g is the gap between the elemental image and the lens array, andL is the total size of the lens array.Because the size of the elemental image at the lens array is bigger than the lens array, some rays passing through neighboring lenses lead to image flipping. As the gap becomes bigger and the distance
z becomes smaller, the part of the image passing through neighboring lenses grows.To solve the image flipping problem in a projectiontype integral imaging system, the size of the elemental image can be matched to that of the elemental lens at the location of the lens array, where the gap is zero in Eq. (1). In this case image flipping does not occur, but the size and location of the reconstructed image are deformed from their intended values, due to gap variation. Therefore, elemental image compensation seems necessary to render the integrated image without distortion or image flipping.
In this paper, we analyze the image transfer matrix for a projectiontype integral imaging system, and propose an elemental image compensation method to resolve the deformation of the reconstructed image, to present the intended size and depth of the integrated image without the image flipping problem.
III. MATRIX ANALYSIS AND PROPOSED SYSTEM
The compensation of the elemental image for the picoprojector is implemented using pixel mapping. Because each pixel has an angle correspondeding to the pixel emitted from the picoprojector, ray tracing at the CDP becomes effective and simple with the matrix method. The elemental image projected at the lens array passes the lens array and travels to the CDP, which is expressed by
where
d_{CDP} is the distance between the lens array and the CDP,y_{picoprojector} is the distance between a ray from the picoprojector and the center of the corresponding lens at the lens array,θ_{picoprojector} is the angle of the ray at the lens array,y_{real} is the distance between a ray from the picoprojector and the center of the corresponding lens at the CDP,θ_{real} is the angle of the ray at the CDP, andf is the focal length of the lens array.The angle
θ_{picoprojector} depends on the positiony_{picoprojector} and the distancez between the lens array and the picoprojector. Substituting Eqs. (3), Eqs. (2) may be approximated.where
n is the number of the corresponding elemental lens. In Eq. (4),y_{real} is the linear expression ofy_{picoprojector} . The magnification is related to the distancez , and the shifting distance takes on different values according to the number of the elemental lens in the lens array. In a projectiontype integral imaging system using a picoprojector, the reconstructed image at the CDP is a function of the distancez between the lens array and the picoprojector.A conventional integral imaging system is described by
where
y_{imaging} is the distance between a ray from the integral imaging system and the center of the corresponding lens at gapg , andθ_{imaging} is the angle of the ray at the gapg .y_{real} in Eq. (5) represents the pinholearray approximation; this expression is the same with inverse reaction of the pickup process, but they_{real} in Eq. (4) is different than the inverse reaction of the pickup process.The elemental image of a projectiontype integral imaging system must be compensated to represent the integrated image at the intended location with the exact intended size. This compensation can be performed to match the corresponding terms of Eq. (4) with Eq. (5), which means
y_{picoprojector} of Eq. (4) should be calculated usingy_{imaging} of Eq. (5). Figure 4 is a schematic diagram of the pixel mapping process for the proposed compensation that adjusts each elemental image according to the change of the subimage region. We implement the proposed algorithm using matrix analysis, and reproduce the elemental image with differentz andg .IV. EXPERIMENTS AND RESULTS
Figure 5 shows a schematic diagram of the experimental setup. We use the SHOWX+TM laser picoprojector with WVGA resolution (848 × 480). In the experiment, we set the distance
z between picoprojector and lens array so that the size of the elemental image is aligned at the lens array. The pixel resolution of the elemental image is dictated by the total pitch of the lens array and the distancez between picoprojector and lens array. The lens array has 13 × 13 square lenses, the size of each elemental lens being 10 mm × 10 mm. The focal length of the lens array is 15 mm.Like the system shown in Fig. 2, two elemental images with different gaps are used to confirm the expansion of the expressible depth. Also, the sizes and locations of the reconstructed images must be adjusted to be like the designed reconstructed images. As shown in Fig. 6, images 1 and 2 are intended to represent the reconstructed image. The size of each square in images 1 and 2 is the same. Two elemental images are generated from image 1 with a gap of 2.14 cm, and from image 2 with a gap of 1.8 cm. Comparing the experimental results for three cases, the reconstructed images are observed using a diffuser and measured at the distance
d_{observed} between lens array and diffuser. In the first case, as shown in Fig. 7(a), the elemental images are not compensated but simply magnified according to the gap g, the expanded elemental images being projected to the lens array. In the second case, as shown in Fig. 7(b), the elemental images are resized and matched to the size of the elemental lens at the position of the lens array. In the last case, as shown in Fig. 7(c), the elemental images are compensated using the proposed method.Figure 8 shows the elemental images on the diffuser located at the different gaps, 2.14 cm and 1.8 cm, generated for case 1 of Fig. 7. The elemental image without compensation is used to check the changes of the elemental image planes. In this experiment, two elemental image planes are implemented. Figure 8(a) shows the elemental image when the diffuser is set 2.14 cm from the lens array, while in Figure 8(b) it is located at 1.8 cm. Comparison of the period of the elemental image and the size of elemental lens show that the elemental image plane for the red image is located at gap 1, while the elemental image plane for the blue image is set at gap 2. The integrated images obtained from these elemental images prove this assumption, as shown in Fig. 9, as they are located at the intended position calculated using the lens equation.
Figure 9 shows the experimental results. The first column in Fig. 9 shows the results for case 1, while the second and third columns show the results for cases 2 and 3. The reconstructed images for images 1 and 2 are located at each
d_{observed} , as shown in the first and second rows of Fig. 9 respectively.As expected, case 1 presents the image flipping problem, and in case 2 the reconstructed images of elemental images 1 and 2 are located at 5.5 cm and 11 cm respectively, which are not the same as the designed depth. Especially, the reconstructed image of elemental image 2 in the second row shows a problem in terms of its size. Meanwhile, case 3 shows the intended size and depth, without image flipping.
V. CONCLUSION
In this paper, we apply a picoprojector to a projectiontype integral imaging system to expand the CDP. We propose an elemental image compensation method using matrix analysis and an image mapping algorithm. The results of basic experiments show that the proposed system has multiple CDPs without the image flipping problem, unlike a conventional system.
The image flipping problem occurs in a projectiontype integral imaging system due to the divergence property of the projector. In this paper, the compensation is performed assuming low numerical aperture light from the picoprojector. However, the proposed method could also be applied directly to a system adopting a conventional projector, considering the working distance of the chosen projector. So to speak, the proposed compensation method can be used to represent the intended integrated image without a flipped image for a projectiontype integral imaging system in general. Experiments are performed to verify the reconstructed image without the image flipping problem. In this paper, reconstructed images with two different depths are investigated using a diffuser. Plane images are used for convenience in the experiments, yet not only separated plane images but also a continuousvolume image can be represented using the proposed system. However, the visibility of the integrated image might be degraded, so as to be unobservable by the naked eye, because of the small aperture of the projection system. This low visibility is caused by the aperture limitation of the projection system and generally occurs in a projectiontype integral imaging system without a diffuser, as reported in previous studies [12]. The main proposal and contribution of this paper concern elemental image compensation to resolve image flipping and deformation, not visibility improvement. The reconstructed image on a diffuser located at the CDP can verify the proposed method to modify the size and location of the integrated images, without image flipping. Visibility enhancement could be achieved by adopting special viewing aids like a 3D screen, which is an optical device to resolve the aperture limitation due to the projection lens set by keeping the parallaxes of a 3D image, and can be a proper solution to improve the visibility of projectiontype integral imaging [1719]. Further studies are needed to apply a 3D screen to the proposed system effectively, which could increase the practical utility of the projectiontype integral imaging system.

[FIG. 1.] (a) Basic layout of an LBStype picoprojector. (b) Field of view of an LBStype picoprojector.

[FIG. 2.] Expansion of CDP in an integral imaging system using a picoprojector.

[FIG. 3.] Image flipping problem in a projectiontype integral imaging system.

[]

[]

[]

[]

[]

[FIG. 4.] Compensation of an elemental image using a pixelmapping process.

[FIG. 5.] Experimental setup.

[FIG. 6.] Intended images 1 and 2 for computergenerated elemental images.

[FIG. 7.] Projectiontype integral imaging systems and projected elemental images for (a) case 1 (no compensation), (b) case 2 (size adjustment), and (c) case 3 (elemental image compensation).

[FIG. 8.] Elemental image planes on a diffuser located at (a) g2 = 1.8 cm and (b) g1 = 2.14 cm.

[FIG. 9.] Experimental results for reconstructed images for the three cases in Fig. 7.