Computer vision research is divided into two fields: image processing and image acquisition. Image processing extracts information contained in an image, and image acquisition obtains the image itself using an imaging system. To extract better information with image processing, better images should be acquired. In general, the following are considered to be features of a good image: (1) resolution, (2) dynamic range, and (3) field of view (FOV). Among these features, this study concerns itself with the acquisition of wide FOV images. A wide FOV imaging system is able to acquire more visual information than a conventional imaging system.

Several methods have been developed for wide FOV imaging systems so far, including rotating, ring-array, fisheye-lens, omnidirectional-mirror, and mirror-pyramid cameras. It is simple to obtain a wide FOV image by rotating a camera and stitching together the resulting series of images, but this leads to the loss of real-time video rate [1]. To acquire images in real time, multiple cameras may be deployed in a ring arrangement [2, 3]. However, it is difficult to make the optical axes of the multiple cameras coincide, which requires additional image processing to stitch together the multiple images. The fisheye-lens camera is a single-camera approach to wide FOV image acquisition [4-6]. However, because the fisheye lens has circular symmetry, utilization of the image sensor can be low, as parts of the image acquired through the fisheye lens may not be used.

The catadioptric (catoptric + dioptric) approach is an imaging method that uses a reflective mirror with a conventional camera. The catadioptric imaging method has been used for single-camera stereo image acquisition by optically dividing the image sensor plane [7, 8], or for omnidirectional image acquisition by using a bowl-shaped convex/concave mirror [9-15]. Several types of mirrors have been developed for omnidirectional imaging, such as hyperbolic, parabolic, elliptic, and conic mirrors [16]. The mirror-pyramid camera is also a catadioptric approach for wide FOV image acquisition [17, 18]. In this approach, three or more pairs of planar mirrors and cameras are used in a pyramidal configuration, to make the effective view-point of each camera coincide. Multiple-camera approaches, such as the ring-array camera or the mirror-pyramid camera, require cumbersome image stitching, and unpleasant seams in the resulting images are inevitable.

This study proposes a new catadioptric wide FOV imaging system that uses a combined hyperbolic mirror. The combined mirror consists of an upper section and a lower section; the upper section is a cylindrical hyperbolic mirror, and the lower section is half of a hyperbolic omnidirectional mirror. The upper section of the mirror is used to obtain a panoramic image with a maximum 213.8° horizontal FOV, and the lower section is to secure a downward FOV of -106.9° using a conventional camera. The upper and lower sections of the combined mirror share a common focal point, with surface continuity. As a consequence, in the image there is no unpleasant seam between the two sections, and it is easy to restore the original image to a natural-looking one.

This paper is organized as follows: In Section II, the designs of the combined mirror and the imaging system are presented. In Sections III and IV, the geometrical optics of the image-acquisition model are described, and an image-restoration algorithm based on the imaging model is addressed, respectively. Experiments to verify the performance of the imaging system and concluding remarks are presented in Sections V and VI.

The hyperbolic function in the x − y plane is described as

where a and b are parameters of the function. The hyperbolic function has a pair of symmetrical focal points, and . A light ray from an object toward F changes direction toward F′ after reflection from the hyperbolic surface. Thus, it is possible to acquire a reflected image through the pinhole of a conventional camera placed at F′. As shown in Fig. 1, the FOV of an imaging system with a hyperbolic mirror is greater than 180°, depending on the design of the mirror.

Figure 2(a) shows the design of the combined hyperbolic mirror proposed in this study; the cylindrical hyperbolic mirror is in the upper section, and the half of the omnidirectional hyperbolic mirror is in the lower section. The upper and lower sections of the combined mirror share a common focal point, with surface continuity. Therefore, using this mirror it is possible to acquire a seamless natural panoramic image with a conventional camera. Figure 2(b) demonstrates image acquisition using the mirror.

The imaging model is a mathematical relationship between an object point O = [x_o y_o z_o] in three-dimensional space, and an image point P = [x_p y_p] on the image plane. An object point in the celestial sphere can be represented by longitude and latitude angles as O = [ψ φ] , where and with respect to F. Hereafter, for simplicity the longitude and latitude angles are described with respect to F instead of the origin of the coordinate system, without loss of generality. Additionally, the imaging camera at F′ is modeled as an ideal pinhole, without lens distortion.

It is noted that a cylindrical hyperbolic mirror is the same as a planar mirror, in the vertical direction. Thus the image acquired through the upper section of the mirror is a wide rectilinear panorama. It is convenient to describe the imaging model for the upper section of the mirror separately in the horizontal plane and in the vertical direction.

3.1.1. Horizontal Plane

The relationship between [x_o y_o ] and x_p in the x − y horizontal plane is illustrated in Fig. 3. Light rays I and II are represented as follows:

where C = [x_c y_c] is the reflection point on the mirror’s surface. Because C is an intersection between line I and the hyperbolic curve, it can be obtained from (1) and (2a) as follows:

By inserting (3) into (2b), it is possible to obtain the image point x_p as

where λ denotes the focal length of the camera. The equations of the imaging model in (3) and (4) can be represented briefly as

where a , b , and λ are design parameters of the imaging system.

3.1.2. Vertical Direction

Because a cylindrical mirror is regarded as a planar mirror in the vertical direction, the imaging model in this direction can be described easily by using the effective viewpoint, as depicted in Fig. 4. The effective viewpoint is the symmetrical point of the pinhole at F′ with respect to the tangential plane at the reflection point C. As shown in Fig. 4, the object point, the reflection point, the effective viewpoint, and the image point on the effective image plane are placed on a common vertical plane. The common plane is redrawn in Fig. 5, where the imaging point Z_p is described as a function of the latitude angle φ of the object point as follows:

In (6), t_c, t_p, and t_i are given by , , and respectively. The values of x_c, y_c, and x_p are obtained from (3), (4), and (5) as stated above.

In Fig. 2(a), the lower section of the mirror is represented as follows:

Figure 6 shows the ray tracing in the vertical cross sectional y − z plane of the lower section of the mirror. Because the lower section has circular symmetry about the y axis, the ray tracing in Fig. 7 is applicable to all radial directions about this axis.

In the y − z plane, the hyperbolic mirror is represented by Eq. (8).

Using a similar approach to those in (2)-(4), it is possible to obtain the relationship between the object point and the image point as follows:

The relationship between an object point O = [ψ φ] and an image point P = [x_p z_p] is depicted in Fig. 7.

Equations (5), (6), and (9) comprise the imaging model between an object point and an image point. The computer simulation of this imaging model is shown in Fig. 8. A rectangular parallelepiped box without a frontal surface is placed in front of the imaging system, as in Fig. 8(a). The size of the box and its distance from the imaging system are represented in the figure. This simulation uses the parameters of the imaging system in Table 1. Figure 8(b) is the image of the box resulting from the simulation. In Figure 8(b), the image above z_p = 0 is from the upper section of the mirror and the image below z_p = 0 is from the lower section of the mirror. As shown, the image from the upper section of the mirror is vertically rectilinear.

For each image point P = [x_p z_p], it is possible to obtain [ψ φ] for the corresponding object point, based on the image-acquisition model in Section III. The well-known Mercator projection is a conformal map that rearranges an object point on the surface of the earth to a map point on a rectangular surface according to [ψ φ] while preserving linear scale [19]. Figure 9 shows the image restoration algorithm in this study using the Mercator projection.

According to the object point [ψ φ] from an original image point, the restored image point, W = [x_w y_w], is represented as follows [19]:

where k is a scale factor and ψ₀ denotes the center of the longitudes; that is, the center of the vertical lines of the original image acquired. In the image-restoration algorithm, object points at the same longitude are placed on the same vertical line in the restored image, and object points at the same latitude are placed on the same horizontal line in the restored image.

The combined hyperbolic mirror in this study is developed as shown in Fig. 10. The parameters of the imaging system are listed in Table 1. The image sensor has a resolution of 640× 480.

Figure 11 shows the FOV of the imaging system. In the figure, α denotes the required FOV of the camera lens, β represents half of the maximum FOV, and γ is the vertical half FOV from the planar mirror section. In accordance with the parameters in Table 1, it is easy to obtain the required view angle of camera lens as 2α = 40.08°. The corresponding maximum horizontal FOV of the imaging system is 2β = 2×106.9° = 213.8°. In Fig. 11(b), the vertical FOV of the imaging system is γ + β where γ is the same as α. Thus, the total vertical FOV is given as 126.94°.

The experimental setup for the imaging system in this study is shown in Fig. 12(a). The checkerboard at the bottom of Fig. 12(a) appears in the original image taken by the imaging system, as shown in Fig. 12(b); the vertical FOV of the imaging system covers over -106.8° - +20.04°. Because the upper part of the mirror is the same as a planar mirror in the vertical direction, the upward FOV of the imaging system is the same as that of the camera lens. The upper part of the original image in Fig. 12(b) is vertically rectilinear, and the lower part is warped. However, in the restored image in Fig. 12(c), the warped objects are corrected to appear natural.

The imaging system is applicable to safe driving and parking of an automobile. Figure 13 shows the result of this application. The imaging system is attached near the side mirror of an automobile, to observe the blind spot. Figures 13(a) and 13(b) show an original image and the corresponding restored image. The left and right sides of the original image relate to the front and the rear sides of the automobile, respectively; the horizontal FOV of the imaging system is over and it is possible to observe the automobile’s blind spot.

A wide FOV image contains more visual information than a conventional image and has many industrial applications, such as surveillance, mobile robots, and automobile safety. In this study, a catadioptric wide FOV imaging system with a combined hyperbolic mirror is proposed. The upper section of the mirror is a cylindrical hyperbolic mirror, while the lower section is half of an omnidirectional hyperbolic mirror. Because the hyperbolic curves of the two sections have a common focal point, the acquired image is seamless and smooth. The maximum horizontal FOV of the image is -106.9° - 106.9°, and the maximum vertical FOV is -106.9° - 20.04°. The image-acquisition model for the proposed imaging system is described using ray-tracing optics. An image-restoration algorithm is also proposed, which uses the Mercator projection to make the original image appear natural to humans. It is expected that the proposed imaging system and image-restoration algorithm will be especially useful for the autonomous navigation of mobile robots, or for automobile safety.

The proposed imaging system experiences the problem of self-occlusion by its camera, which should be overcome for practical purpose. It is expected that this problem can be solved or minimized by an optical method using a beam splitter or slanted hyperbolic cylinder mirror; these solutions are currently under study.