We investigated an anti-aliasing (AA) filter for digital camera photography by which the excessively high-frequency components of the image signal are suppressed to avoid the aliasing effect. Our optomechanical AA filter was implemented by applying rapid relative motions to the imaging sensor. By the engineered motion blur of the mechanical dithers, the effective point-spread function (PSF) of the imaging system could be tailored to reject the unwanted high-frequency components of the image. For optimal operations, we developed a spiral filter motion protocol that could produce a Gaussian-like PSF. We experimentally demonstrated that our AA filter provides an improved filtering characteristic with a better compromise of the rejection performance and the signal loss. We also found that the pass band characteristic can be enhanced further by a color-differential acquisition mode. Our filter scheme provides a useful method of digital photography for low-error image measurements as well as for ordinary photographic applications where annoying moiré patterns must be suppressed efficiently.
Sampling discrete points from an original continuous signal unavoidably produces ambiguity of information. Known as the aliasing effect, this effect limits the utilizable bandwidth of the analog-to-digital signal conversion. As the analog signal is acquired discretely in the signal’s domain, the complete contents cannot be obtained in an accurate form. It is well known that only the slowly varying features can be correctly reconstructed from the converted digital signal according to the Nyquist-Shannon sampling theorem . Higher frequencies are falsely converted down through the corresponding mixing process of signal sampling. In the resultant spectrum, they are aliased and possibly collide with the original contents of the same frequency band. In order to avoid this effect, a low-pass filter is routinely placed before an analog-to-digital converter (ADC). It is usually called an anti-aliasing (AA) filter for its function. The AA filter has been regarded as an essential element of the signal conversion.
The same principle applies to digital imaging systems by which continuously distributed information of space is sampled with a matrix of discrete photon sensors known as pixels [2-4]. The acquired digital image is the same as the original only in the slowly varying features. The aliasing effect of digital imaging is more frequently observed in color imaging systems when they employ single monolithic image sensors which have periodically arranged color pixels. Because of the lowered sampling frequency, each color component has a higher chance of the aliasing effect. It becomes the most unacceptable when fine periodic patterns of an object are acquired at good optical resolutions. By superposing its image on the pixel matrix, a kind of moiré pattern is produced, which gives a very unrealistic and perceptively annoying sense to the human viewer. In the fields of digital imaging applications, the aliasing effect acts as a deterministic noise source with the incorrectly reconstructed dataset. This has serious impact when quantitative properties are to be extracted from the acquired image data through linear or nonlinear post-processing schemes. For example, accurate evaluation of the object’s color may be hindered by the aliased components when a digital color camera is utilized for inspecting a finely structured object in applications such as machine vision or biomedical diagnosis [5, 6]. One must note that the aliased false signal, once acquired, cannot be removed by the post-processing schemes. It must be rejected at the stage of the image acquisition by an optical means.
An optical AA filter is a low-pass filter that acts on the 2-dimensional image information. It is usually equipped on the image sensor package to scatter the light fields in controlled manners [2, 7, 8]. It introduces a slight blur of the image contents to remove the fast varying features before being acquired. In this principle, the AA filter can be implemented by various methods. Polarization-sensitive beam splitting is one of the most popular schemes while any light diffusing elements can be used for the same purpose. Some of the AA filter implementation methods may provide operational flexibility [7-9]. Their filter characteristics are freely adjusted by the users. The optomechanical AA filter is a recently introduced method that supports such an adjustability for the users’s preference . In this implementation of the AA filter, the relative position of pixels to the image mechanically dithers to make an engineered motion blur. The intended blur directly gives a low-pass filter effect. By the active mechanism, the optomechanical AA filter can be adjusted by the mechanical dithers and, of course, can be turned off to make no effect on the image.
In general, the performance of an AA filter is quantitatively measured by the optical transfer function (OTF) in the spatial frequency domain, or its modulus of the magnitude transfer function (MTF) [10, 11]. The idealized AA filter completely rejects all the signal components above the Nyquist frequency while preserving well the low-frequency components. Hence, the magnitude of the OTF,
The optomechanical AA filter is an attractive image-filtering scheme. It utilizes mechanical actuators that vibrate the image sensor or the objective lens in the transverse plane during the photon integration time. Owing to popularity of the optical image stabilization (OIS) technology, this type of AA filter can be implemented at minimum cost. An OIS-equipped camera has internal actuators for compensation of the camera’s motion in a shot, which can be used simultaneously for the AA filtering . By taking advantage of this convenience, a Japanese camera maker, Pentax of Ricoh Imaging Co., recently commercialized a digital single-lens reflex (DSLR) camera, K-3, with an optomechanical AA filter . A very useful feature is found in its adjust-ability. The user can freely turn the AA function off for ordinary objects that make only slight visible moirè patterns. One more advantage which has not been emphasized yet can be found in its greater freedom of filter design. In the optomechanical AA filter, the point-spread function (PSF) of the filter can be directly tailored by the actuation protocol to provide an optimal MTF.
In this research, we have investigated the optimal design of the optomechanical AA filter which can effectively remove the aliasing effect of the digital color camera. Instead of using the circular motion introduced earlier by Pentax , we developed a spiral motion protocol which can make Gaussian-like PSF and MTF characteristics. We observed that our deliberate design of the optomechanical dithers greatly improves the high-frequency suppression. We also found that the pass band characteristic can be enhanced by a color-differential acquisition which is a unique advantage of the optomechanical AA filter. We successfully demonstrated a digital imaging scheme of little aliasing effect and minimum loss of image acuity by this method.
The aliasing effect in the digital color imaging is to be overviewed in this section for complete understanding of our AA filter scheme.
In a typical signal conversion, a continuous signal is sampled by a series of signal detections given periodically at a rate of the sampling frequency,
Digital imaging is different from a simple analog-to-digital conversion for its two-dimensional (2D) and areal sampling nature. In digital imaging, the signal conversion to a discrete form is carried out through a pixilation process. Because of the 2D property, two spatial frequencies of
The most popular type of digital color imaging sensors utilizes the Bayer color filter array (CFA) to obtain three color channels with a single sensor chip [14, 15]. Figure 2 shows the typical arrangement of color pixels in the Bayer CFA. In Fig. 2, the boxes of R, G and B stand for pixels of red-, green- and blue-sensitive photodetectors, respectively. In every group of 2×2 pixels, two G-pixels are arranged diagonally while R- and B-pixels are put in the other two pixels. The pixel pitch, denoted by
Due to the color pixel arrangement, the aliasing effect is more complicated in digital color imaging . The sampling frequency differs by the color channel. The Nyquist frequency must be carefully defined for each color channel. Figure 3 shows the 2D spectral map of the Bayer CFA-based color image signals. For the red or blue (R/B) channel, the sampling interval equals to 2⋅
[FIG. 3.] 2D spectral map of the Bayer CFA-based color image signals. The base band area of the G channel is defined by the solid lines of the Nyquist rate while those of the R and B channels are bounded by the dashed lines.
The Bayer CFA-based image sensor acquires a 2D image of spatially encoded color, called
In this research, we investigated an optomechanical AA filter that provides better anti-aliasing performance. To make an optimal AA filter, we developed a filter design method that utilizes a spiral-motion protocol. The PSF of the imaging system, hence, is not fully determined by the optical elements but additionally tailored by the filter’s motion. Here, the optically defined PSF of the system is given by
where ⊗ denotes the conventional integral convolution operation, and
The filter’s response of
to be a function of
for a circular band of width Δ
which explicitly relates the amplitude modulation to the motion-induced PSF. The inverse relation was, thus, calculated by
which is the very design formula for producing a targeted PSF of
An experimental setup was constructed to study the optomechanical AA filter in a digital color imaging system. Figure 4 shows the schematic diagram of our experiment setup. A color image sensor was mounted on a 2-axis piezo-electric actuator which was driven by two electric signal generators operated synchronously. Each generated a sine signal in amplitude modulation as Eq. (2) and (3) suggested. The image sensor was mechanically dithered by the actuator on the image plane to get the desired motion blurs. A photographic objective lens was placed in front of the image sensor. The image of a test object was acquired while the image sensor was dithering. The raw image data were recorded by a computer in the form of color mosaics with no demosaicking interpolation applied. Separately, a microscopic camera was set up to monitor the in-plane dither motion of the sensor. A fiber-optic illuminator was attached on a side of the image sensor. The microscopic camera could capture the image of the fiber core under motion while the optomechanical filter was operating. This microscopic image could be interpreted as the PSF of the filter motion. In our setup, the color image sensor was a low-cost CMOS image sensor based on the color pixels of the Bayer CFA. It was equipped with 1280×1024 pixels covered by micro-lenses on the top with a pixel pitch of
For our optomechanical AA filter, the actuator was operated by two different filter motion protocols: circular and spiral motions. Its contribution to the effective PSF could be evaluated by directly measuring the PSF of the filter,
[Fig. 5.] Microscopic pictures of the circular (a) and the spiral filter motion (b), captured by the microscopic camera, along with the schematic illustration of the spiral trajectory and the consequent intensity distribution (c).
In our optomechanical AA filter, a Gaussian-like PSF is preferred for its excellent suppression of the high-frequency components. An exact Gaussian function, however, has long tails outside the central region sacrificing the time efficiency. An alternative design of the quadratic function was used in this research. It is also known as the Welch window function in the field of Fourier analysis. We designed the filter motion to produce a quadratic PSF as shown in Fig. 5(c) by using the design formula of Eq. (7). The intensity distribution of the quadratic function was specified by the width,
In our preliminary imaging tests, our system acquired some pictures of finely textured objects while its optomechanical AA filter was switched on and off. Figure 7 shows the digital color image acquired with no filter (a), one acquired with a spiral filter motion (b) and the close shots of the textures on the shirts and the tie of the object (c). In the presentation of the color images, each color image was separated into three color channels of R, G and B, displayed individually in gray scale. The R and B channels were prone to producing clear moiré patterns as seen in Fig. 7(a). Strange colored patterns irrelevant to the object’s texture could be observed in the final digital images. On the other hand, they completely disappeared as the optomechanical AA filter was operating by the optomechanically induced motion blurs. The imaging test also suggested that the optomechanical AA filter brought a considerable loss of acuity, perceivably received at the object’s boundaries when magnified.
In our experiment, the image transfer characteristics of our optomechanical AA filter were quantitatively analyzed, and the suppression of the aliasing effect was evaluated as well as the signal loss of the base band. For this purpose, a resolution target was prepared in which periodic stripe patterns of multiple spatial frequencies were arranged side by side. The responses of our color imaging system were recorded for three color channels. Through the MTF analysis, the imaging characteristic was evaluated as a function of spatial frequency. Figure 8 shows the resolution target used in this study. Only a part of the target is shown in Fig. 8. It consists of 14 rectangular blocks of different spatial frequencies. In each block, black and white stripes alternated periodically in a duty ratio of 50%. Under a uniform illumination, the target located ~2 meters apart was imaged by our imaging system as described with Fig. 4. The raw image data were sent to a computer for further analysis. The amplitude of the fundamental signal component that each block produced was obtained from the captured digital image. The signal magnitude given as a function of frequency was interpreted as the effective MTF of the system. Neglecting the other sources of blurs, the optomechanical filter motion was regarded as the main contributor that dominantly determined the MTF of the system.
As described earlier, an aliased signal component appears at a frequency different from the original in the acquired digital image’s spectrum. Due to the deterministic nature of the aliasing effect explained with Fig. 1, the aliased frequency can be easily predicted from the known original frequency. Each block of the resolution target shown in Fig. 8 made a square-wave signal along the
The extended MTF for the aliasing effect analysis was obtained by the procedure that follows. (I) A digital color image was taken by the imaging system with the resolution target. (II) The raw dataset of color mosaic directly obtained from the sensor was decomposed into three independent monochrome images of R, G and B channels. For the red and blue channels, the size of images became 640×480 pixels. For the green channel, the full-size image of 1280×1024 pixels was reconstructed by borrowing the pixel values from the nearest neighborhood in the
The effect of our optomechanical AA filter was analyzed by the extended MTF. The performance of the filter motion protocols could be quantitatively compared. At first, the circular filter motion protocol was given. Figure 9 shows the extended MTF of the red-channel image (a) and that of the green-channel image (b) with various diameters of the circular motion. The horizontal axis of the normalized spatial frequency, denoted by
[Fig. 9.] Extended MTF of the red-channel image (a) and that of the green-channel image (b) with various diameters of the circular filter motion. The acquired stripe patterns of the red channel (c) are also shown in no filter motion (upper row), and D=2.5 pixel units (lower row).
For the red and blue channels, the Nyquist frequency was found at
The results of Fig. 9 demonstrated that the aliasing effect was reduced by increasing the diameter
For the spiral filter motion, the extended MTF was measured in the same way with various widths of the PSFs. Figure 10 shows the extended MTF of the red-channel image (a) and that of the green-channel image (b) with various widths of the spiral filter motion. The driving parameters were
[Fig. 10.] Extended MTF of the red-channel image (a) and that of the green-channel image (b) with various widths of the spiral filter motion. The acquired stripe patterns of the red channel (c) are also shown in no filter motion (upper row), and W=3.5 pixel units (lower row).
The results of Fig. 10 showed that the high-frequency components were well suppressed when
Concerning the acuity of the acquired image, any type of AA filter was found to exhibit a signal loss of the base band. This issue is more serious for the green channel because its base band is two times larger by the higher pixel density. For the case of
To cope with the acuity loss of the green channel, we developed an interesting image acquisition protocol cooperating with the optomechanical filter. Note that the photon integration time is another design parameter in our optomechanical filter when driven by the spiral filter motion. By reducing the integration time of the image sensor below the filter’s drawing time of
Figure 11 shows the timing chart of the color-differential filtering and image acquisition (a), the measured MTF of the green channel (b). In our color-differential filtering scheme, the green channel had a shortened integration time and its timing matched the amplitude function of
The experimental condition of the color-differential filtering was the same with those of the simple spiral filter experiment described with Fig. 10. The only difference was made by selectively reducing the integration time of the green channel to 25 ms, 50% of the red and blue channels. The improved acuity of our color-differential filtering method was quantitatively evaluated by the MTF analysis. Figure 11(b) shows the results of the green channel’s for comparison. The case of
The digital filtering could further enhance the MTF with little concern of the aliasing errors once suppressed by the AA filter. A high-frequency boosting filter, also known as a sharpening filter, could be used to amplify the high-frequency components. Note that such a digital filter could amplify the aliasing effect, if not properly suppressed by an AA filter. In our post-processing image enhancement, a finite impulse response (FIR) filter of image sharpening was applied to the image acquired by the color-differential filtering. Figure 12 shows the partial images of the resolution target obtained from the raw data in no filter operation (a), those of the spiral filter,
[Fig. 12.] Partial images of the resolution target obtained from the raw data in no filter operation (a), those of the spiral filter, W=3.5 and K=5, in full integration (b), those of the color-differential spiral filter with a green channel of 50% exposure (c), and the images of the color-differential acquisition mode which were sharpened by the digital FIR filter (d).
Inside each image of Fig. 12, four blocks of black and white stripes are shown, named as
The reduced acuity loss of the color-differential filtering scheme was also confirmed by the MTF analysis given by Fig. 11(b). The color-differential filtering after the digital sharpening (dotted line) was compared to the AA filter response of full integration (triangular dots on a solid line). The intermediate frequency range of
In this research, we investigated the AA filtering method based on the optomechanical dithering element. By the spiral motion protocol, a very useful filter design strategy was established in this study. In principle, any physically allowed design of the AA filter can be implemented by designing the AA filter motion. We experimentally confirmed that our optomechanical AA filter can provide an excellent filtering power with a better compromise of the AA performance and the signal loss. In this study, we developed a filter design formula and the extended MTF analysis for quantitative evaluation of the AA filter characteristics. We believe that our AA filter can be useful in low-error imaging measurement technologies which require integrity of image data for further quantitative analysis. As well, it can be utilized for ordinary photographic applications such as video capturing where annoying moiré patterns must be suppressed.