The use of micro- and nanotechnologies in electronic components such as semiconductors, sensors, and various industrial products such as chemical, textile, pharmaceutical, and robotics, has been continuously increasing. Accordingly, there is a continuing demand for measuring the surface profiles of 3D micro- and nanostructures. As the size of 3D structures decreases to micro- and nanoscale, 3D shapes should be precisely measured at this level, which results in longer measuring times. Moreover, in order to improve product quality and reduce defect rate, it becomes increasingly important to quantify the surface profiles in real time by installing inspection equipment directly on the production line [1]. Therefore, the techniques for assessing the 3D surface profiles of micro- and nanostructures should have not only reasonable precision with high accuracy, but also real-time measurement capability. Furthermore, these techniques should be cost-effective, highly reliable, and preferably non-contact, so that the sample is measured without contamination or deformation [2].

Representative techniques for measuring the surface profile of a 3D micro- or nanostructure have been proposed for a long time. An electron microscope [3] has higher resolving power (in the order of a nanometer) than a typical optical microscope, but it operates in vacuum and requires sample preprocessing such as gold coating. Moreover, transmission electron microscopy (TEM) [4] is quite expensive, and scanning electron microscopy (SEM) [5] requires significant image-acquisition time. Atomic force microscopy (AFM) [6] has excellent vertical and horizontal accuracy with subnanometer resolution, but has a relatively small field of view and generates a surface profile very slowly due to its scanning nature. Laser scanning confocal microscopy (LSCM) [7], another point-scanning technology, has a comparable axial optical resolution, but its point-scanning feature results in relatively slow 3D surface profile construction, as in the case with AFM. White-light interferometry (WLI) [7], one of the most widely used wide-field surface profilers, has a very precise axial resolution (in the order of a nanometer), whereas its lateral resolution is the same as for a conventional optical microscope. However, it is difficult to measure very steep surfaces with WLI because the interference fringes disappear [7]. Also, as it utilizes the minute interference phenomenon of the light source and is very sensitive to changes in the external environment, the design and implementation of its optical system is very difficult. Fringe-pattern projection [8] is capable of measuring a 3D surface profile rapidly with the patterned light deformed on the sample surface, with axial resolution about twice that of LSCM. Structured illumination microscopy (SIM) [9], another wide-field imaging-based technology, has the advantage of a simple configuration to insert a grating pattern into the optical illumination path in wide-field microscopy (WFM) while maintaining comparable axial resolution to LSCM, which is highly competitive in terms of imaging speed and cost. Nevertheless, there is a hurdle to acquire images in real time, since at least three sequential images with a spatial phase shift of the grating pattern are required to generate the 3D surface profile. Therefore, none of these 3D profile measurement techniques satisfies all the functional requirements including cost-effectiveness, high precision, and real-time measurement, and there remains a demand for a new, outstanding 3D profile measurement method to overcome these limitations.

HiLo optical microscopy is a relatively new method similar to SIM with an optical resolution comparable to LSCM. It was first introduced in the field of fluorescence imaging in 2008 [10], and there have been various studies such as in vivo imaging using speckle illumination [11], endoscopic-image enhancement with structured illumination [12, 13], imaging-speed improvement by using two types of fluorescent dye to acquire two images simultaneously [14], and a light-sheet-illumination-based 3D HiLo technique for removing residual background noise [15]. This technique utilizes a grating similar to SIM’s, but it is faster, simpler, and more economical than SIM, in that it only requires two images while keeping its lateral resolution the same as SIM. This technique has a potential for measuring a 3D surface profile without an expensive scanning optical device, as the LSCM used in fluorescence imaging was modified to reflected-light LSCM widely utilized in 3D surface profile inspection. Moreover, up till now, there has been no report on 3D surface profile measurement with the HiLo optical imaging technique.

In this paper, a novel technique for measuring the 3D surface profiles of micro- and nanostructures with HiLo optical imaging is proposed. First, the mathematical background for the HiLo optical imaging technique is introduced, and the effect of the grating’s spatial frequency on axial resolution is induced. Next, the optical system for measuring a 3D surface profile with the HiLo optical imaging technique is designed, and the algorithm for detecting surface positions is optimized for HiLo images. In addition, the optical system is implemented with appropriate design parameters, and its optical resolution in both lateral and axial directions is experimentally assessed. Finally, it is demonstrated that this technique is comparable to LSCM, one of the current state-of-art techniques, by evaluating its optical performance from the point of view of measuring a 3D surface profile for several microscale 3D structures.

In general, an optical image is the result of combining the in-focus signal (in-focus image, I_in(x)) and the out-of-focus signal (out-of-focus image, I_out(x)) of an object. However, the out-of-focus signal causes the image to blur. If just the in-focus signal could be obtained, by removing the out-of-focus signal, then the axial optical resolution could be improved. While LSCM uses the pinhole and SIM uses the grating used to eliminate the out-of-focus signal, the HiLo optical imaging technique uses a grating similar to that in SIM. However, the HiLo approach has an advantage over SIM, in that only two images, i.e. the structured and uniform illumination images, are required. The structured illumination image I_s(x) is obtained by illuminating with structured light, generated by placing a grating in the optical path of incident light. On the other hand, the uniform illumination image I_u(x) is photographed with uniform illumination [16].

The structured illumination image and uniform illumination image can be mathematically expressed as below:

where Iin(x) is the in-focus signal, Iout(x) is the out-of-focus signal, κg is the grating’s spatial frequency, M is the grating modulation depth, and the sine function is the mathematical model for the illuminated grating pattern.

It is noted that the structured illumination does not affect the out-of-focus images, since the structured patterns are formed only in the focal plane. In other words, out-of-focus images are not expressed as the mathematical term with a product of sine function since the grating image is not projected on the out-of-focus plane [17]. D(x) is defined as the absolute value of the difference between I_u(x) and I_s(x) , and D(x) can be represented as below by expanding the function with the Fourier series:

The in-focus image with low-spatial-frequency components I_low(x) is obtained by applying a low-pass filter with cutoff frequency κ smaller than to D(x). To get the in-focus image with high-spatial-frequency components I_Hi(x) , the rest of the depth-resolved image, a high-pass filter with same cutoff frequency as the low-pass filter is applied to the uniform image I_u(x). Finally, the in-focus image I_HiLo(x) is generated by combining the Hi image and Lo image with the weighting factor so that the intensity profile is matched between these Hi and Lo images [16]. The above HiLo optical imaging technique is graphically explained in Fig. 1.

[FIG. 1.] Diagram of the HiLo optical imaging technique in the Fourier domain: (a) absolute difference value of uniform image and structured image, (b) obtaining the HiLo image by applying a low pass filter (LPF) to D and a high pass filter (HPF) to the uniform image.

Optical sectioning through structured light with a grating is a widely used technique in the field of fluorescence imaging. The axial resolution is known to be determined by the period p = 2π /κ_g of sinusoidal illumination generated by the grating, the numerical aperture (NA) of the objective lens, and the illumination wavelength. Stokseth [18] experimentally measured and showed the axial optical transfer function (OTF) model with respect to grid frequency v and normalized axial movement u. Moreover, Neil et al. [17] revised the OTF model for the light reflected from a mirror as below:

where , , f (ṽ) =1− 0.69ṽ + 0.0076ṽ2 + 0.043ṽ3, J1 is the Bessel function of the first kind, λ is the wavelength of the illuminated light, α is half of the angular aperture of the objective lens, NA is the numerical aperture of the objective lens, and p is the illuminated grid period.

The axial resolution, i.e. the minimum distance δ for distinguishing the intensity difference between axial positions, can be derived from Eq. (3), and is at the minimum (δ_min ) when the grating period is twice the cutoff frequency, by the Abbe limit, as shown in Eq. (4) [19].

where n is the refractive index of the optical medium.

However, as the grating’s spatial frequency decreases, the noise increases due to the harmonic effect caused by the third and fifth components those are not considered in the Bessel function, which results in lowering of the signal-to-noise ratio (SNR). For this reason, it is necessary to select an appropriate grid period experimentally in order to minimize noise [20].

The illustration of the optical configuration for acquiring the structured and uniform illumination images is presented in Fig. 2. A mercury arc lamp, a white-light source generally used in fluorescence imaging, is selected as the light source. This lamp has a peak wavelength of 436 nm, and a 440 nm band-pass filter (FB440-10, Thorlabs, Inc., U.S.A.) is placed after the lamp to avoid chromatic aberration. The structured illumination is constructed by locating the customized grating on the image-conjugate plane so that the image of the grating is formed on the focal plane of the objective lens, whereas the uniform illumination is produced without the grating. The light illuminates the specimen through a tube lens (f = 200 mm, LA1708, Thorlabs, Inc., U.S.A.) and an objective lens (50×, NA = 0.95, MPLANAPON50X, Olympus, Inc., Japan). Between the objective and tube lenses, a thin-plate beam splitter (50:50, CCM1-BS013, Thorlabs, Inc., U.S.A.) that separates the incident light from the reflected light is used to minimize the ghost image. The reflected light from the opaque sample surface forms an image through an imaging lens (f = 150 mm, LA1433, Thorlabs, Inc., U.S.A.) and is captured by a high-speed CCD camera (1388 × 1038, 30 fps, Manta G-145-30ps, Allied Vision Technology GmbH, Germany). Before the CCD camera, the same band-pass filter used in the illumination optical path is applied again to eliminate the ambient light. A piezo positioner with a closed-loop controller (0~200 μm focusing range, MIPOS 500 & NV 40/1 CLE, piezosystem jena GmbH, Germany), which is directly installed on the objective lens, scans axially over the sample surface to construct its 3D image stacks.

[FIG. 2.] Experimental setup for the HiLo optical imaging technique. Structured illumination image and uniform illumination image are taken sequentially by moving the grating filter.

The selection of the grating’s spatial frequency is important for optimizing the optical configuration for imaging, and four gratings with different spatial frequencies of 20, 25, 40, and 50 lp/mm are fabricated on the glass plate with a chrome mask pattern. Among them, the one with 50 lp/mm is chosen as optimal, since its line spacing in the image of the grating is experimentally 0.50 μm, which is close to the theoretical value (λ / NA = 0.46 μm) in Eq. (4); see Table 1.

[TABLE 1.] Grating pattern and illuminated grid period on a sample surface

Grating pattern and illuminated grid period on a sample surface

The sequence of two-image acquisition and HiLo image generation is executed by customized C# codes with visual studio (visual studio community 2015, Microsoft Corporation, U.S.A.) and ImageJ [21] with HiLo plug-in 1.2 [22] respectively. At each focal plane, the structured and uniform images are obtained with the two different illuminations by inserting and removing the grid sequentially, and the HiLo image at given focal plane is processed using the two images. The 3D HiLo image stack is generated by accumulating depth-resolved HiLo images at different focal planes.

In order to detect surface position in 3D, image processing with the 3D HiLo image stack should be performed using MATLAB (MATLAB R2018a, Mathworks, Inc., U.S.A). The core algorithm of the image processing is to estimate the surface height at a given lateral location by analyzing the intensity profile in the z direction at that lateral position in the 3D HiLo image stack.

This process is similar to focus-variation microscopy [7] or shape from focus [23], in that it finds a focal plane on the image stack acquired while moving the focus using a piezo positioner. However, as shown in Fig. 5, it is difficult to find the focal plane accurately because the intensity change in the acquired image is small despite the focus position shift, and the measurement error is also relatively large. On the other hand, the image stack with the background removed by applying the HiLo optical imaging technique shows a steep increase in intensity at the focal plane so that a more accurate focal plane can be found compared to focus-variation microscopy.

The way to determine the surface position is to search for the brightest point along the z direction in the image stack [7]. However, it is difficult to measure the exact height by the method of finding the brightest point due to the noise generated by the diffuse reflection occurring on the surface of the sample. Therefore, the mean gray value M(x, y) of a 3 × 3 matrix near the lateral position (x, y) is used instead to find the axial position z value of the focal plane. If the mean gray value M(x, y) at given position is smaller than the threshold, which can be calculated as the mean gray value of a HiLo image at a totally out-of-focus position, then that point is excluded from the calculation, thereby reducing the influence of noise and improving the operation speed.

where z_focal(x, y) is the surface position at given lateral position (x, y).

In this process, the height can be measured by finding the brightest point along the axial direction at each point on the image stack. However, the brightest point corresponds to one of the discrete positions of the piezo positioner, and therefore it is not the actual focal-plane position. To estimate the focal plane more precisely, it is more reasonable to regard the point corresponding to the peak value of the curve as the focal plane by curve fitting around the brightest point (Fig. 3) by using Gaussian interpolation [23] or a second-order polynomial [24]. This method has the disadvantage of longer computation time, compared to simple brightest-point searching, but it has the advantage of yielding more reliable results. Further research is expected to reduce computation time and to enhance accuracy for surface profile measurements by applying multiple noise-suppression methods [25].

[FIG. 3.] Illustration of curve fitting to determine the focal plane.

To evaluate the performance of the HiLo optical imaging technique for surface profile measurement, the lateral resolution is measured using the USAF-1951 positive target group 7 element 6 (3" by 3", #36-275, Edmund Optics, Inc., U.S.A.). To compare its performance to that of LSCM, the same sample is measured with a commercially available instrument (OLS4100, Olympus, Inc., Japan), using the same objective lens (50×, NA = 0.95, MPLAPON50X, Olympus, Inc., Japan) in both HiLo and LSCM, for a fair comparison.

As shown in Fig. 4, the lateral resolution measured by WFM, HiLo and LSCM at the edge is 0.77 μm, 0.38 μm, and 0.25 μm, respectively. They are all larger than the theoretical diffraction limit (0.28 μm for both WFM and HiLo, and 0.17 μm for LSCM), but it is confirmed that the lateral resolution in HiLo is improved by 50% compared to that of WFM. The lateral resolution in LSCM is about 52% smaller than that of HiLo, but it can be seen that it has a comparable lateral resolution, considering that the slopes of the intensity profile in the boundary are similar to each other.

[FIG. 4.] Lateral resolution comparison of (a) WFM, (b) HiLo by grid period, and (c) LSCM. FWHM for WFM, HiLo, and LSCM is 0.77 μm, 0.38 μm, and 0.25 μm, respectively. Target: USAF-1951 Positive, group 7 elements 6; objective lens: 50×, NA = 0.95; wavelength of light: WFM & HiLo 440 nm, LSCM 405 nm.

The lateral resolutions are almost the same at the edge, regardless of different grating periods in HiLo. Rather, the smaller the grating period, the greater the noise around the edge. Therefore, when selecting the grating period, the grid period should be set to be larger than the minimum period indicated by Eq. (4).

To assess the axial resolution, the full width at half maximum (FWHM) of the intensity profile in the z direction is measured and compared to that of the Stokseth OTF model (Eq. (3)). Figure 5 shows the axial intensity profile measurement for HiLo and general WFM, using a protected silver mirror (Ø = 1", PF1-03-P01, Thorlabs, Inc., U.S.A.). The axial intensity profile in HiLo is similar to that of the Stokseth OTF model. Moreover, since the axial intensity profile has a peak value at the mirror surface plane, the mirror surface position can be obtained easily. On the other hand, in the case of general WFM, it is difficult to identify a particular point as a surface point, although the intensity tends to decrease as the position of the focal plane becomes away from the surface.

[FIG. 5.] Axial intensity profile comparison of HiLo optical imaging technique (black circles and line) and general WFI (yellow triangles and line) using a protected silver mirror. The Stokseth OTF model (dashed line) is also presented as the theoretical value.

The grating period is an essential parameter for determining axial resolution, as shown in Eq. (4). As seen in Fig. 6, the FWHM of the axial intensity profile decreases as the grating period became smaller when measuring the axial intensity profile for a protected silver mirror (Ø = 1", PF1-03-P01, Thorlabs, Inc., U.S.A.) with different grating periods. In terms of axial resolution, the smaller the grating period, the higher the axial resolution. However, as shown in Fig. 6, the ratio of main lobe to side lobes, which can be considered as the SNR, increases as the grating period increases. So, choosing the optimal grating period requires a tradeoff between the axial resolution and the noise due to side lobes. In this study, the optimal grating period is set to be 25 μm at which the outlier around the edge is the smallest.

[FIG. 6.] Variation of axial intensity profile with grid period (left), and measured FWHM for each grid period (right).

The surface profile in the protected silver mirror is measured to analyze the precision of the axial surface position. The image stack is obtained by moving the focal plane along the z direction with steps of 0.1 μm, and the 3D profile is calculated from Eq. (5). It is observed that the standard deviation of the 3D measurement is 69 nm. For a more accurate result, as described in Section II and shown in Fig. 3, the surface position is recalculated by applying a second-order polynomial curve fitting near the brightest point. It gives a smaller value for the standard deviation of 17 nm. Therefore, it is confirmed that the height-profile precision of the HiLo optical imaging technique is about 20 nm (Fig. 7), which is about 1.5 times the standard deviation of 13 nm when measured with commercial LSCM.

[FIG. 7.] 3D surface profile and height profile of protected silver mirror, by applying the HiLo optical imaging technique and searching for the brightest point along the z axis. Measured standard deviation of the height profile is 69 nm without curve fitting, and 17 nm with curve fitting.

The performance of the HiLo optical imaging technique for measuring a 3D surface profile is evaluated by using actual sample (Fig. 8(a)) and is compared to the result from a commercially available LSCM (Olympus OLS 4100, Olympus, Inc., Japan) with the same objective lens (50×, NA = 0.95, MPLAPON50X, Olympus, Inc., Japan) (Fig. 8(b)). The first selected sample is a MEMS rectangular well made by lithography. The designed rectangular well size on the top surface is 250 μm by 85 μm by 0.75 μm. The 3D surface profile acquired by the HiLo optical imaging technique is represented in Fig. 8(c), and is compared to the result from LSCM (Fig. 8(d)). It is noted that the measured shape and 3D surface profile is almost the same as the designed shape and the measured result from LSCM. However, their measured heights are smaller than the design height, which seems to be due to a combination of manufacturing error and measurement error.

[FIG. 8.] Results of 3D surface profile measurement and height profile of selected sample (a) by (b) LSCM and (c) HiLo; (d) height profiles from both techniques.

Another sample, the tip of a stationary knife, is selected to determine the ability to measure a rough and inclined surface (Fig. 9(a)). 3D surface profile measurements (Fig. 9(d)) confirms that samples with rough surfaces exhibit adequate 3D reproducibility. In addition, the slope angles of 6°, 16.3°, and 33.1° can be identified by measuring the height profile.

[FIG. 9.] Results for the tip of stationary knife: (a) surface, (b) WFM image, (c) HiLo image, (d) 3D surface profile measurement, (e) height profile presenting three different angles.

In order to develop a high-speed, high-precision, simple, and cost-effective method for measuring a 3D surface profile, the HiLo optical imaging technique originally developed in the field of fluorescence imaging is altered and implemented to optimize for measuring 3D surface profile. Although it cannot match the LSCM when applied to 3D surface profile measurement, a high resolution of up to 20 nm in the proposed 3D surface measurement technique can be accomplished. As a result, high-speed 3D surface profile measurement in the order of submicron level can be achieved with the HiLo optical imaging technique by merely inserting an additional simple optical component into a conventional microscope without expensive light source such as a laser. While the feasibility of measuring the 3D surface profile with the HiLo optical imaging technique is demonstrated, there are additional issues that need to be improved through further studies.

First, the physical movement of the grating filter makes it difficult to enable real-time imaging. The simplest way to eliminate the physical movement of the grating filter is to use a spatial light modulator (SLM) or digital micromirror device (DMD) [26]. The grating filter movement time (about 0.5~1 s) can be shortened to several milliseconds by applying an SLM or DMD. However, the imaging speed is limited to less than half of the camera’s imaging speed since the method of acquiring a uniform image and a structured image sequentially remains unchanged. Both the SLM and the DMD show higher noise than a separately manufactured grating filter, due to light leakage between the pixel apertures and irregularities in the grating when the grating direction is not formed horizontally with the pixels. Also, this is not cost-effective, considering the price of SLM and DMD devices. Another method of high-speed imaging is acquiring a uniform image and a structured image at the same time by changing the image-acquisition method itself in parallel. By using two light sources with different wavelengths, uniform light and structured light illuminate simultaneously on the specimen, and the reflected light is separated by a band-pass filter to simultaneously image in two cameras. This method needs a more complicated optical system and requires correction for the wavelength difference of the two light sources, but it can be a fundamentally better alternative as it can acquire a real-time image using a general-purpose camera without moving the grating filter.

Second, there should be a way to reduce overshooting and undershooting at the edge, commonly called the bat-wing effect. As shown in Fig. 8(c), it can be seen that overshooting data is generated at the edge. A number of previous studies have suggested ways to improve edge-detection performance, and it is necessary to apply these methods in further study [27, 28].

Third, a large amount of numerical computation is required for 3D image processing, and the 3D reconstruction algorithm needs further improvement. With further development of computing devices such as CPUs and GPUs, computation speed is expected to be enhanced dramatically.