Currently, vision-based traffic surveillance is widely used across the world to recognize illegal parking and obtain information about illegally parked vehicles, which causes a significant drain in the manpower, materials, and financial resources related to traffic control. With the development of computer vision technology, vision-based surveillance systems based on an image processing technique are increasingly and extensively used in the field of transporttation. However, in recent years, research in this field has been concentrated only in the areas of transportation flow, speed detection, pedestrian surveillance, and moving object recognition.

Thus far, a few studies on illegal parking have been reported, such as [1-4]. However, only the related technol- ogy framework has been constructed and there is still a lack of a systemic, concrete analysis and an understanding of the image processing algorithm for illegal parking detection and recognition. Further, the validity and practicality of the corresponding algorithm is yet to be proven. Therefore, in this paper, we propose an illegal parking surveillance algorithm based on image processing technology: a digital camera was set in the illegal parking region to acquire the video frames. An adaptive Gaussian mixture model (GMM) is used for the background subtraction in the complex environment to identify the regions of moving objects in our test video. Stationary objects are detected by using the pixel-level features in time sequences. Stationary vehicles are recognized by using the local features of an object, and thus, information about illegally parked vehicles is successfully obtained. The result of this research work is that the illegal parking time of a vehicle is obtained. Further, an automatic alarm system is used according to the different regulations of different illegal parking regions.

The rest of this paper is organized as follows: the proposed system is described in Section II. The experimental results that show the efficiency of the proposed method are presented in Section III. Finally, the conclusion is summarized in Section IV.

A flowchart of the proposed algorithm to detect and recognize illegal parking is shown in Fig. 1.

Effective detection of moving objects is the basis of the detection of illegal parking. Currently, frame difference, optical flow field, and background subtraction are mainly used for such detection. The frame difference method leads to a transparent (easy to perceive or detect) phenomenon of moving objects, which results in a misclassification of moving objects. Further, it is difficult to satisfy the requirements of real-time detection by using the optical flow field method because of the complex calculations. Background subtraction is the most frequently used method at present. The principle of this method is to show the scene background with a model before detecting a moving object first; this model is called the background model. Then, the moving object is detected through a subtraction of the background and the observed image. The algorithm proposed in this paper has successfully implemented background subtraction in complex traffic scenes by using adaptive GMM, which robustly overcomes the effects of light and the shaking of branches and can satisfy the detection condition that there exist continuous moving objects in the detection areas.

We consider N frames of a video sequence for a specific background pixel point f(x,y) to be a time sequence {X₁, X₂, …, X_t} = { f(x,y,i) : 1 ≤ i ≤ t} (this sequence cannot be occupied by a foreground object). Then, the time sequence can be represented as a stack of K Gaussian distributions, that is, a mixture of the Gaussian distributions. Fig. 2 shows the probability distribution of the gray value of a certain pixel point in a time sequence.

The probability density function of the current pixel point X_t can be represented as follows:

where K denotes the number of distributions and the value of K ranges from 3 to 5, ω_j,t represents the estimate of the j^th Gaussian weight in the mixture at time t, μ_j,t indicates the mean value, ∑_j,t represents the covariance matrix of the j^th Gaussian in the mixture at time t, and η denotes the Gaussian probability density function, which can be represented as follows:

To simplify the calculation, set K = 3 and assume that the color channels of RGB are mutually independent. Further, the covariance matrix is set as ∑_j,t = σ_j,t ²I. Here, the parameters μ_j,t and σ_j,t in the Gaussian model are obtained according to the maximum estimation algorithm [5].

In general, the background is the main part of GMM and the variance of background model is small. At the same time, the parameter of GMM changes along with the pixel of the current frame. To obtain the effective components of the background in GMM, the values of ω_j,k⁄σ_j,k are ranked in a descending order by using the method proposed in [6]. The Gaussian distribution, which is most likely to describe a stable background process, will be located at the top of the sequence and the distribution caused by a temporary turbulence will be located at the bottom of the sequence. Thus, select the first B Gaussian distributions from the above sequence as the background pixel model:

where T denotes the threshold parameter to determine the background model and is equal to 0.5. The first B Gaussian distributions after the ranking are the backgrounds.

Because of the changes in the external factors such as light and weather (rain or snow), the background changes as well. To recognize a long video surveillance, adaptive updating is needed to maintain the background. According to the method proposed in [6], we need to check whether the pixel point of each current image matches with K known Gaussian distributions in order to determine whether the model parameter should be updated or not. If the distance between the pixel value and the average value of a certain distribution is smaller than 2.5 times the standard deviation, |X_t ‒ µ_j,t| < 2.5σ_t. Therefore, we can consider that the pixel point matches the distribution successfully and the parameter is updated according to certain rules; the other unmatched distributions retain the original parameter as is. The parameters update can be expressed as follows:

where α denotes the learning rate. The prior weights of the K distributions at time t, ω_j,t are adjusted as follows:

If there is no matched Gaussian component in the mixture model, the bottom distribution obtained after the ranking of the values of ω_j/σ_j in the descending order will be replaced by a new Gaussian distribution. Therefore, the updating process of the background in GMM can be represented as follows:

In general, the traditional background updating process only considers the updating background model at the pixel level; that is, we assume that the time sequence of each pixel is independent to that of the others, thereby ignoring the special characteristics of the neighboring pixels. Therefore, this updating process is unable to overcome the problem of moving vehicles becoming a part of the background because of a long parking time; thus, this process cannot be used for the detection of illegal parking. To overcome the above problem, information such as movement tracking and object segmentation must be considered. This information can be obtained through the foreground, which can be represented as follows:

A regional template of an object can be obtained according to the structure information of a moving object segmented by an object region. This template reflects whether the current object is in the detection region or not and then, determines whether to update the background or not; the template can be represented as follows:

If the current pixel is detected as an object, the background does not update; otherwise, the background automatically updates according to the background of the pixel level, which can be represented as

The background can be obtained efficiently through the abovementioned background model, after which the detection of a foreground object can be achieved by using background subtraction.

The detection of illegal parking consists of two parts: first, the time sequences of pixel-level features are used for analyzing and detecting still objects, and then, the stopped vehicles are detected by using the local features of the object.

1) Detection of Stationary Object

Normally, the background value of each pixel is relatively stable over a long period of time and the pixel value of the point changes when the foreground object gets through, in which case the degree of change is bigger than the background value itself (affected by the environment). Therefore, if we detect that the gray value of a pixel point on a time sequence changes considerably in the illegal parking region, we can confirm that there is a foreground object. However, the change in the pixel value detected on the time sequences may be caused by a moving object that is just passing through the illegal parking region (the moving status does not change) and the object that has entered the illegal parking region and stopped. Therefore, to efficiently recognize illegal parking, the situation in which a pixel value changes considerably just because there is a moving foreground object passing through should be ruled out. This suggests that there is a moving object that passed through the detected region but did not stop when the pixel value of the background changed suddenly and returned to the initial background status value again within a short period of time. Further, the pixel point of the background is occupied by the foreground, which entered the illegal parking region and stopped; there might be illegal parking when the pixel value changes suddenly and the degree of change of the pixel values remains stable in the subsequent period of time. In addition, because of the effect caused by the surroundings and the change in the light conditions, the pixel value of the background might be turbulent on relatively long time sequences and may be unable to indicate the change of pixel value of background within a short period of time. This has already been discussed in the above background updating part, and here, we ignore this situation. Fig. 3 shows the change in the gray value of a certain pixel point under the following two situations: the moving object passes through the detected region and the moving object enters the detected region and stops.

2) Detection of Stationary Vehicles

Because of the mixed traffic environment, the still object detected in the illegal parking region might be a bicycle, a person, or some other non-motor vehicle. Therefore, it is necessary to recognize motor vehicles that are parked illegally. In this study, we perform regional segmentation of an object by using the seed filling algorithm [7], which is based on eight neighborhoods and extracts the still square of the illegal parking region. Then, the spatial property is characterized by using the duty ratio between the square of the object detected and the area of the illegal parking region. The duty ratio of the object in the region is defined as follows:

where a denotes the area of the object detected and A represents the area of the illegal parking region. Considering the duty ratio as the identification parameter, the still interference of non-motor vehicles and persons in an illegal parking region is filtered, after which the illegal parking vehicles are detected efficiently.

3) Algorithm

The steps of the proposed algorithm for detecting illegally parked vehicles are described below:

Step 1: Calculate the foreground pixel value of the current pixel point in the illegal parking detection region,

If F(x,y,t) > θ (θ denotes the degree of change allowed; here, θ = 10), we can confirm that the pixel value changes significantly and the foreground object has entered the region.

Step 2: After the abovementioned significant change, calculate the degree of change of the detected pixel value in the adjacent frame in the time sequence,

On the basis of the movement characteristics and the color characteristics of the object, it is impossible for the pixel value to change uniformly in a short period of time. Therefore, if I(x, y, t + j) ≤ ω ( ω denotes stability; here, ω = 5), record the time t + j, and if there are continuous n frames that satisfy I(x, y, t + j + m) ≤ ω, m = 0, 1, …, n − 1, we can confirm that the pixel point is stable.

Step 3: When the pixel point becomes stable, calculate the difference value between the current pixel point and the initial background point,

If P(x, y, t + j + 30) ≥ θ, the current pixel point is occupied by a still foreground object, or it is still the initial background.

Step 4: Justify the type of object according to the duty ratio of the foreground object S. if S ≥ δ (δ denotes the duty ratio classification threshold; here, δ = 0.35), we can confirm that the foreground object is a vehicle.

Step 5: To confirm that there is illegal parking and set off an automatic alarm, calculate the parking time of the vehicle as soon as it goes into a stable state according to the requirements of different parking prohibited regions.

The algorithm proposed in this paper is proven by using practical traffic scenes. The video sequences are recorded by using a digital camera with a resolution of 1024 × 576 pixels. The algorithm is developed on the MATLAB (R2013a) platform, and the CPU of the machine is Intel Core 7-2600 @ 3.40 GHz; the machine has an 8-GB memory. According to the algorithm proposed in this paper, the playing rate of a video sequence can be up to 29 frames per second; this satisfies the requirement of real-time surveillance.

Fig. 4 shows the processing results of background estimation and foreground extraction using the adaptive GMM for the background.

We assume that the dark green region in Fig. 5(b) is the illegal parking region, that is, the valid detection region of the algorithm, and the algorithm proposed in this paper is aimed at this region. When a still object is detected in the illegal parking region, the time sequence of the pixel level is used for analyzing and judging whether there is an illegal parking phenomenon or not.

By using the local features of an object, we can identify whether it is a vehicle or not and execute object segmentation to mark the illegally parked vehicle by using a boundary abuttal rectangle. Fig. 5(d)–(f) represent the detection effect of the 450^th frame, 8120^th frame, and 9105^th frame of the video sequence separately, and the 8120^th frame and 9105^th frame are two frames in the parking process. The problem that objects become a part of the background because of long time parking can be overcome efficiently by using the algorithm proposed in this paper. The 450^th frame in the figure is the time when the vehicle just enters the stable status, which can recognize the detection of an illegal parking incident and automatically calculate the parking time of the vehicle. Because of the different regulations for illegal parking in different regions (it is completely prohibited to park somewhere and is illegal to park for a long period of time elsewhere), specific time parameters for illegal parking can be set according to the illegal parking rules of different illegal parking regions in order to set an automatic alarm efficiently.

The proposed algorithm has been tested on our own recorded datasets, which contain different backgrounds and foregrounds in three time sequences. Table 1 shows the performance comparison of the proposed method with some conventional methods. Some test results with different backgrounds and fore-grounds in three time sequences are shown in Fig. 6.

Because of the necessity of vehicle control, traffic surveillance systems are improved by the use of a computer vision system. Adaptive GMM background extraction and foreground extraction have been performed for a complex traffic scene. The pixel-level features in the time sequences and the local features of the objects are used for vehicle identification and the calculation of the illegal parking time of vehicles; they are also used for detecting and identifying an illegal parking incident and providing a technical means for the intelligent traffic management of urban traffic. The proposed algorithm is very highly efficient and can be executed in real time.