Due to the continuous decrease in prices and the improvement in lighting efficiency, light-emitting diodes (LEDs) are gradually replacing traditional light sources. Recently, LED lighting has become available over a wide range of electronic devices and appliances. Widespread use of LED lighting has great potential in wireless communications because it enables interference-free, secure, low-cost, and energy-efficient data communications. Optical camera communications (OCC) [1-6] is a form of wireless communication using LED transmitters and camera receivers, and has recently gained attention for various applications.

The majority of late-model smartphones are equipped with complementary metal-oxide semiconductor (CMOS) cameras providing the ability to capture photos and videos. During the receiving process in a camera, the captured videos can provide both visual information and transmitted information from LEDs by distinguishing light flicker patterns. In rolling shutter-based CMOS sensors, different rows of pixels are exposed in rapid succession, thereby sampling the incident light at different time instants. This converts the temporally modulated light coming from the display into a spatial flicker pattern in the captured image. The line-by-line capture of CMOS sensors can cause a rolling shutter effect, which provides a spatial flicker pattern in the captured image and can be used for decoding the modulated illumination pattern. Using the rolling shutter effect, smartphone cameras have the potential to enable a great number of OCC applications.

The flicker component is variant to the LED dimming level and other common imaging degradations resulting from light smear, motion blur, and focus blur. The effect of all of these degradations can be observed in the flicker pattern component, and this affects signal recovery. Although the rolling shutter effect offers a lot of benefits, it requires compensation for irregular flicker patterns from LEDs with the help of a robust decoding process. Due to the above problems, a well-developed decoding method for a rolling shutter camera is important, in order to ensure good communication performance, and it should be verified before use in real-world applications.

A series of recent papers reported that it is possible to detect signals using rolling shutter cameras. Line-of-sight (LOS) light-to-camera communications using frequency shift keying (FSK) was introduced [7, 8]. A hybrid modulation scheme including M-ary FSK and 2-phase-shift keying (2-PSK) was proposed [9]. Compatible on-off keying (C-OOK) [10] considers a pair of asynchronous bits, allowing for the detection of two-thirds of these missing packets. Aoyama et al. [11] proposed modulation schemes for line scan sampling considering discontinuous reception. Although various modulations to transmit data were introduced, none of the above works dealt with the blurry-lights effect, which results in non-identical pattern widths of ‘OFF’ and ‘ON’, and thus, significant performance degradation in data decoding is caused. Chow et al. [12] mitigated the blooming effect (the saturation of pixels) using second-order polynomial fitting, histogram equalization, and Sobel kernels. Although this technique enhances OCC data signal performance, it requires additional computation capabilities resulting in burdens on mobile devices. An Otsu thresholding scheme was proposed to define the data logic in the rolling shutter pattern [13]. However, this scheme is highly dependent on vertical column pixel selection, which should be conducted for individual image frames. While the technical feasibility of the rolling shutter effect has been analyzed in a variety of research, details of the rolling shutter effect vary among research works, and none of the above works considered a simpler and more accurate decoding method for rolling shutter-based OCC in the presence of a variable dimming level.

In this paper, a simple transition-based decoding method for OCC using common CMOS rolling shutter cameras is proposed. The LED flicker encodes the transmitted signal with data rates that far exceed the frame rate of the camera. For supporting dimming functionality, we apply multi-coded variable pulse position modulation (MC-VPPM) [14], where one MC-VPPM symbol pulse is constructed with 10 pulses, and enables dimming control in 10% increments. On the receiver side, the rolling shutter process produces image pixel values row-sequentially. In a practical scenario, irregular flicker patterns are observed due to light blurring and the relative position between the LED and the mobile device. Hence, the conventional approach of identifying ‘ON’ and ‘OFF’ pulses based on a particular frequency or threshold may not correctly receive information in the presence of light blurring. The proposed approach decodes data using a high-low, low-high transition of two sequential symbol pulses. For this, the extended symbol pulse is determined whether the second symbol pulse can be included for the next bit decoding process or not. Because the proposed decoding approach does not depend on complicated image processing techniques, it is possible to simply and effectively mitigate the influence of the light blur. Using this method, we tested our proposed decoding method under different scenarios using a hand-held smartphone. By observing the experimental results obtained by changing the experimental parameters such as ISO, dimming, and receiving camera angle, the performance of the proposed technique was analyzed.

The main contributions of this paper are as follows. First, a simple transition-based decoding process is provided without the requirement for complicated image processing techniques. Second, robust performance under the proposed scheme against the effect of imaging degradation, including light blur, was presented. Third, practical evaluations with variant dimming levels are provided. Based on the MC-VPPM symbol pulse that is compatible with dimming control, the proposed transition-based decoding method can simply and effectively mitigate the influence of light blur during the decoding process, and thus, it enables to perform robust data communication and dimmable illumination simultaneously.

The paper is organized as follows. In Section II, the rolling shutter effect and its use in visible light communications is described. In Section III, the proposed transition-based data decoding method is described. The description includes a brief discussion of the transmitter and receiver processes. The performance test is discussed in Section IV, followed by the conclusion in Section V.

The rolling shutter mechanism is a method of image acquisition in which each frame is not captured at a single point in time but by horizontally scanning rows of pixels. Most CMOS sensors contain pixels that are arranged in sequentially activated row scanlines, and therefore, do not capture the entire image at once. On activation, each scanline of the sensor array is exposed, sampled, and stored sequentially [15]. When this procedure is completed the scanlines are merged in order to form a single image. Rolling shutter is the term used to describe this process.

Figure 1 shows the procedure of capturing the lines in the image one at a time. Rolling shutter operation of CMOS cameras can actually be utilized in OCC for transmitting data from an LED to mobile devices, such as smartphones. If the LED transmitter is switched ‘ON’ and ‘OFF’ at frequencies higher than the frame rate of the CMOS camera, bright and dark fringes are captured in a single image. When the LED transmitter is on, a bright pixel is stored at the activated row of pixel. When the LED transmitter is off, a dark fringe is stored at the activated row of pixels. Hence, by analyzing the bright and dark fringes in an image, the data logic of the OCC can be obtained at a data rate much higher than the frame rate. This procedure continues until all the scanlines are exposed and the image is completed. Most mobile devices have built-in CMOS cameras nowadays; hence, the deployment cost of wireless communications can be much lower if OCC is implemented using a mobile-device CMOS camera.

In this section, we present an overview of the transitionbased data decoding method for OCC. A block diagram of the proposed scheme is shown in Fig. 2. The following subsections describe the sequential process of the transition-based decoding method.

On the transmitting side, the data stream is divided into signal frames, and the structure of a signal frame is shown in Fig. 3. Each signal frame contains two guard symbols, a synchronization pulse, and a data symbol. The guard symbols are located before and after the synchronization pulse and are used to reduce detection error by clarifying the distinction between synchronization and the data. The reason for the overhead of the guard bit and synchronization bit sequence in addition to the data bit string is to provide highly reliable communication. Conventional rolling shutter-based schemes work well under ideal experimental conditions, but if they are affected by the camera settings or ambient light environment, the pixel width for each bit will vary due to the light blur, which degrades bit decoding performance. The proposed method provides a higher data rate than the global shutter-based methods, while providing more reliable communication than the existing rolling shutter based techniques.

For data modulation, MC-VPPM [14] with a code length of 1 is considered. This presents a modulation structure similar to conventional VPPM [3]. As shown in Fig. 4, the pulse position under MC-VPPM with a code length of 1 will change according to the incoming data, and the number of 0 and 1 chips constituting a symbol pulse changes according to the dimming level. A single symbol consists of 10 chips, and this enables the adjustment of the dimming level by 10%.

The synchronization pulse is used to synchronize the LED transmitter and the camera receiver, and consists of a total of 20 chips. To meet the target dimming level, the synchronization pulse can be produced as seen in Table 1. Because a synchronization pulse contains frequent level transitions, this can be observed as a single long pulse in an image obtained from a rolling shutter sensor. Based on the target dimming level, the data symbol format is presented in Table 2. The position of 1’s chips in the synchronization pulse was set to be as uniform as possible in order to minimize the light flicker. Then, pulse transitions between two adjacent synchronization pulses are analyzed by setting the width of the synchronization pulse to be the largest in one image obtained through the rolling shutter sensor, thereby performing data decoding.

In a signal frame structure, a data symbol segment contains a total of eight data symbols, where each symbol consists of 10 chips. Considering two guard symbols, there are 10 symbols between adjacent synchronization pulses, i.e., guard symbols (two symbols) and data symbols (eight symbols). Using an LED driver, visible light LED luminaires provide data frame transmission and light illumination. To support basic functionality of LED sources, there should be no visual flickering when bit streams are transmitted using LEDs. In order to safeguard the human eye, the modulation frequency to drive the LED must be high enough to avoid flicker. Several research studies related to the LED flickering issue show that a satisfactory modulation frequency is at least 100 Hz to avoid bit flickering.

At the receiver, the image frame obtained from a rolling shutter sensor is processed for data decoding. Each image is obtained in grayscale format, in which 0 means a completely dark pixel while 255 means a completely bright pixel. In rolling shutter mode, light integration takes place on all rows of pixels in sequence, and then, it operates like a scanning function. During the capture time for the camera, the LED changes state many times while the shutter of the image sensor opens. The main criteria in the design of the decoder are minimizing decoding errors and reducing decoding computations. The following sub-subsections describe the sequential process of data decoding.

3.2.1. Image acquisition and ROI detection

In the image acquisition block, the image sensor captures a 2D pixel image. In the received image, the pixel region for data decoding is a portion of an image. To extract the data from the pixel region, the flicker pattern and the rest of the image must be separated. This process is called region-of-interest (ROI) detection. Using image binarization based on a pixel threshold, the resultant ROI can be obtained. Under the rolling shutter procedure, temporal modulation of the display is converted into a spatial flicker pattern. In the ROI detection process, we can separate the flicker pattern and the rest of the image, and thus, the data pixel region is extracted.

3.2.2. Horizontal projection

In general, an image plane and the illumination area are not aligned perfectly when a user walks around holding a mobile device with a built-in rolling shutter camera under LED lighting. Due to this fact, the ROI in the image should be projected horizontally in order to recover the data logic 1 and 0.

3.2.3. Symbol pulse width measurement

To implement transition-based decoding, the extended high symbol pulse and low symbol pulse are determined as a reference pulse size. The extended symbol pulse is generated from consecutive symbol pulses with the same level. Remember that a synchronization pulse consists of a total of 20 chips, and thus, it always presents the longest high pulse in the received image at most of the dimming levels greater than 30%. Using this fact, the extended high symbol pulse for data decoding is defined by the second longest high symbol pulse, not by the longest. For low symbol pulses, the extended low symbol pulse is set to the longest low symbol pulse.

Consider an example in which the dimming level is 70%, and bits 1 and 0 come in succession. Then, a total of 20 chips for two symbol pulses are presented as ‘0001111111 1111111000’. In this case, the number of high chips is 14, and this can be regarded as an extended high symbol pulse. When data bits 0 and 1 come in succession, a total of 20 chips for two symbol pulses are presented as ‘1111111000 0001111111’. Here, the number of low chips is 6, and this can be regarded as an extended low symbol pulse.

3.2.4. Frame synchronization

In order to synchronize signal frames in time, the synchronization pulse is detected, and observed as 20 high chips in the signal with the widest high pulse width. To find the start of the frames, guard symbols of 1 and 0 should be inserted before and after the synchronization pulse to distinguish the start and end of the data. Guard symbol 0 is composed of ‘1111111000’ (70% dimming), and it always finishes with a low pulse before a synchronization pulse. Similarly, the guard symbol 1 consists of ‘0001111111’ (70% dimming), and it always starts from a low pulse after a synchronization pulse. This means that the insertion of guard symbols generates the level transition at the start and end points of the synchronization pulse, and thus, it makes it simple to synchronize the signal frame.

3.2.5. Data decoding

The received image is not homogeneous in terms of brightness mostly because the relative position between light source and mobile device varies. To normalize the intensity levels of ‘ON’ and ‘OFF’ in a single image, conventional approaches utilize histogram equalization, image filtering, and vertical column pixel selection, which eventually increases computational complexity and processing time. In addition, these approaches are vulnerable to the light blurring phenomena, such as light smear, motion blur, and focus blur, which result in variations of the width of high and low symbol pulses in a single image or sequential images. In this case, it is difficult to distinguish between consecutively connected symbols and a single symbol, thereby causing serious errors in data decoding.

To solve this problem, the transition-based method is proposed, which can decode data by observing level transitions of only two successive symbol pulses. If two symbol pulses are configured as high-low and low-high, data can be decoded as 0 and 1, respectively. Note that the proposed decoding scheme is not affected by the changes in pulse width, which varies from one measurement to the next due to light blurring.

As mentioned in the pulse width measurement process, when two consecutive bits are received as 0-1 or 1-0, two consecutive high symbol pulses or low symbol pulses are observed, which results in twice the width of a normal high or low symbol pulse. Considering this fact, the second symbol pulse is checked for whether it corresponds to an extended symbol pulse or a normal symbol pulse. Note that an extended symbol pulse is always twice as long as a normal symbol pulse. When the second symbol pulse is identified as an extended symbol pulse, the second symbol pulse considered in the previous bit decoding is used as the first symbol pulse in the next bit decoding. Otherwise, the second symbol pulse is not considered for the next bit decoding process.

Figure 5 presents examples of a transition-based decoding scheme when a 50% dimming level is considered. In decoding Example 1, the first level transition occurs in the ‘L-(H, H)’ pattern, and thus, it is decoded to a data bit of ‘1’. Note that an extended symbol pulse ‘(H, H)’ is located at the second pulse position in the first consecutive pulse set {1} for bit decoding, and thus, the corresponding extended symbol pulse is used as the first symbol pulse at the second bit decoding. In the second consecutive symbol pulse set {2}, the second level transition occurs in the ‘(H, H)-(L, L)’ pattern. Therefore, it is decoded as a data bit ‘0’. Because an extended symbol pulse ‘(L, L)’ is located at the second pulse position, this pulse is considered as the first symbol pulse at the third bit decoding. In the third consecutive symbol pulse set {3}, the third level transition occurs in the ‘(L, L)-H’ pattern. Applying the same decoding rules, it is decoded as data bit ‘1’

Note that pulse set {1} is a set used to decode the first symbol right after the synchronization pulse, which means the bit value of the guard symbol. Based on this, a total of 10 symbols can be decoded and data bits are obtained with 8 symbols excluding the first and the last symbols indicating guard symbols.

In decoding Example 2, the first level transition occurs in the ‘L-H’ pattern, and thus, it is decoded to a data bit of ‘1’. Note that a normal symbol pulse ‘H’ is located at the second pulse position in the first consecutive pulse set {1} for bit decoding, and thus, the corresponding normal symbol pulse is not used for the second bit decoding. In the second consecutive symbol pulse set {2}, the second level transition occurs in the ‘L-(H, H)’ pattern. Therefore, it is decoded as a data bit ‘1’. Because an extended symbol pulse (H, H) is located at the second pulse position, this pulse is considered as the first symbol pulse at the third bit decoding. In the third consecutive symbol pulse set {3}, the third level transition occurs in the ‘(H, H)-L’ pattern. Applying the same decoding rules, it is decoded as data bit ‘0’.

In summary, the transition-based decoding process acquires two pulses following the synchronization pulse, and checks whether the two separated pulses are composed of low-high or high-low pulses, which correspond to data bit ‘1’ or ‘0’. If the length of the second pulse in two consecutive pulses is identified as an extended symbol pulse, the second pulse is further considered in the next data bit decoding. Otherwise, the second pulse is not included for the next bit decoding.

The performance test of the proposed transition-based decoding scheme was conducted in an indoor environment. For OCC, the 20 watt commercial LED luminary controlled by a UART in a Raspberry Pi device was used to transmit data at 10 kHz. The image sensor in a Samsung galaxy smart-phone was used as a data receiver, and the size of captured images was 3200 × 2400. As shown in Fig. 6, the images were taken between 1.5 m and 2 m line-of-sight (LOS) distances centered on the LED lights. To observe the effect of the dimming level in the proposed scheme, we performed experiments with 100 captured images at varying dimming levels from 30% to 80%. Considering the hand-held scenario, the rotation angle of the camera was randomly chosen at between -60 and 60 degrees. Figure 7 shows examples of received images under the experiment environment.

To highlight the rolling shutter effect in the receiving process, the ISO, which is the camera’s setup standard for sensitivity to light, should be raised. For normal light exposure, ISO 100 is the standard, and the higher the number, the higher the sensitivity to light. Figure 8 shows the change in the rolling shutter effect when the ISO is changed from 80 to 800. We can see that the rolling shutter effect becomes distinct when the sensitivity to light is increased to ISO 800 compared to ISO 80. To deploy the rolling shutter effect even when the light intensity of LEDs is weak, it is necessary to raise the ISO so that the rolling shutter effect becomes more apparent by reacting sensitively to weak light.

The image region where the LED is located in the image is a portion of the total image region. Because white LED lighting was used, the illumination area of the image has a very high pixel value, compared to the surrounding background. Based on this, image binarization was performed to extract the ROI from every frame. Considering a gray-level image, a global thresholding value of α ⋅ I_max (I_max = 255, α = 0.6) was used for image binarization. Figure 9 shows histograms of the sum of the pixel values located on the same horizontal axis (x-axis) and vertical axis (y-axis) in the image. It can be seen that the illumination area becomes apparent, compared to the background. In our scheme, the ROI of the LED illumination region was extracted using a threshold value at half of the maximum sum of the pixel values along the x-axis.

As shown in Figs. 7, 8, and 10, even if reflection or blurring exists near the illumination area, we can see that noise is generated only in the high region of the same horizontal line. This is due to the fact that the scanning process in the rolling shutter camera is in the horizontal direction. In this regard, if 20% of the pixel values on the same scanline have high values, all the pixel values on the scanline are set to a high value, which is called horizontal morphology. Based on this process, the ROI image is projected to generate a barcode-like image.

In Fig. 10, even if the dimming level is 50%, high and low pulse widths can change due to the performance of the receiving camera or the ambient light environment. In this situation, the proposed transition-based decoding scheme enables accurate data decoding. Unlike conventional methods which decode all the bits by measuring the pixel width of all pulses, the proposed scheme only measures the pixel width of specific pulses to confirm whether it is an extended pulse or not. The main decoding process is based on ‘ON’ and ‘OFF’ changes of consecutive pulses. Therefore, the most important feature of the proposed technique that it is possible to decode correctly even in the presence of light blur effect.

It is assumed that at least two synchronization pulses are located in one image to extract data. Experiments at various angles between the image plane and the illumination area showed that data decoding is possible from −60 to 60 degrees. Figure 11 shows examples of a misaligned illumination area in an image. When an LED is misaligned by 41 degrees, three synchronization pulses are located in a single image. At 72 degrees, however, only one synchronization pulse is observed. Because the start and end points of data symbols in a data frame can be determined by two successive synchronization pulses, angles between the image plane and the illumination area should lie in an angle range where at least two synchronization pulses are observed in order to decode data successfully. After the horizontal morphology process, projections were made to clearly distinguish the low and high pulses in images. Figure 12 presents the projected image seen as a barcode when a dimming level of 50% is considered.

The data decoding process begins by finding the two longest high pulse widths, regarded as the synchronization pulse. Then, data decoding is performed sequentially by using the next two symbol pulses. The threshold width to distinguish the extended high symbol pulse is measured from the width of the third largest high pulse among all the high pulses, w_h, because there should be two synchronization pulses in an image. By setting the threshold width as β ⋅ w_h (β = 0.74), the extended and normal high pulses can be successfully distinguished. For the reference threshold to distinguish the extended low pulse, the low pulse right after the synchronization pulse is measured, because it always shows the width of a normal low pulse, w_l. By setting the threshold width as γ ⋅ w_l(γ = 1.1), the extended and normal low pulse can be identified. Therefore, it is an important feature that the proposed method can correctly decode without prior information of the dimming level at the receiving end.

From Fig. 12, the third largest high pulse with width w_h, and the normal low pulse with width w_l can be observed. Note that the synchronization pulse is followed by a low pulse. Because the sequence of pulses located after the synchronization pulse is a low-high pattern symbol pulse, and the dimming level was set to 50%, the modulated pulse chips are ‘0000011111’ and the data bit is decoded as 1. Note that the next high pulse has the width of an extended high pulse. Therefore, data decoding should start from the high pulse in the next decoding. Because the next two pulses for decoding are a high-low pattern symbol pulse, modulated pulse chips are set to ‘1111100000’ and data is decoded as 0. Because the width of the low pulse located at the second position is an extended width, data decoding is performed from the low pulse at the next decoding.

In this way, data decoding is performed until the next synchronization pulse appears. As a result of the experiment, the modulated symbol pulse patterns were decoded as ‘L-H-H-L-L-H-H-L-H-L-H-L-L-H-L-H-H-L-H-L’. Therefore, the original data bits are decoded as ‘1-0-1-0-0-0-1-1-0-0’. Removing the guard bits located before and after the synchronization pulse, we obtained the final eight data bits of ‘0-1-0-0-0-1-1-0’. Table 3 shows the decoding correction ratio of the proposed scheme with regard to various dimming levels. We can see that the proposed scheme can provide robust decoding performance at various dimming levels. Because the proposed decoding approach observes two consecutive pulse patterns with transition, it is both simpler and more accurate than the conventional approaches based on complicated filter computations.

Among the various OCC applications, the proposed scheme can be used for indoor positioning. The camera captures multiple LED luminaries positioned on the ceiling, which transmit their unique identification code using OCC. Thus, by querying a central system, the mobile device translates the LED identifiers into location coordinates that can be used to derive its own position.

In this paper, we proposed a simple transition-based technique to decode data using a rolling shutter camera. In rolling shutter sensors, different rows of pixels are exposed in rapid succession, and thereby, the temporally modulated light coming from the LEDs is converted into a spatial flicker pattern in the captured image. Because of the image degradation caused by light blurring and the relative position between the LED and the mobile device, conventional approaches use complicated image processing techniques resulting in burdens on mobile devices. Unlike conventional schemes, the transition-based approach decodes data using the level transition of two consecutive pulses. Comparing the second pulse and the predetermined extended symbol pulse decides whether the second pulse should be included in the next bit decoding process or not. Using the proposed scheme based on two sequential pulses with level transition, it is possible to simply and effectively mitigate the influence of light blur without complicated image processing techniques. Experimental results show that the proposed decoding scheme can provide robust decoding performance at varying dimming levels.