According to the US National Heart, Lung, and Blood Institute, 5 million Americans have heart failure, and 300,000 die from it every year. Total costs for treating heart failure in the United States are estimated to be $34.8 billion in 2008 and this number is increasing year by year.

To prevent heart diseases it is important in time diagnose heart problem and if a problem is detected, then take appropriate actions. Instant heart diagnosis would aid in heart disease prevention.

Instant heart analysis can be achieved by developing a smartphone application that at any time of the day without any additional hardware and sensors will diagnose patient’s heart [1]. The diagnosis can be instantly sent to a medical doctor for further analysis.

Today there are several applications available on the market. These applications use reflection photoplethysmographic imaging to extract the heart rate signal. Photoplethysmographic imaging is a noninvasive means of sensing the cardiovascular blood volume pulse [1]. These applications are capable of detecting heartbeat and heart rate from the finger placed over the camera lens with the flash turned on [2,3]. However, besides the detected heart rate they do not provide any further analysis. The idea of using photoplethysmographic imaging for heart rate detection was also extended to sensing the pulse from the human face using a single webcam [4].

In this paper we describe a heartbeat detection and error correction algorithms, using mobile phone’s camera. The detected heartbeats are further processed and inter-beat intervals are extracted. The time-series constructed of interbeat intervals is termed heart rate variability (HRV) and is known to contain information about nervous system activity and heart.

The HRV is a powerful source of information and in recent years attracted a large body of researchers [5]. Various signal-processing tools have been applied to the HRV signal producing physiological marker that found to be useful in several medical applications including diagnosis of myocardial infarction [6], diabetes [7], and chronic heart failure [8]. Particularly it was found that patients with a restrictive left ventricular filling pattern had a lower HRV index compared to patients with a non-restrictive pattern [7]. The HRV analysis is also a relevant indicator of athletic conditions [9].

Besides presenting heart beat detection algorithm, the goal of this paper is to compare HRV extracted from a video recorded by smartphone camera and HRV extracted from an electrocardiogram (ECG). Due to the fact that accuracy of estimated HRV depends on the sampling rate we investigate whether the limited camera’s frame rate produces HRV comparable to HRV extracted from ECG. We also calculate and compare statistics of both groups of HRV signals.

These statistical characteristics vary for people with healthy heart and heart with congestive heart failure or other heart conditions. Therefore, it is important to verify that these characteristics remain unaltered regardless of the sensor used in recording of a cardio-signal.

The experimental setup consists of a Samsung Galaxy S2 (Samsung, Seoul, Korea) and Mitsar EEG 201 recorder that was employed as an ECG recorder. To record video a palmar side of right index finger was placed over the camera lens. The flash was turned on while the video is being recorded. To simulate recording in a real environment, subjects were told to apply pressure that feels comfortable to them.

For the purpose of reducing computational load we selected the lowest available resolution of 176 × 144 and to increase the precision of extracted HRV we set the sampling rate to the maximum available frame rate of 30 fps.

Simultaneously with video recording, we recorded the ECG with Mitsar EEG 201 that was a ground truth signal in our experiment. Two electrodes were placed on the patient’s body to record ECG. One of the electrodes was attached to patient’s ear and the other placed on the skin surface around the heart.

Five male subjects participated in our experiment ranging from 21 to 32 years old. All the signal analysis was conducted in MATLAB (MathWorks, Natick, MA, USA) as described below.

We developed a heartbeat detection algorithm to analyze and compare HRV extracted from the video signal. The ECG signal is relatively clean and by simply finding maxima that are located above a predefined threshold we were able to detect R-beats.

On the other hand it is not trivial to detect R-beats form video frames due to the fact that the color intensity is affected by the pressure applied by the finger. Moreover, the color response is different for each color channel. Scully et al. [2] suggested calculating average for a window of 50 × 50 pixels in green channel. We replicated the experiment and found that in case when the subject apply significant pressure to the finger against the camera lens, values of the green channel become close to zero and therefore not applicable for heart beat detection in Fig. 2(b). On the contrary calculating the average for a blue channel produces a noise like signal when the finger is resting on the lens without any pressure. The heartbeat disappears when the subject presses the finger against the lens. An average of the red channel produces a relatively clean signal regardless of the pressure. Although the signal often fluctuates close to the maximum and sometimes gets threshold. Through trial and error we found that composite signal obtained as a sum of normalized averages of each color channel produces a signal applicable for heart beat detection (Fig. 2). We faced a number of difficulties associated with finding heartbeats in the composite signal (Fig. 2).

As it can be seen the beginning and the end of the signal contains high amplitude spikes that should be filtered out. These spikes are caused by the transition of intensity adjustment from the time when camera is turned on and the finger is yet not placed over the camera and the time when finger is over the camera.

To remove the spikes we applied Random Sample Consensus (RANSAC) algorithm [10]. RANSAC finds all the data points that conform a selected model. The data points that do not fit the model are considered as outliers and therefore removed.

The second problem comes from significant amplitude variations (Fig. 2). To overcome this problem we proposed the following beat detection algorithm:

1. Apply pass-band filter with the normalized pass frequencies [0.05 0.5];

2. Calculate the range signal for a running window of 11 samples (approximately 1/3 of a second) (Fig. 3(a), green line).

3. Apply pass-band filter to smooth the range signal (Fig. 3(a), blue line). The normalized pass-band is [0.01 0.1].

4. Apply local maxima detector to the smoothed range signal (Fig. 3(a), blue line).

5. The detected local maxima (Fig. 3(a)) are used as an initial guess in the original signal, look for nearby maxima (Fig. 3(b)).

Every heartbeat is associated with a rapid change of the intensity. In between beats the variations are insignificant. By calculating the range signals we are able to distinguish one beat from another. The obtained range signal is noisy and to remove the noise and smoothen up the range signal we applied the band-pass filter. The maxima located in the resulted signal are approximately at the positions of the maxima in the original composite intensity signal. The detected maxima are not accurate due to the phase shift caused by pass-band filtering. To correct the location of maxima in the original signal, the approximate locations of the maxima are taken as initial gausses. We look for the maxima in the window of 10 samples centered at the initial gausses.

To obtain HRV signal we simply calculated the diffe-rences between consecutive beats and divide each of the differences by the frame rate.

The above described beat detection algorithm produces a clean HRV signals, nevertheless some errors may occur due to ectopic or missed beats. Fig. 3(b) shows a missed beat around 3180th sample.

Generally we distinguish two types of errors: missed beats that cause occurrence of peaks in the HRV signals in Fig. 3(a), and incorrectly detected R-waves that cause errors that we termed z-type of error in Fig. 3(b) usually these errors are result of ectopic beats.

The goal of the error correction procedure is not to simply remove the errors but to correct them. If we simply remove the erroneous samples the total length of the original ECG will be shorten, which is undesirable for our further analysis. We should keep in mind that every sample in HRV corres-ponds to the interval between R-waves.

In case when R-wave detector fails to detect R-waves, the interval increases causing the peaks in HRV (Fig. 3(a)). On average such long intervals will correspond to the number of true intervals that were missed.

The peaks are simply replaced with constant values. The number of constant values is calculated as follows,

where V_peak is the peak value and E_hrv is the average value calculated for the entire HRV. The values that will be substituted in place of the peak value are calculated as follows,

The z-type of errors are simply corrected by replacing the two incorrect values with their average.

Five male subjects ranging from 21 to 32 years old participated in our experiment. Each subject was asked to place the right hand index finger over the camera lens for five minutes.

The above-proposed beat detection and correction algorithms were applied to extract HRV from the composite signal obtained by summing up the normalized average intensities of red, green, and blue channel. In this section we compare two groups of HRV signals. One extracted from the video and the other from ECG.

The first test consists in calculating the mean square error (MSE) between each pair of HRV signals. The MSE is given in milliseconds. From Table 1 it is seen that in four out of five cases the MSE does not exceed 10 ms.

A small error such this provides evidence that our algorithm is capable of extracting HRV from a mobile phone’s camera with an error of a few milliseconds.

The reader is able to visually compare both HRV signals presented on Fig. 5. With the naked eye it is easy to observe that HRV extracted from the camera (green lines) have higher variations than the HRV extracted from the ECG (the blues lines). Even though there is a good match for low variation, the differences mostly appear at high frequencies.

To numerically compare the alteration in HRV variation between corresponding HRV signals we calculated the root mean square of successive differences (RMS-SD) and the standard deviation of NN intervals (SDNN).

RMS-SD is the square root of the mean squared differrences of successive NN intervals. RMS-SD is measured in milliseconds. This measure estimate high-frequency variations in heart rate in short-term NN recordings that reflects an estimate of parasympathetic regulation of the heart. Table 2 presents estimated RMS-SD for both groups of HRV signals. The last column in the table shows the ratio between the two values of RMS-SD. The ratio in four out of five values is less than 2. Considering that the duration of the recording is only 5 minutes we believe that for longer time-interval the ratio will be converging to a one. Further investigation is needed to prove this point.

The SDNN appears to be more accurate and in for cases out of five it is less than 1.2. The comparison results are shown in Table 3.

Both RMS-SD and SDNN are important measures in heart diagnosis. Decrease in RMS-SD (<10) accompanying with lowered SDNN (<20) is related to high risk of cardiac disease development [11].

Analysis of power spectral densities (PSD) estimated for short HRV signals (5 minutes ECG) is useful in analyzing autonomic nervous system. The total power calculated for the band of 0 to 0.4 Hz reflects the degree of autonomic nervous system activities. The decrease of total power is observed in stressed subjects or subjects with chronic diseases.

High frequency components (0.15 to 0.4 Hz) carry information about parasympathetic or vagal activity. Estimating the total power of this band allows diagnosing a number of cardiac pathologies.

Due to valuable information that PSD contains it is of great importance to verify that the estimated PSD for the HRV signals extracted from video recordings matches the PSD estimated from ECG.

In our experiment we neglected the fact that inter-beat intervals are not uniformly sampled and for the purpose of PSD estimation we resampled HRV signals. We applied Burg estimation method to uniformly resampled HRV signals. The qualitative results are shown on Fig. 6. It can be seen that there is a good match between low frequencies. However, starting from 0.3 Hz the PSDs starting to deviate. Moreover, the PSDs estimated for HRVs extracted from the video signals are always overestimated. These results are similar to the observations we have seen in Fig. 5, where HRV signals (the green lines) extracted from the videos are in good agreement the HRV signals extracted from the ECG (the blue lines) except that the former ones have slightly higher variations at finer scales.

The ability to diagnose heart and autonomic nervous system on the go has significant contribution to the society.

In this paper we proposed the heartbeat detection and error correction algorithms that employ smartphone camera as a sensor. We also showed that extracted HRV signals are in a good agreement with the HRV signals extracted from ECG.

We are planning to extend our testing set and test HRV extraction algorithms in various conditions, i.e., during sport exercises and rest. We are planning to add automatic recording update to the server so medical doctors will have access and possibility to analysis data at any time. We believe that the developed app will indeed change the health care and will save lives.