Real-time human activity recognition based on time-domain features of multi-sensor

LIU Yu, YU Yue, LU Yong-le, GUO Jun-qi, DI Ke, CHEN Yong-wei

(Photoelectrical Engineering Institute; Chongqing University of Posts and Telecommunications,Chongqing 400065, China)

Real-time human activity recognition based on time-domain features of multi-sensor

LIU Yu, YU Yue, LU Yong-le, GUO Jun-qi, DI Ke, CHEN Yong-wei

(Photoelectrical Engineering Institute; Chongqing University of Posts and Telecommunications,Chongqing 400065, China)

An algorithm of activity pattern recognition based on the time-domain features of accelerometer,gyroscope and barometer was proposed for high-accuracy real-time human activity pattern recognition. The time-domain feature obtained from multi-sensor is selected as the only feature parameter, and the activity recognition is realized through feature extraction operation. The data tests on the independent hardware and software platforms in the laboratory indicate that the average recognition rate, for the real-time activity with 8 kinds of daily activities and 4 kinds of falling down activities, is above 94% with extremely short recognition time of about 2 s each time. The test result proves that this algorithm not only significantly improves the real-time precision of the present algorithms, but also expands the types of activity recognition, which has great application prospect in wearable intelligent terminal areas.

activity pattern; real-time recognition; multi-sensor; time-domain feature

With the continuous development of MEMS technology, the MEMS sensors with low price, small volume and high sensitivity are widely used in most kinds of wearable intelligent terminal. Accelerometer, gyroscope and barometer are integrated in the wearable intelligent terminal developed in the laboratory which can obtain real-time three-axis data of acceleration, three-axis data of angular velocity and air pressure. Human activity patterns can be speculated according to the signal feature extraction and classification algorithm based on multisensor[1-3]. Using multi-sensor to achieve activity recognition is a new field of pattern recognition research direction, and has a wide prospect of application.

Feature extraction is the most important part of the activity recognition. Feature extraction and selection can seriously affect the recognition result. Preece and others[4]studied many methods about feature extraction, and compared the characteristics of wavelet and frequency domain characteristics. It is found that the recognition effect based on frequency domain is better. HE and others[5]classified the coefficients of the supported vector machine and the acceleration signal auto-regressive model. It is found that, although the frequency domain characteristic is better than the time domain characteristic, the frequency domain characteristic, like the usual FFT (Fast Fourier Transformation), requires a large amount of calculation. It is clear that the frequency domain features are not suitable for activity recognition on wearable intelligent terminal.

In this paper, the time-domain features of accelerometer, gyroscope and barometer are evaluated as realtime parameters. Through feature extraction algorithm,the feature computation time is reduced so that real-time human activity pattern recognition is efficiently realized.The recognition time is extremely short, about once every 2 s. By analyzing a large amount of time-domain data from multi-sensors, the most appropriate features are extracted to obtain stable and precise recognition results. Actions which can be identified contains 8 kinds of daily activities and 4 kinds of specific patterns of falling: still, walk, run, jump, sit down, fall down, go upstairs, go downstairs, fall forward, fall backward,left-side lying, right-side lying.

1 Flow of algorithm

In this paper, 100 sets of real-time data obtained from accelerometer, gyroscope and barometer are used to identify the activity patterns. The acquisition frequency of the data is 50 Hz. Because of the efficient timedomain feature extraction, the recognition time is extremely short, and the recognition time is about 2 s. First, 3-axis(x, y, z) data of acceleration, 3-axis data of gyroscope,and barometer data are acquired every 2 s. The original data of the accelerometer, gyroscope and barometer are pre-processed by filtering algorithm, then the multisensor data is processed to extract the features. A total of 12 eigen values are extracted by using the time-domain feature (variance of acceleration modulo value, crosscorrelation coefficient and so on). 12 eigen values will be composed of 12-dimensional eigenvectors, inputted to the trained BP (Back Propagation) neural network in order to obtain the result. The flowchart of the algorithm is shown in Fig.1.

Fig.1 Flowchart of the algorithm

2 Flow of the data processing

2.1 Raw data processing of accelerometer

Much noise is contained in the original signal of acceleration, not suitable for direct feature extraction and classification. Therefore, it is necessary to preprocess the original signal, eliminate the noise and glitches as much as possible and keep the original features of data. The curve of acceleration can be smoothed by filtering processing and become more suitable for feature extraction and classification after filtering processing. In this paper,Hamming window filter is used to preprocess the original signal. The width of the filter window can be set to achieve the desired effect. The effects before and after using Hamming window filter are shown in Fig.2.

Fig.2 Acceleration signal filtering

2.2 Raw data processing of gyroscope

Gyroscope can be affected by the random errors in the actual measurement process, making the data irregularly fluctuate. In this paper, a Kalman filter is used to improve the gyroscope’s output. It can effectively converge and inhibit random errors, making the output value of gyroscope closer to the true value.

Kalman filtering is an optimal estimation method under the minimum error covariance criterion. It has short computation time and high real-time performance.Actual motion parameters are used to modify the estimated value of future motion state and improve the estimation accuracy and stability.

The state equation and the measurement equations of Kalman filter is:

Where: A is the state transition matrix; H is the measurement matrix; W and V is the state and the measured noise matrix respectively, which are uncorrelated Gaussian white noise; Q and R are the variance of the noise.

The state vector prediction equation is:

The state vector covariance matrix prediction is:

The state vector updating equation is:

The state vector covariance update equation is:

The Kalman gain matrix is:

The result about using Kalman filter before and after is shown in Fig.3.

It can be seen from Fig.3 that, after several iterations, the Kalman filter can accurately get the true output of the gyroscope.

Fig.3 Using Kalman filter before and after

2.3 Raw data processing of barometer

In order to eliminate the pulsed and cyclical disturbance of the barometer data, the average median filter is used to process the barometer data[8]. First, N pieces of data are continuously collected from barometer. Then the maximum and minimum values are removed, and the remained N–2 data are averaged. The filtering method combines the characteristics of the median filter with the arithmetic average filter, which can eliminate the occasional impulsive interferences and has good inhibition to the periodic interferences. The results of filtering are shown in Fig.4.

Fig.4 Barometer signal filtering

It can be seen from Fig.4 that the fluctuation of barometer data is within ± 0.2 m after filtering, which is improved by nearly 80%.

3 Feature extraction

The time-domain features of one accelerometer can also be used into the application of pattern recognition[6-7].With the increasing demand for activity pattern recognition with high accuracy and rich types, one accelerometer has been unable to meet the demand.

In this paper, extensive data analysis has been made among the daily activities of human body, and the timedomain eigen values of accelerometer, gyroscope and barometer are used to determine the pattern. The type of pattern recognition is increased and the accuracy of corresponding recognition is improved by using multi-sensors.

Based on the pretreatment of each signal, the signal feature parameters of multi-sensor are extracted. Experiments in this paper only use time-domain features. In this paper, the Variance of Acceleration Modulus, Crosscorrelation coefficient, Mean of the barometer output difference, Output variance of barometer, Extreme value of acceleration, and Degree of inclination are used as eigen values. Specific formula as follows:

1)Variance of acceleration modulus

In this formula, Aiis acceleration modulo value,the number of data, A is the average of acceleration modulo values.

2)Cross-correlation coefficient

The cross-correlation coefficient C is the ratio of the covariance of the two-axis acceleration to the product of the standard deviations of the two-axis acceleration. For example, the cross-correlation coefficient of X-axis and Y-axis is:

In this paper, four kinds of activities can be distinguished by using variance of acceleration modulo value and acceleration cross-correlation coefficient of y- and z-axis. The distribution of eigen values is shown in Fig.5.

Fig.5 Eigenvalue distribution

3)Mean of the barometer output difference

4)Output variance of barometer

In this paper, barometer’s difference in the preceding paragraph and the barometer’s output variance are used to distinguish the movement of walking up and down stairs or at the same height. The recognition result is shown in Fig.6

Fig.6 Distribution of mean of barometer output difference and output variance of barometer

5)Extreme value of acceleration:

In this formula, T is the range of the accelerometer,are the maximum and minimum values,respectively, during the sampling period. This value is used to describe the carrier in a very short period of time occurred in the amplitude of the acceleration.

6)Degree of inclination

It is found through a large number of data analysis that: when human body’s pitch angle is greater than 45°or less than -30°, people may easily fall, and when human body’s roll angle is greater than 30° or less than-30°, people is also prone to fall. The extreme value of acceleration and degree of inclination can be used as an important basis for judging the state of human fall. Two modes, like fall backward and sit down, can be recognized by extreme value of accelerometer’s Y-axis and degree of inclination of gyroscope’s Y-axis. Comparison chart is shown in Fig.7.

The use of extreme value of acceleration and degree of inclination can effectively distinguish fall and sit down, can also be used to identify four specific patterns of falling down, including: falling forward, falling backward, left-side lying, right-side lying. Distribution of acceleration’s extreme values and degrees of inclination is shown in Tab.1.

Fig.7 Distribution of extreme values of accelerometer and degree of inclination

Tab.1 Four specific patterns of fall down

4 Experimental results analysis and discussion

In this paper, a feed forward BP neural network classifier is chose as the classification algorithm. The neural network is composed of one input layer, two hidden layers and one output layer. The input layer contains 12 neurons; the first hidden layer contains 7 neurons, the second hidden layer contains 6 neurons; the output layer contains 8 neurons. The feed forward BP neural network classifier needs many times of training to obtain the most suitable network weight.

Experimental terminal used in laboratory integrates one 3-axis accelerometer, one 3-axis gyroscope and one barometer. The device is worn at the waist of the tester,as shown in Fig.8a. Man-machine interface is used to make a communication with terminal, and each group of actions data from terminal is displayed and simulated on the man-machine interface, as shown in Fig.8b.In order to train the neural network model, the experimental data were collected from 30 subjects (21 men and 9 women). Testers were asked to collect data for eight daily patterns of motion and four specific patterns of falling down, and each time the data was collected as one data sample. These samples were used to train BP neural networks. Inputting the eigen values of the real-time calculation into the trained BP neural network can quickly identify each action of the tester. 20 subjects (15 males, 5 females) were randomly selected from the 30 subjects for accurate statistics. The results of the experimental precisions are shown in Tab.2 and Tab.3.

It can be seen from the Tab.2 and Tab.3 that, in the case of pattern recognition once every 2s, the recognition accuracies of 8 daily activities and 4 patterns of falling are all higher than 90%, and the result of recognition is good.

Fig.8 Real-time recognition test

Tab.2 Recognition accuracy of 8 daily activities

Tab.3 Recognition accuracy of 4 types of fallings

5 Concluding remarks

The feature extraction algorithm using frequency domain and wavelet is too complex, not conducive to realtime identification, and one single accelerometer has been unable to meet the increasing demand for activity pattern recognition with high accuracy and rich types. In this paper, an algorithm based on the time-domain features of accelerometer, gyroscope and barometer is proposed. The complexity of the algorithm is reduced, for which the time of feature computing is decreased and the real-time performance is effectively improved. The algorithm is tested on the wearable intelligent terminal developed by the laboratory. Under the condition of once every 2 s, the average recognition precision of the test is over 94%, and the recognition accuracy of the activity pattern about go upstairs and go downstairs is more than 93%. The recognition accuracy of the activity pattern is greatly improved compared with other algorithms. Therefore, the proposed algorithm of real-time human activity pattern recognition based on multi-sensor has great application prospect.

[1]Halabi A E, Artail H. Integrating pressure and accelerometer sensing for improved activity recognition on smartphones[C]//Third International Conference on Communications and Information Technology. IEEE, 2013: 121-125.

[2]Liu S, Gao R X, John D, et al. Multi-sensor data fusion for physical activity assessment[J]. IEEE Transactions on Biomedical Engineering, 2012, 59(3): 687-96.

[3]Palumbo F, Barsocchi P, Gallicchio C, et al. Multi-sensor data fusion for activity recognition based on reservoir computing[J]. Communications in Computer & Information Science, 2013, 386(2): 24-35.

[4]Preece S J, Goulermas J Y, Kenney L P J, et al. A comparison of feature extraction methods for the classifycation of dynamic activities from accelerometer data[J].IEEE Transactions on Biomedical Engineering, 2009, 56(3): 871-879.

[5]He Zhen-Yu, Jin Lian-wen. Activity recognition from acceleration data using AR model representation and SVM[C]//International Conference on Machine Learning and Cybernetics. 2008: 2245-2250.

[6]Lau S L, David K. Movement recognition using the acelerometer in smartphones[C]//Future Network and Mobile Summit. 2010: 1-9.

[7]Bujari A, Licar B, Palazzi C E. Movement pattern recognition through smartphone’s accelerometer[C]//Proceedings of the IEEE Consumer Communications and Networking Conference. 2012: 502-506.

[8]Zhu You-lian, Huang Cheng. An improved median filtering algorithm combined with average filtering[C]//Third International Conference on Measuring Technology and Mechatronics Automation. IEEE, 2011: 420-423.

[9]Qian Wei-xing, Zeng Qing-hua, Wan Jun-wei, et al. Pedestrian navigation method based on kinematic mechanism of human lower limb[J]. Journal of Chinese Inertial Technology, 2015, 23(1): 24-28.

[10]Lu Yong-le, Zhang Xin, Gong Shuang, et al. Recognition of multiple human motion patterns based on MEMS inertial sensors[J]. Journal of Chinese Inertial Technology, 2016,24(5): 589-594.

1005-6734(2017)04-0455-06

10.13695/j.cnki.12-1222/o3.2017.04.006

2017-04-05；

2017-07-14

國家自然科學(xué)基金資助項(xiàng)目（51175535）；國際聯(lián)合研究中心科技平臺與基地建設(shè)（cstc2014gjhz0038）；重慶市基礎(chǔ)與前沿研究計(jì)劃項(xiàng)目（cstc2015jcyjBX0068）；重慶郵電大學(xué)博士啟動基金（A2015-40）；重慶郵電大學(xué)自然科學(xué)基金（A2015-49）；重慶高校優(yōu)秀成果轉(zhuǎn)化資助項(xiàng)目：基于自主傳感的消防單兵定位與體征及環(huán)境檢測多功能可穿戴系統(tǒng)（KJZH17115）

劉宇（1972—），男，碩士生導(dǎo)師，教授，主要從事慣性導(dǎo)航及傳感器件的研究。E-mail: liuyu@cqupt.edu.cn

基于多傳感器時(shí)域特征的實(shí)時(shí)人體行為識別

劉 宇，余 躍，路永樂，郭俊啟，邸 克，陳永煒
（重慶郵電大學(xué) 光電工程學(xué)院，重慶 400065）

提出了一種基于加速度計(jì)、陀螺儀、氣壓計(jì)輸出時(shí)域特征的高精度、高實(shí)時(shí)性的人體行為模式識別算法。該算法選取多傳感器輸出的時(shí)域特征值作為唯一特征量，通過特征提取運(yùn)算實(shí)現(xiàn)行為的實(shí)時(shí)識別。通過在實(shí)驗(yàn)室自主研發(fā)的軟硬件平臺上進(jìn)行測試，在識別時(shí)間縮短到 2 s一次的條件下，對于8種人體日常行為模式和4種摔倒模式的平均識別率可達(dá)到94%以上。該算法對于現(xiàn)有算法實(shí)時(shí)精度有明顯提高，且拓展了模式識別的種類，在可穿戴智能終端領(lǐng)域具有很好的應(yīng)用前景。

行為模式；實(shí)時(shí)識別；多傳感器；時(shí)域特征

U666.1

A