Dual-modality smart shoes for quantitative assessment of hemiplegic patients’ lower limb muscle strength

Huajun Long,Jie Li,Rui Li,Xinfeng Liu,and Jingyuan Cheng ?

1Department of Data Science, University of Science and Technology of China, Hefei 230027, China;

2Department of Neurology, the First Affiliated Hospital of USTC, University of Science and Technology of China, Hefei 230001, China;

3Department of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, China

Abstract: Stroke can lead to the impaired motor function in patients’ lower limbs and hemiplegia.Accurate assessment of lower limb motor ability is important for diagnosis and rehabilitation.To digitalize such assessments so that each test can be traced back at any time and subjectivity can be avoided,we test how dual-modality smart shoes equipped with pressuresensitive insoles and inertial measurement units can be used for this purpose.A 5 m walking test protocol,including the left and right turns,is designed.The data are collected from 23 patients and 17 healthy subjects.For the lower limbs’motor ability,the tests are performed by two physicians and assessed using the five-grade Medical Research Council scale for muscle examination.The average of two physicians’ scores for the same patient is used as the ground truth.Using the feature set we developed,100% accuracy is achieved in classifying the patients and healthy subjects.For patients’ muscle strength,a mean absolute error of 0.143 and a maximum error of 0.395 are achieved using our feature set and the regression method;these values are closer to the ground truth than the scores from each physician (mean absolute error: 0.217,maximum error: 0.5).We thus validate the possibility of using such smart shoes to objectively and accurately evaluate the muscle strength of the lower limbs of stroke patients.

Keywords: stroke;machine learning;smart shoes;lower limbs’ muscle strength

1 Introduction

With approximately 12.2 million new cases worldwide in 2019,stroke has become one of the most prevalent diseases[1].It is difficult to cure,prone to recurrence,and slow to recover.Patients need regular clinical assessments to measure their rehabilitation progress after the acute phase of in-hospital treatment[2].One of the main purposes of rehabilitation is to improve independent mobility,where the assessment of lower limb muscle strength is essential[3].

Two types of methods are used for the muscle strength assessment: medical scales and equipment.The MRC scale(Medical Research Council scale for muscle examination)[4]is often used to grade the patients’ lower limb muscle strength.The patient was observed by a physician and given a score ranging from 0 to 5.The results might be biased,as different physicians might give different scores to the same patient.To digitalize the assessment,equipment can be used,such as a muscle testing dynamometer,an optical motion capture system or a force plate (additional details in Section 2.1).The whole process can be traced back,and subjectivity can be avoided.

We use the double-modality smart shoes as our measuring device,because they are small in size,easy to use,and comparatively inexpensive;thus,they have the potential for continuous assessment at home.To map the large amount of test data to the motor ability of the lower limb,we follow the general data mining process,first building the general feature set and then converting the feature set into the motor ability using the regression method.To obtain the ground truth,two physicians observed the tests and provided their independent scores using the MRC scale.

Our work validates the possibility of using dual-modality smart shoes to objectively evaluate the stroke patients’ lower limb muscle strength.The main contributions are as follow:

? We extend the 5 m walk test including the left and right turns,and demonstrate with the muscle strength’s regression results indicating that this extension is essential.

? We propose the dual-modality fusion features and prove the importance of these newly proposed features.

? We propose a two-step evaluation method.A subject is first classified as patient or healthy.If classified as a patient,the lower limb muscle strength is then calculated using our feature set and the regression method.Based on the data collected from 23 patients and 17 healthy subjects,100% classification accuracy is achieved.The regression result (mean absolute value error: 0.143,maximum error: 0.395) is even closer to the ground truth than the individual physical scores(mean absolute value error: 0.217,maximum error: 0.500).

2 Related work

2.1 Devices for assessing lower body mobility

Various devices have been developed for assessing motor abilities,including muscle strength meters,optical motion capture systems,force plates,inertial sensors,and smart shoes.Mentiplay et al.[5]demonstrated that handheld plyometrics can measure muscle strength reliably and effectively.However,these measurements require the assistance of other people,and only muscle strength at rest can be obtained.Rastegarpanah et al.[6]used the VICON MX force plate system with an optical motion capture system to analyze spatialtemporal gait parameters in healthy and stroke groups.Wang et al.[7]used F-scan insoles to study temporal changes in each gait phase over four weeks in hemiplegic patients.Although an optoelectronic system and a pressure carpet are commonly combined,these devices are expensive and require the professional operation,making long-term continuous monitoring difficult.

Wearable systems enable measurement at any place.For example,Yang et al.[8]attached multiple inertial measurement units (IMUs) to the legs to study walking speed on the hemiplegic side versus the non-hemiplegic side.Smart shoes are another option,as people normally walk with shoes on.Smart shoes equipped with IMU and/or pressure sensors can be cheap,easy to use,and suitable for monitoring individual gait patterns for a long time.In Table 1,we provide an overview of the existing smart shoes and the shoes we’ve developed and used in our research.Our system features 6-axial IMUs and a pressure-sensitive matrix covering the whole plantar,providing multi-modality and detailed data for gait monitoring.

Table 1. Smart shoe systems.

2.2 Experimental design

Experimental design plays an important role in data gathering.Current test protocols include the treadmill walking test[14],up-and-down stair test[15],long straight corridor walking test[8,16],and time up-and-go test[17].Walking on a treadmill allows strict control of the walking speed but requires an extra device. The walking up and down staircase test can better test the walking ability under extreme conditions,but also brings greater safety risks.The 5 m walk test is closely related to daily life,easy to perform,and has a decreased risk of safety.Because Bonnyaud et al.[17]reported that walking parameters during patient turning were related to lower limb motor ability,we extend the 5 m walk test to include the right and left turning.

2.3 Data processing algorithms

We take the early works using pressure sensors and IMU as the basis of our algorithms.Galli et al.[18]calculated the walking speed of both feet and the difference in acceleration and angular velocity at the joints to quantitatively compare the hemiplegic children’s walking ability.Wang et al.[19]calculated the acceleration parameters from the inertial sensors and proposed the gait normalization index to evaluate the lower limb motor ability.Echigoya et al.[20]calculated the changes in the center of plantar pressure and analyzed the important parameters that allow independent walking.Chisholm et al.[21]calculated the spatial-temporal parameters of the center of plantar pressure and analyzed the correlation with the severity of the sensorimotor impairment by comparing the hemiplegic side of the patient with the non-hemiplegic side.Chen et al.[14]calculated gait phase duration,walking speed,step length,and step width.We adopted several of these parameters.Because our hardware offers two sensing modalities,we propose the dual-modality fusion features,whose high importance will be discussed further in Section 5.3.

3 Experimental setup and acquisitions

Our smart shoe system is shown in Fig.1.Each shoe is equipped with a textile pressure sensing insole,covering the whole planta (horizontal 16× vertical 25) and two 6-axis IMUs (tri-axial accelerometer plus tri-axial gyroscope) located at the forefoot and hindfoot positions.The sample rate is 60 Hz,and the data are transmitted to the mobile phone via Bluetooth.

Fig.1. Our smart shoe prototype,plus a mobile phone for data storage.

Data acquisition was conducted in the First Affiliated Hospital of USTC,University of Science and Technology of China.The test subjects were instructed to walk at a selfselected speed after putting on both smart shoes,following three routines:

?Straight: go straight from the starting point for 5 m,turn around,walk straight back to the starting point,and stop.

?Right-turning: from the starting point along the edge of a square (all the sides are 5 m in length),turn right at the corners and stop at the starting point.

?Left-turning: similar to right-turning,only the turning direction is left at the corners.

In total,23 stroke patients and 17 healthy subjects were involved in this study (details in Table 2).The stroke patients have hemiplegia on one side and no significant symptoms on the other side,they are conscious and able to walk at least with assistance.The healthy subjects have no stroke symptoms,normal lower limb motor ability,and clear consciousness.For each patient,two clinically experienced physicians used the MRC scale to give individual evaluation scores on the lower limb muscle strength.The scores ranged from 3-to 5-.We map 3-,3,3+,4-,4,4+,5-to 2.67,3.00,3.33,3.67,4.00,4.33,4.67,respectively,and use the average scores of the two physicians as the ground truth.These experimental data were obtained by informed consent from all patients.All procedures were conformed to the National Institutes of Health (NIH) guidelines,and approved by the Ethics Committee of the First Affiliated Hospital of University of Science and Technology of China (USTC) (2023-KY-441).Besides,the authors have declared that the ethical guideline of the 1975 Declaration of Helsinki was rigorously adhered to in this study.This study is fully compliant with the regulation of relevant ethical about research involving in human participants.

Table 2. Clinical characteristics of the participants.

4 Data processing and feature extraction

We follow the classic data mining process.The data from the pressure insole and IMU of each shoe are first pre-processed,and then the data from both shoes are synchronized and fed into the general feature extraction flow (shown in Fig.2) to obtain frame features,step features,and whole features.These features are used later as input for classification and regression algorithms,as described in Section 5.

4.1 Pre-processing

The pressure data go through linear interpolation,upsampling and Gaussian smoothing,while the IMU data are only linear-interpolated.

Linear interpolation.Data could be lost in the wireless transmission.To ensure a constant sample rate,the data are repaired using linear interpolation.Given thatnsamples between timetxandtyare lost,from the data we obtain before and after this period datatx,dataty,theith lost data are recovered as:

Up-sampling.Up-sampled pressure image generates more accurate image features[22].The pressure images are thus upsampled to 32×100 using bilinear interpolation.

Gaussian smoothing.To obtain a smoother distribution,a 5×5Gaussian filter is applied on each pressure image.

4.2 Frame features extraction

The pressure and IMU data at each sample is defined as a“frame”,from which the frame features are extracted.

For the pressure data,Zhou et al.[23]proposed the use of the TPM feature set (743 features).Guo et al.[24]further expandedthe feature number to 1830.We adopt 23 of the frame features that are confirmed to be associated with lower limb motor ability of stroke patients[14,19-21],supplemented withFdes16(t),which measures the overall variance in the pressure images.These features are grouped into the biomechanical features (fromFdes1toFdes9) and the image features (fromFdes10toFdes24).

For the IMU data,we extracted from the raw signal 16 frame features (fromFdes25toFdes40).

The symbols used are defined in Table 3 and will be used throughout the paper.

Table 3. The summary of the symbols in the paper.

Biomechanical features:in total,9 features.

? Total force (Fdes1(t)):

? Area (Fdes2(t)) (the count of pixels that are above the δ):

where the mean and min operation take into account all thep(x,y,t)within the whole test.

? Average pressure (Fdes3(t)):

? The centre of mass(CoM)Fdes4(t)Fdes5(t)):

? The CoM’s speed,its magnitude (Fdes8(t)) and the projections on the horizontal (Fdes6(t)) and vertical direction(Fdes7(t)):

? The CoM’s moving direction (Fdes9(t)):

Image features:in total,15 features.

? Maximum,minimum,range,median,mean,and standard deviation of the pressure imagep(x,y,t) (fromFdes10(t) toFdes15(t)).Specially,“range” is defined as:

? Coefficient of variation (Fdes16(t)):

? Information entropy (Fdes17(t)).

? The image’s Hu Moments (fromFdes18(t) toFdes24(t))[25].

IMU features:in total,16 features.

? Acceleration of the forefoot IMU,its magnitude(Fdes28(t)) and the projections on thex,y,zaxis (in the local coordinate) (fromFdes25(t)toFdes27(t)):

? Angular velocity,its magnitude (Fdes32(t)) and the projections (Fdes29(t),Fdes30(t),andFdes31(t)):

? Acceleration (fromFdes33(t) toFdes36(t)) and angular velocity (fromFdes37(t) toFdes40(t)) of the hindfoot IMU,similar to that of the forefoot.

4.3 Step features extraction

Multiple frames are grouped into one step.As shown in Fig.3,one step is defined as the period from one heel strike to the next.The total force (Fdes1(t)) is used to divide the whole walking test into many steps.The values are grouped into three clusters using K-means algorithms.Defining the centroids of the these clusters as:x1,x2,andx3(in the ascending order),the threshold in Fig.3 is calculated as:

Fig.3. The relation between the gait cycle and the total pressure.

The grouped frame features are fed into the step features extraction layer and three types of step features are obtained:the phase duration features,the statistical features,and the dual-modality fusion features.

Phase duration features.Each step is divided into the unipedal gait phase and the bipedal gait phase.The duration features for thekth step are:

? The duration of the whole cycle (Spd1(k)),the stance phase (Spd2(k)),the swing phase (Spd3(k)),and the double support phase (Spd4(k)).

? Stance and swing phase ratios (Spd5(k),Spd6(k)).

? The ratios of the double support phase to the previous and the next stance phase (Spd7(k),Spd8(k)).

Statistical features.Taking the frame features within each step as the input,the statistical features of thekth step are calculated:

? Maximum,minimum,range,median,mean,standard deviation,coefficient of variation,and information entropy(fromSst1(k)toSst8(k));

? The number of peaks (Sst9(k)) and valleys (Sst10(k));

? Skewness (Sst11(k)):

? Kurtosis (S st12(k)):

Dual-modal fusion features.The hemiplegic patients are impaired in walking.Their performance under certain extreme conditions might be different from that of the healthy subjects,e.g.,the acceleration of the hemiplegic side in the left-right direction when the total force on the nonhemiplegic side is maximum (the foot strikes the ground or the toes go off the ground).To capture the subject’s performance under such conditions,we extend this method: when one frame parameter reaches its maximum or minimum value,the value of another frame parameter is taken as one dualmodality fusion feature.Given two frame parametersFdesi(t)andFdesj(t),i≠j,their dual-modality fusion features are defined as:

There are 40 frame features for left and right foot respectively,we thus get=3160 dual-modality fusion features for each step.

4.4 Whole features extraction

Taking the step features within one test,the whole features are their statistics and the bipedal symmetry coefficients.

Statistical features:the maximum,minimum,range,median,mean,standard deviation,coefficient of variation,information entropy,the number of peaks and valleys,skewness,and kurtosis of the step features (fromWst1toWst12).

Symmetry features:hemiplegic patients have different motor abilities on the hemiplegic side and the other side.Define the whole statistical features of the left foot and the right foot as,(i∈[1,12]),the symmetry coefficient (Wsyi) is:

To improve the effectiveness of whole features in classification and regression,the whole features are normalized using Z-score standardization.

whereuand σ are the mean and the standard variance ofx.

4.5 Feature selection

After the above workflow,270114 features are obtained,that are much more than the test subjects’ number.Feature selection is adopted to overcome the overfitting problem.

There are three types of feature selection methods: the filter,the wrapper,and the embedding method[26].We choose the embedding method because it combines the advantages of the other two methods: the short computation time of the filter method and the optimal feature set that can be generated by the wrapper method for a specific selector.

The embedding method include several steps: (ⅰ) specify a particular selector (e.g.,decision tree,support vector machine);(ⅱ) train the corresponding task (classification or regression) using the given feature set (in our case,the whole features);(ⅲ) obtain each feature’s importance to the result,(ⅳ) rank and select thekmost important features.

To obtain a stable ranking,the data are randomly divided into five portions,one portion is excluded each time,the rest four portions are used for training.To conduct a smaller selected feature set to mitigate the overfitting issues,each feature from the obtained feature set will be selected in a descending order based on the mean feature importance of the five portions and calculated the Pearson correlation coefficients with every feature included in the selected feature set.If all the correlation results are smaller than 0.9,the feature will be added to the selected set until the size of the selected feature set is up to 10.Finally,we get an efficient and well-performing feature set.

5 Results

A two-step approach is designed.First,the subject is classified as “patient” or “healthy”.If “patient”,the lower limb muscle strength is predicted.

5.1 Patient-healthy classification

The first step is to classify the subject.We study the different test combinations: straight (Str),right turning (RT),left turning (LT),RT+LT,andAll(Str+RT+LT).When multiple tests are included (viz.RT+LT,orAll),the probabilities in different tests are summed up.The class with the higher probability is taken as the classification result.

We treat every test of one person as a sample.The number of samples (Nsample) for a given test combination is then:

whereNp=23,Nh=17 is the number of the patients and the healthy subjects,andNtestis the number of test(s) in the combination.The sample numbers of the five test combinations are then 40,40,40,80,and 120.

We study the effects of using different test combinations,feature selectors (C-DT: decision tree,C-SVM: support vector machine,C-RF: random forest,and C-LR: logistic regression),and classifiers (DT: decision tree,SVM: support vector machine,KNN: K-nearest neighbor,RF: random forest,LR:logistic regression,and MLP: multi-layer perceptron).Fivefold cross-validation is adopted,with no subject crossover between the training and test sets,and the results are in Table 4.It can be see that using decision tree as the selector andsupport vector machine as the classifier,the classification accuracy reaches 100% when the data fromRTtests are used.

Table 4. Healthy-patient classification result (F1-Score),with test(s)combination,feature selector and classifier evaluated.The bolded part is the optimal result.

We then take the data from theRTcombination and study the effect of the sensing modalities,including: (ⅰ) IMU,as if there were only IMUs on the shoes;(ⅱ) Pressure,as if there were only the pressure insole;(ⅲ) IMU+Pressure,both sensing modalities are available,but the dual-modality fusion features are excluded,this is to test the effects of our newly proposed dual-modality fusion features;(ⅳ) All: all sensing modalities and features are included.Again multiple classifiers are tested,and the results are shown in Table 5.It can be seen that the classification of features calculated using only pressure is poor,and We can only use IMU to classify patients with healthy subjects.

Table 5. Sensing modalities evaluation (selector: C-DT).

5.2 Regression result for muscle strength estimation

After classifying the subjects,further regressions are performed to predict the patients’ lower limb muscle strength.

We study the effects of using different test combinations(Str,RT,LT,RT+LT,andAll),feature selectors (R-DT: decision tree,R-SVM: support vector machine,R-RF: random forest,and R-SR: stochastic gradient descent regression) and regressors (DT: decision tree,SVM: support vector machine,KNN: K-nearest neighbor,RF: random forest,GB: gradient boosting tree,and MLP: multi-layer perceptron).When there are multiple tests in the combination,the average of the predicted muscle strength from different tests is taken.Leaveone-out method is adopted,with no subject crossover between the training and test sets.The difference to the ground truth is measured by mean absolute error (MAE),root mean square error (RMSE),and maximum error (ME):

The results are shown in Table 6.It can be seen that using decision tree as the selector and random forest as the classifier,the best MAE is 0.138 when the testsRT+LTare used.We then fix the selector and compare the MAE’s and ME’s of different test combinations.The results are given in Table.7.It can be seen that althoughRT+LTis of the smallest MAE,its ME is much higher than that ofRT,whose MAE is the second optimum and just a little higher thanRT+LT.Because the estimated muscle strength should maintain not only a low overall prediction error,but also a low error for each patient,RTis taken as the optimum.WhyRTis the best test may be explained by the fact that 78% (18/23) of the patients are rightsided hemiplegic and thus show a greater difference during right turning.We then compare the algorithm’s performance with that of the two physicians.The results are shown in Fig.4.It can be seen that for this specific dataset,the algorithm performs even better in giving the muscle strength than the physicians as an individual (algorithm: MAE 0.143,RMSE 0.178,ME 0.395;physicians: MAE 0.217,RMSE 0.269,ME 0.500).

Fig.4. The predicted results based on left-turn and right-turn experiments,compared with the evaluation values of two doctors.

Table 6. Muscle strength regression results (MAE),with test(s) combination,feature selector,and classifier evaluated.The bolded part is the optimal result.

Table 7. Finding the optimum test combination for the regression task (RDT as the selector).The bolded part is the minimum error.

5.3 Features importance analysis

To better understand which features play more important role in estimating the muscle strength,we used the forward search algorithm.The initial feature set is empty,and the best result is obtained by adding one non-repeating optimal feature at a time.The names and results of the 10 features added to the feature set in order are shown in the Table 8.Three types of features are selected:Fdesi_Max(Min)Time_Fdesj_S1,Fdesi_Max(Min)Time_Fdesj_S1_Asymmetry,andFdesi_S1_S2.HereFdesiandFdesjdenote a frame feature(e.g.,acceleration in they-direction of the right forefoot(Fdes30(t)) is named RForeGyroY).S1andS2denote a statistical calculation (e.g.,mean,maximum).The three types are then: theS1operation onFdesj’s values at the maximum(minimum) time of valueFdesiwithin each step;the asymmetry coefficient ofFdesi_Max(Min)Time_Fdesj_S1between the left and right feet;and perform theS1operation onFdesi’s values within a step,then perform theS2operation on allFdesi_S1in the test.

Table 8. The names and results of the features added to the feature set in order.

It can be seen that eight of ten top features are the newly proposed dual-modality fusion features.This demonstrates that the great importance of our new features to the muscle strength estimation in Fig.5.When using 4-7 features,the results have leveled off,while at greater than 7 features,the error starts to increase and may start to overfit.This suggests that using 4-7 features is most appropriate in this experiment.

Fig.5. The experimental results using forward search algorithm.

6 Conclusions

In this paper,we demonstrate the possibility of using a pair of smart shoes to non-intrusively and objectively assess the hemiplegic patients’ lower limb muscle strength.In doing so,we designed the extended 5 m walk test protocol,also include the right and left turnings,which are proven to be useful in the assessment.We create a feature set to describe the characteristics of the walking,including the newly proposed dual-modality fusion features,which are also proven to be useful.Based on the data gathered from 23 patients and 17 healthy subjects,a 100% classification result of “healthypatient” is achieved.For estimation the muscle strength,regression methods are evaluated,the algorithm’s best performance is MAE 0.143,RMSE 0.178,and ME 0.395,both better than that of the physicians as an individual (MAE 0.217,RMSE 0.269,and ME 0.500).

There are still rooms for improvement,for example,finding out on which side is the hemiplegic leg,or segmenting the test data into going straight and turning.These operations can create new features that might be useful.Estimation of the other subjective medical assessment values (e.g.,NHISS scores) can also be carried out using the same dataset.Through long-term data acquisition at home,the rehabilitation results could be evaluated.We believe that smart shoes could become a useful tool in quantitative assessment of the lower limb muscle strength for hemiplegic patients.

Acknowledgements

This work was supported by the Fundamental Research Funds for the Central Universities (2150110020).

Conflict of interest

The authors declare that they have no conflict of interest.

Biographies

Huajun Longreceived his master’s degree from the University of Science and Technology of China in 2023.He is particularly interested in sports analysis based on smart shoes.

Jingyuan Chengis a Professor at the School of Computer Science and Technology,University of Science and Technology of China (USTC).She received her B.S.degree in Applied Physics and Ph.D.in Electronic Science and Technology from USTC in 2002 and 2007,respectively.Her research interests lie in human-centered large-scale sensing matrices and their applications,especially in the format of smart textiles.