Design of a DSLM-based cerebral palsy action rehabilitation training system①

Lu Zhenli (盧振利)②, Ma Zhipeng , Shen Xuanlin , Borovac Branislav , Liu Yan, Shan Changkao,Wang Hong , Zhao Yu , Roumiana Ilieva , Bojan Nemec , Marjan Mernik

(*School of Electrical Engineering and Automation, Changshu Institute of Technology, Changshu 215500, P.R.China) (* *China University of Mining and Technology, Xuzhou 221116, P.R.China) (* * *Department of Rehabilitation Medicine, Changshu No. 2 People Hospital, Changshu 215500, P.R.China) (* * * *Faculty of Technical Sciences, University of Novi Sad, 21000-Novi Sad, Serbia) (* * * * *School of Business, Changshu Institute of Technology, Changshu 215500, P.R.China) (* * * * * *Faculty of Management, Technical University of Sofia, Sofia 1000, Bulgaria) (* * * * * * *Department of Automatics, Biocybernetics and Robotics, Jo?ef Stefan Institute, Ljubljana1000, Slovenia) (* * * * * * * *University of Maribor, Maribor 2000, Slovenia)

Abstract

Key words: domain specified language mechanism(DSLM)， robot-assistant system， human-computer interaction (HCI)， cerebral palsy rehabilitation training

0 Introduction

Cerebral palsy (CP) is the most common and most serious disease in children, which can cause various obstacles in exercise, posture, language, hearing, vision and intelligence[1]. Since the exercise, language, and intelligence of the child can only reach the level of normal people through rehabilitation training, the earlier the child with cerebral palsy takes care intervention, the better the rehabilitation effect is.

After being introduced, Kinect has the advantages of high resolution and low cost of shooting depth maps, and has been applied in the field of sports rehabilitation[2-4].

Combined with Kinect design, a robot-assisted cerebral palsy rehabilitation training system is designed to assist rehabilitation therapists, simplify the traditional one-to-one heavy process, and inspire children to actively participate in training through interactive means, thus enhancing their communication ability, recognizing ability, limb function, hand-eye coordination and cognitive level.

Ding et al[5]proposed an upper limb virtual rehabilitation system based on Kinect interaction, developed the action library for patients with upper limb dysfunction, and set up the scene for different rehabilitation actions.

Based on Kinect a cerebral palsy rehabilitation training system was designed in the action interaction mode in Ref.[6]. Three groups of rehabilitation exercises were designed for the rehabilitation of patients with quadriplegia and cerebral palsy.

Based on the domain specified language mechanism(DSLM) design for the cerebral palsy rehabilitation robot assisted training system, this study uses Kinect to design the upper limb training system for patients with spastic cerebral palsy and improve the ability of patients to coordinate their limbs.

1 DSLM-based system architecture design

Domain specified language (DSL) is a computer programming language with limited expressiveness for a certain field[7]. In the early stage of system construction, due to the inconsistent language model of users and builders, it is difficult to collect requirements, which requires solving the problem through domain professional language.

Designing this system requires identifying the rehabilitation actions one by one and grouping the different actions into corresponding training combinations. In contact with the rehabilitation teacher for the corresponding training needs. The system is designed to understand different rehabilitation actions, perform different motion recognition for different actions, and plan a set of actions into a group of training to facilitate the training operation of the rehabilitation teacher. In this system, there are free activity training and upper limb training. The free activity training includes 4 movements of moving left and right, jumping up and down, and the upper limb training is a clap game consisting of 3 movements.The following mainly introduces the design of the upper limb training system.

2 Design of DSLM-based action training system

Aiming at the poor limb ability and uncoordinated movement of patients with spastic cerebral palsy, an action training system based on DSLM is designed. The color camera and infrared depth camera are used to collect the action position of the child, and draw the obtained data into thebody skeleton map to achieve bone tracking[8]. Next, the action is identified by geometric algorithm matching to determine whether the action is up to standard.During patient movement training, the system evaluates each movement and saves training records. After completion of the training, therapists record data to provide targeted supplementary motor motion correction and rehabilitation. The system architecture is shown in Fig.1. Fig.2 is the system physical map.

The experimental platform consists of 2 parts, the Kinect sensor and the PC. Kinect V2 is a 3D somatosensory camera with image recognition, instant motion capture, voice recognition, microphone input, and community interaction.

Fig.1 System architecture diagram

Fig.2 System physical map

The overall structure of Kinect V2 is shown in Fig.3. One is an RGB camera, which can capture up to 30 frames per second, and the other is a 3D device consisting of an infrared emitter and an infrared camera. The structured light depth sensor[9]is used to collect depth data and detect the relative position of the patient.Compared with Kinect V1, Kinect V2 has higher precision in color image and depth image. It can get more user postures and more joint points for users. Its detection range is wider and the angle is wider.

Fig.3 Kinect V2 hardware diagram

For the PC, the main software platform Visual Studio 2013 is the Windows platform application development environment; Kinect for Windows SDK 2.0, which enables developers to use Kinect as an input device and develop various applications; and OpenCV3.0 as a cross-platform computer vision library, running on the Windows operating system.

Skeletal motion recognition uses the skeletal image generated by Kinect. Kinect obtains the depth image of the subject through the infrared camera[10], and distinguishes the human body from the environment from the depth image. After completing the division of various parts of the body, according to the traced each joint point generates a human skeleton model containing 3-dimensional information of various joints of the body, thereby displaying a human skeleton image, as shown in Fig.4.

Fig.4 Human skeleton model building process

The skeletal model[11]is set up based on the characteristics of the human body. Kinect can track 25 bone joints, obtain bone frame data from the bone data stream, and set a smoothing function to correct the joint point jitter. After the conversion of world coordinates to image coordinates, it is displayed on the screen.

The recognition method of the action is mainly based on geometric algorithm matching. For the corresponding training action, the distance or angle between the joint points is calculated by acquiring the coordinate position information of one or more joint points in the space, and the threshold value is compared to quickly match the identified actions. The person faces Kinect, and the right direction is the positive direction of thex-axis, the upward direction is the positive direction of they-axis, and the backward direction is the positive direction of thez-axis.

Calculate the distance between joint pointA(x1,y1,z1) andB(x2,y2,z2) as follows:

When calculating the angle between 3 joint points, the space vector method can be used to calculate the joint angle. To use the vector, one need to map the Kinect coordinate system to the mathematical coordinate system. The coordinate pointsA(x1,y1,z1) andB(x2,y2,z2) that are not coincident on the Kinect coordinate system are converted to the mathematical coordinate system. On the composition vector AB, the conversion formula is as follows:

With the transformation formula, shoulder jointS(Sx,Sy,Sz), elbow jointE(Ex,Ey,Ez) and hand jointH(Hx,Hy,Hz) are used to calculate the angle of the elbow joint, one can directly use the space vector. The calculation of the angle betweenESandEHis as follows:

3 Training action recognition design

In order to train the upper limb mobility of the sputum type patient, combined with the training requirements of the therapist, the following 3 movements are designed, namely, the arms lift action, the arms crossed action, and the arms held flat action. The following 3 different actions are identified and designed.

1) Arm liftaction

The arm lift action is shown in Fig.5. When recognizing the arm lift action, the coordinate information of the 4 joint points of the left and right shoulders and the right and left hands is selected. When the arm is lifted, the distance between the shoulder joint and the hand joint on they-axis, that is, the height is set to be greater than the threshold value of 0.2 m, while the distance between the shoulder joint and the hand joint on thex-axis, that is, the level is less than the threshold value of 0.15 m.

Fig.5 Arm lift action

2) Arm crossed action

The arm crossed action is shown in Fig.6. When recognizing the arm crossed action, the coordinate information of the 6 joint points of the left and right shoulders, the left and right elbows, and the left and right hands is selected. When the upper limb is in a cross-lift state, the action can be judged according to the angle between the arm and the arm. When the angle is less than 80 °, and the distance between the shoulder joint, the hand joint and the elbow joint on the

Fig.6 Arm crossed action

z-axis, that is, the level is less than 0.05 m, the recognition action is judged, and the distance between the right and left hand joints needs to be limited to less than 0.05 m.

3) Arm held flat action

The arm held flat action is shown in Fig.7. When recognizing the arm held flat action, the coordinate information of the 6 joint points of the left and right shoulders, the left and right elbows, and the left and right hands is also selected. When the arm is lifted in front, the distance between the shoulder joint and the hand joint on thez-axis, that is, the depth is set to be greater than the threshold value of 0.15 m, and the distance between the shoulder joint and the hand joint on they-axis, that is, the height is less than the threshold value of 0.05 m, and the elbow joint angle is greater than 145 ° to determine the double upper limb lifting action.

Fig.7 Arm held flat action

4 Human-computer interaction system design

The human-computer interaction (HCI) system is developed based on the MFM function module of Visual Studio in DSLM. In the test process, it assists in judging whether the subject’s upper limb training action is correct or not, and records the patient’s training information. Fig.8 is the human-computer interaction interface.

Fig.8 Human-computer interaction interface

Before starting the training, one need to see the address where the bone frame map is saved. When the subject is in the proper position, choose upper limb training or mobile training. The depth frame and the skeleton frame display the training image of the subject in real time. When the motion is recognized, the motion skeleton map is automatically saved to the selected one. In the folder, and in the edit box, the action is successful and the skeleton frame is saved. When a group of training is completed and another group of action training is required, the dot refresh will clear the edit box to facilitate data reading and recording.

After the system design is completed, test the system. The following 3 tests are performed separately. Fig.9 and Fig.10 are test results of the arms lift action.

Fig.9 Arm flat physical map

Fig.10 Arm lift action test

Fig.11 and Fig.12 are test results of the arms crossed action.

Fig.11 Arm crossed physical map

Fig.12 Arm crossed action test

Fig.13 and Fig.14 are test results of the arms held flat action.

Fig.13 Arm held flat physical map

Fig.14 Arm held flat action test

Fig.15 shows the skeleton frame saved during the test action.

Fig.15 Skeleton frame saved during the test action

5 System performance evaluation

According to investigations and studies, patients with spastic cerebral palsy have decreased limb flexibility, increased joint stiffness, and their muscle strength cannot be fully exerted. Excessive flexor reflexes often occur, mainly due to impaired basic functions such as reaching, grasping, and releasing in daily life[12,13]. Designing a clap game consisting of these 3 movements, the cerebral palsy of the child is active in the shoulder, elbow, wrist and metacarpophalangeal joints during exercise[14], regulating muscle contraction and improving arm coordination and flexibility.

When training a group of actions, the action 10 group is a cycle, and the training effect is judged according to the time required to complete a set of actions, and the time is calculated according to the saved number of bone frame frames.

CaseA male patient is 7 years old, with a clinical diagnosis of spastic cerebral palsy. Symptoms mainly include difficulty in movement of the hand limbs, poor grip ability, good communication ability, and long concentration of attention.

A total of 10 tests were performed in groups of 3 actions, and the results of each test are shown in Table 1.

Table 1 Robot parameters

According to the time of each recognition of the training action and the time required to complete a set of actions, the time change diagram required for the 3 actions of the cerebral palsy in training 10 groups can be drawn as shown in Fig.16, Fig.17 and Fig.18.

Fig.16 Time change diagram of arm lift

Fig.17 Time change diagram of arm crossed

Fig.18 Time change diagram of arm held flat

According to the above 3 charts, it can be seen that in the 10 tests of the 3 training exercises of the upper limb training, as the test progresses, the time of completing the training action gradually decreases and tends to be stable, and the rehabilitation training effect is reflected.

The time variation curve of the 3 training actions is shown in Fig.19. It can be seen that the arm held flat training takes up more time than the other 2 actions, and the results can help the therapist adjust the different movements of the child and amount of training according to the information displayed in the chart.

Fig.19 Action training time change graph

When performing training in which one set of 3 upper limbs is performed, the effect can also be judged based on a change in the completion time of a set of actions. The group motion training time change graph is shown in Fig.20.

According to the chart information, when the child performs the upper limb motion training, the time required for a group of movements decreases with the increase of the number of trainings. The system plays an active and guiding role in the upper limb training of the child.

Fig.20 Group action training time change chart

6 Discussion

The application of the cerebral palsy rehabilitation training system is expected to provide a basis for rehabilitation training. There are 3 typical phenomena in rehabilitation training: the first category is the training has a high correct rate, which means this rehabilitation training is useless; the second category, the correct rate has been very low, indicating that this rehabilitation training has no meaning for this individual. It is recommended to add other auxiliary training; the third category is continuous improvement, and such patients need long-term rehabilitation training. In order to understand whether such rehabilitation training is effective, it is necessary to record 30 daily movements, record the change of daily training movement time, train patients for one month in one year period, compare the change of training movement time in the previous and subsequent cycles, and judge whether to continue this type of training and whether to increase or decrease the amount of training for a certain action.

7 Conclusion

This paper designs a DSLM-based cerebral palsy rehabilitation robot assisted training system for the upper limb rehabilitation training of children with cerebral palsy, combined with Kinect’s bone recognition and interactive interface, to record the data of children with cerebral palsy undergoing rehabilitation training in real time and conduct relevant evaluations and analyses. The system enhances the fun and interactivity of the cerebral palsy rehabilitation training system and provides new technology and support for robot-assisted cerebral palsy rehabilitation training.