Delphi and Analytic hierarchy process for the construction of a risk assessment index system for post-stroke shoulder-hand syndrome

Yu-Huan Zhang, Chun-Bo Fan, Yong-Mei Luo, Dong Chen, Chun-Xiao Yang, Dong Pang*

1 Department of Neurology, The Second Hospital of Harbin Medical University, Heilongjiang, 150001, China.

2 Department of Neurology, Peking University Third Hospital, Beijing, 100191, China.

Abstract Background: Shoulder‐hand syndrome (SHS) is one of the common complications after stroke, which is difficult to cure once it occurs. Early risk identification is an effective measure to prevent and treat SHS, but there is no effective tool to assess the risk assessment of SHS. Objective: To develop a validated tool to assess the risk of SHS occurrence after stroke. Methods: This was an observational study with a 3‐step process: (1) Literature review to establish initial indicators; (2) Application of a modified Delphi method for two rounds of correspondence, with final indicators obtained by modifying each round based on expert opinion; (3) Application of hierarchical analysis to determine the weights of each indicator. Results: The initial literature review constructed 4 primary indicators and 24 secondary indicators; after the first round of Delphi, a total of 10 secondary indicators were deleted and 6 secondary indicators were added, and the final indicators included 3 primary indicators and 15 secondary indicators, and in the second round, consensus was reached; by AHP analysis, the highest weight was given to existing risk factors (0.5584), followed by relevant medical history (0.3196) ; lastly, demographic factors (0.1220), and the scores of other secondary indicators met the requirements. Conclusion: This study establishes and constructs a post‐stroke SHS risk assessment tool, which provides a basis for early identification of SHS and early intervention. Meanwhile, this study provides a methodological reference for the development of other indicators sets.

Keywords: Analytic hierarchy process, Delphi method, Stroke patients, Shoulder‐hand syndrome, Shoulder‐hand syndrome, Risk assessment

Background

Shoulder‐Hands Syndrome (SHS) is a common complication in stroke patients [1, 2], with a prevalence ranging from 12% to 49% and an incidence of about 70% [3, 4], and is a major cause of disability in stroke patients [5]. However, because there is no effective specific treatment for shoulder‐hand syndrome [6], once SHS occurs, patients are highly likely to have irreversible permanent functional impairment, which seriously affects the rehabilitation of upper limb function and quality of life of stroke patients. Therefore, early detection, accurate determination of risk factors for the occurrence of shoulder‐hand syndrome after stroke, and timely provision of effective interventions are of great importance in reducing the incidence and improving the prognosis of patients with shoulder‐hand syndrome.However, there is a lack of clinical risk assessment tools for post‐stroke SHS.

Delphi, also known as expert investigation method, is essentially a feedback anonymous letter inquiry method, which has been applied to business, education, health care and other fields [7]. In recent years,Delphi is often used in the construction of risk assessment index system [8]. It has been applied to the establishment of risk assessment index system for infectious diseases, infectious diseases and AIDS in natural disasters at frontier ports. Analytic hierarchy process (AHP) is mostly used to solve the problem of multi‐objective evaluation [9]. It calculates the hierarchical ranking through the fuzzy quantitative method of qualitative indicators, selects the importance of influencing factors through pairwise comparison, and establishes a hierarchical structure model. The advantage of AHP is to reduce the randomness of indicator weight setting, and improve the validity, reliability and standardization of early warning evaluation Scientific and authenticity [10]. In practice, AHP is often combined with Delphi method to determine the weight of evaluation indicators.

In this study, based on literature review and group discussion, the Delphi and AHP were used to develop a risk assessment scale for post‐stroke shoulder‐hand syndrome, with the aim of providing a basis for assessment and intervention by health care professionals. The aim is to provide a reference for healthcare professionals to assess and intervene.

Methodology

Establishment of the subject group

The subject group of this study consisted of six members, including one deputy chief nurse practitioner, who was engaged in nursing management; three nurse practitioners in charge, who were engaged in clinical nursing in neurology or rehabilitation; and two postgraduate students, who were engaged in clinical nursing in neurology. The subject group was responsible for the construction of the index entry pool, the screening of correspondence experts, the development and distribution of expert consultation questionnaires,and the aggregation and statistical analysis of the consultation results.

Literature review and index screening

The risk indicators for the development of shoulder‐hand syndrome after stroke were collected through an extensive review of domestic and international literature. English databases including PubMed,Cochrane Library, Elsevier, Wiley, etc. and Chinese databases CNKI,VIP, WanFang, CBM were searched for the terms “cerebral apoplexy/stoke”, “reflex sympathetic”, “reflex sympathetic”, “reflex sympathetic dystrophy (RSD)/complex regional pain syndrome type 1(CRPS‐1)/shoulder‐hand syndrome (SHS)” and “Risk factors/influencing factors/potential factors”. The basic framework and entries of the high risk of shoulder‐hand syndrome after stroke were initially developed after several discussions in the subject group. Five experts from tertiary care hospitals in Heilongjiang province, China,were invited to conduct expert group meetings for the questionnaire,and after 2 rounds of discussion, the initial pool of 20 entries in 3 dimensions, including A‐demographic factors (4 indicators),B‐previous relevant medical history (11 indicators), and C‐existing risk factors (5 indicators), was finally formed. The details are shown in Table 1.

Expert consultation

Selection of experts for consultation The Delphi method of consulting experts, also known as the expert correspondence survey method, should have authority and representativeness in the selection of experts. The selection criteria for the experts were: specializing in neurology and rehabilitation, with 10 years or more of experience in the treatment or care of shoulder‐hand syndrome diseases; having a bachelor's degree or above; having a middle‐level or above title; and having a certain degree of motivation. There is no set standard for the sample size of the expert group [18], and this study followed the principle of informed consent and voluntariness, and a convenience sampling method was used to invite 23 experts from Beijing,Heilongjiang, Shanxi, Jiangsu, Xinjiang Autonomous Region, Hebei,Sichuan, and Hunan for correspondence. Among them, 2 were male and 21 were female, aged 36‐60 (45.35 ± 5.73); 7 had senior titles,11 had associate senior titles, and 5 had intermediate titles; 16 had master's degrees or above and 7 had bachelor's degrees; 1 was vice president, 1 was deputy chief physician, 1 was director of rehabilitation, 3 were deputy directors, 1 was director and 2 were deputy directors of neurology, 14 were chief nurses, and 2 were nurses There were 2 nurses with 10 to 15 years of experience, 6 with 16 to 20 years of experience in the department, and 15 with more than 20 years of experience; 19 (82.61%) in the field of neurology and 4(17.39%) in the field of rehabilitation.

Delphi Expert Consultation Consultation was conducted by e‐mail or in person and was collected within 1 week after distribution. This consultation questionnaire includes 3 parts: (1) Questionnaire instructions. It introduces the background of the study, the purpose and significance of the study; (2) The questionnaire of experts'personal information, the self‐assessment form of judgment basis and its influence degree (Ca) and familiarity degree (Cs). The basis of judgment and its degree of influence are divided into coefficients according to theoretical analysis, practical experience, domestic and foreign information, and intuitive choice; the degree of familiarity is divided into coefficients according to very familiar, more familiar,generally familiar, less familiar, and unfamiliar [19]; (3) The body of the questionnaire. The initially formulated pool of entries was presented in the form of an advisory form, and the Likert 5‐point scale was used to divide the importance of each indicator into five levels,namely, very important (5 points), relatively important (4 points),generally important (3 points), not too important (2 points), and unimportant (1 point). On the right side and bottom of the consultation form, there are columns for revision, deletion and addition, and experts can give their opinions accordingly and indicate the importance of the proposed addition. The expert's opinion is the main focus, and the subject group collates and summarizes the results of the expert consultation. The subjective conditions for the selection of entries were that 30% or more of the experts suggested deleting the entries, and the objective conditions were that the entries with a mean index importance score ＜ 3.5 or a coefficient of variation (CV) ≥0.25 were deleted [20]. Experts' suggestions for modifying or adding entries will be included in the next round of consultation to solicit other experts' opinions. The revised questionnaire for the 2nd round of consultation will still be distributed by email or in person, and will be collected within 1 week after distribution. After the 2nd round of consultation, the experts' opinions basically converged and the consultation was ended. The interval between the 2 rounds of consultation in this study was 1 month.

Determining index weights using AHP method

Construction of hierarchical model The primary and secondary indicators of the post‐stroke SHS risk assessment scale were obtained by Delphi method to establish the primary risk assessment system.

Table 1 Summary of sources of shoulder-hand syndrome risk assessment indicators after initial stroke

Establishment of judgment matrix and index system weights Using the "Post‐stroke shoulder‐hand syndrome high‐risk assessment scale" as the decision target, two comparisons were made between the indicators at the criterion and program levels, and the weights of the indicators were assigned and the matrix was calculated. Consistency test (CR), when CR ≤ 0.10 indicates that the AHP method yields results without thought confusion [21].

Quality control

Two personnel were responsible for literature review to reduce omissions in the initial construction of the scale; experts were selected in strict accordance with the inclusion criteria; questionnaires were carefully verified when they were collected, and if there were omissions in the entries, they were returned to the experts in a timely manner and returned after completion; experts were reminded to return the questionnaires on time to ensure that the sessions were conducted smoothly; double data entry was used and cross‐checked to ensure correct data entry.

Statistical methods

SPSS23.0 and YaaHP.1 were applied for data entry and statistical analysis. Count data were described by frequency and composition ratio, and measurement data were described by mean, standard deviation, and coefficient of variation (CV). The positive degree of experts was expressed by the return rate of the questionnaire; the degree of authority was expressed by the expert authority coefficient(Cr); the degree of coordination was expressed by Kendall's coordination coefficient (Kendall's W); the degree of concentration of experts' opinions was expressed by the mean and CV of the importance assignment of each indicator; the AHP method was applied to determine the weight of each entry [22]. The test level was α = 0.05.

Results

Positive level of experts

In this study, 2 rounds of consultation were conducted, and 25 questionnaires were returned from 2 rounds of correspondence, with a return rate of 92.0%.

The degree of authority of experts

The degree of authority of experts is expressed by Cr, which is determined by 2 factors, namely, the basis of experts' judgment on the issue (Ca) and the familiarity of experts with the index (Cs). The Cr for the 1st round of this study was 0.912 (Ca = 0.902 and Cs = 0.922)and the Cr for the 2nd round was 0.920 (Ca = 0.943 and Cs = 0.896).Its value is ＞ 0.70, indicating a high degree of authority of the experts in this study.

Degree of coordination and concentration of expert opinions

The degree of coordination of expert opinions is reflected by the Kendall's coordination coefficient (Kendall's W), and the value of Kendall's W ranges from 0 to 1. The larger the value is, the better the degree of coordination of experts is, and generally the values are from 0.3 to 0.5 [23]. The degree of concentration of experts' opinions was expressed by the mean and CV of the importance assignment of the entries. The degree of coordination and concentration of opinions of the 2 rounds of expert consultation in this study are shown in Table 2.

Consultation results

In the 1st round of expert consultation, the entries deleted in one round of correspondence based on the screening indicators with mean importance assignment ＜ 3.5 or coefficient of variation ＞ 0.25 were:A3 (smoking 3.35 ± 1.03), A4 (tobacco and alcohol 3.13 ± 0.92), B5(hypertension 3.78 ± 1.04), B6 (preterm contraction 3.17 ± 1.03),B7 (myocardial hypertrophy 3.13 ± 1.06), B8 (cardiac infarction 3.65± 1.19), B9 (pulmonary disease 3.57 ± 1.08), B10 (diabetes mellitus 3.91 ± 0.90), B11 (hypoproteinemia 3.61 ± 1.12), B12 hypocalcemia(3.52 ± 0.32), and B14 (defective para‐visual field examination 3.30± 1.30). The mean importance assignments and CVs of all other entries met the requirements and were retained. In accordance with the experts' suggestion, “past medical history” was modified to“relevant medical history”; the entries of cervical spondylosis, upper limb trauma, neck and shoulder surgery, mastectomy on the affected side, excessive stretching of the affected limb, and patient's knowledge and behavior of prevention were added according to the experts'opinion, and the results of the 2 rounds of expert consultation were summarized. After discussion and statistical analysis, the first draft of the scale was finally formed, which consisted of 15 secondary indicators for 3 primary indicators, including demographic factors (2 indicators), relevant medical history (7 indicators) and existing risk factors (6 indicators), as shown in Table 3.

Table 2 The degree of concentration of expert opinion, the degree of coordination statistics table

Table 3 Risk assessment indicators for SHS after stroke and their weights (Round 2)

Weighting results of the indicators

The first ranked indicator was “existing risk factors”, the second ranked indicator was "relevant medical history", and the last ranked indicator was "demographic factors"; in terms of sources, 6 (40.00%)As for the sources, 6 (40.00%) of the indicators were from prospective studies, 3 (20.00%) were self‐made indicators, and 6 (40.00%) were from expert recommendations, and the scores of all indicators met the requirements. The specific results are shown in Table 3.

Discussion

Reliability and scientific validity of the study results

The Delphi method is based on a structured procedure in which experts' opinions on assessment scales are repeatedly solicited,summarized, and modified until the results are generally agreed upon[24, 25]. The AHP method is a systematic analysis and comprehensive decision making founded by Professor Thomas in the late 20th century, and is now widely used in outcome evaluation and situation prediction [26]. In this study, the consulting experts were invited from neurology and rehabilitation departments, covering various directions of clinical medicine and clinical nursing, which is more representative. the positive degree of experts in the two rounds of consultation was over 90%, and the authority coefficient of experts was higher than 0.70, indicating that the experts consulted had a high degree of authority, were more concerned about this study, and had a good degree of cooperation [27]. The Kendall's W of the second round of expert consultation was 0.314, in which the primary index was 0.439 and the secondary index was 0.439, indicating that there was more agreement among the experts. compared with the results of the concentration of expert opinions in the two rounds of consultation, the assigned mean was higher and the CV was lower in the second round than in the first round, indicating that the concentration of expert opinions had increased and tended to be the concentration of experts'opinions increased and tended to be concentrated [28]. The weights of each dimension in the scale are close to each other, and the results of the consistency test are ＜ 0.10, which indicates that each dimension has the same importance, and the weights of each item are consistent with the expert evaluation results, and there is no confusion among the evaluators, so the results are more reliable. This study combined the Delphi and AHP methods to reduce the bias brought about by the subjective opinions of consulting experts in the study, and verified the correspondence between the weights of attributes obtained for each entry of each dimension and its degree of importance [16], and the results of the study were reliable and scientific.

Content analysis of the post-stroke SHS risk assessment scale

In this study, a post‐stroke SHS risk assessment scale was initially developed through literature analysis, group discussion, Delphi correspondence, and hierarchical analysis, which included 3 primary indicators and 15 secondary indicators. Among the primary indicators,“existing risk factors” had the highest weight of 0.5584, which refers to the existing probability factors leading to the occurrence of SHS in stroke patients, followed by “related medical history” with a weight of 0.3196, which refers to the existing or previous related medical history leading to the occurrence of SHS in stroke patients. The lowest weight of 0.1220 was given to “demographic factors”, which refers to the demographic factors associated with the occurrence of SHS in stroke patients. In addition, it should be noted that in the secondary indexes, “shoulder hemi (total) dislocation” is based on the finger palpation method [29]: the distance between the acromion and the humeral head of the affected shoulder joint is less than 1/2 cross finger as 1 degree; more than 1/2 cross finger and less than 1 cross finger as II degree; more than 1 cross finger as III degree; and normal as 0 degree. “Shoulder joint protective device” means any type of device that provides protection to the shoulder joint. "Excessive stretching of the affected limb" refers to the presence of the affected limb due to improper placement of the patient's limb, and “patient's preventive knowledge and behavior” is a short question‐and‐answer format in which the evaluator judges the patient's preventive knowledge and behavior related to shoulder‐hand syndrome through a combination of questioning and observation. behavior.

Limitations and outlook

The weighting of the indicators in the development of this assessment scale is mainly based on the AHP method, which may have some subjectivity, and different statistical methods can be tried to assign weights to the entries in future studies. This scale is still in the initial development stage and no clinical empirical study has been conducted. Therefore, the next step in this study will be to conduct a clinical study of the scale to evaluate its reliability and validity, to examine the sensitivity and specificity in the assessment process of stroke patients, and to clarify the scoring and stratification boundaries so that it can become a reliable assessment tool for stroke SHS prediction.

Conclusion

This study developed a post‐stroke SHS risk assessment tool consisting of 3 primary indicators and 15 secondary indicators, each weighted.Most of the indicators were set according to guidelines, so that other countries or regions can use or modify the indicators in their practice.In addition, this set of indicators provides a methodological reference for the development of other indicators sets.