Development and validation of the Nurse’s Workplace Mental Health Questionnaire

Jing Li ,Rong Zhng ,Mioxun Hong ,Nnyn Li

a Nursing Department,Guangdong Provincial People’s Hospital,Guangdong Academy of Medical Sciences,Guangdong,China

b Department of Discipline Inspection and Supervision,Guangdong Provincial People’s Hospital,Guangdong Academy of Medical Sciences,Guangdong,China

c Shantou University Medical College,Guangdong,China

d School of Nursing,Southern Medical University,Guangdong,China

Keywords:Mental health Nurses Questionnaire Reliability Validity Workplace

ABSTRACT Objectives: This study aimed to develop and validate the Nurse’s Workplace Mental Health Questionnaire(NWMHQ).Methods:The questionnaire was developed based on the two continua model of mental illness and health proposed by Keyes.The initial questionnaire was generated through literature review,two rounds of Delphi expert consultation,followed by a pilot survey.Finally,the reliability and validity of the questionnaire were validated through an online survey of 2,815 registered nurses selected from the public hospitals in 11 provinces from June to July 2020.Results:The item-content validity index(I-CVI)of the questionnaire ranged from 0.750 to 1.000 and the average scale-level content validity index(S-CVI/Ave)was 0.906.Cronbach’s α coefficient was 0.948 and test-retest reliability was 0.850.The self-rating depression scale score was negatively related to the NWMHQ score(r=-0.664,P ＜0.01).The exploratory factor analysis(EFA)yielded six factors(emotional status,psychological security,positive relationship,resilience,self-efficacy,and subjective well-being),consisting of 32 items.The cumulative variance contribution rate was 65.58%.Confirmatory factor analysis (CFA) showed an acceptable fit.Conclusion:The NWMHQ developed in this study showed good reliability and validity.This questionnaire may help assess the mental health status of nurses and help nursing managers to develop appropriate targeted psychological interventions.

What is known?

· The nursing occupational environment has a considerable impact on the mental health of nurses.It is,therefore,necessary to consider occupational environment factors when evaluating the mental health of nurses.

· Few psychometric instruments have been developed to assess mental health in nurses’ occupational environment.

What is new?

· This study developed the Nurse’s Workplace Mental Health Questionnaire(NWMHQ)and verified its reliability and validity.

· The NWMHQ can help nurses evaluate their psychological condition in the nursing environment,and guide nurses seeking targeted psychological interventions.

1.Introduction

The importance of optimal well-being and mental health among nurses has received increasing attention and debate in recent years,across both the academic and public discourse.It is known that mental health problems can result from exposure to unhealthy working conditions [1].However,nurses are more vulnerable to mental health problems than other workers due to specific workplace factors,such as greater emotional demands,excessive workload,and workplace violence[1,2].These stresses increased sharply during the COVID-19 pandemic.Therefore,the characteristics of the occupational environment need to be taken into account when assessing workplace mental health.

As a profession focused on preventing diseases and improving health,nurses engage in close contact with patients every day for long periods,creating considerable risk to their own lives during the COVID-19 pandemic [3].Nurses were the healthcare workers most frequently infected with COVID-19 [4,5].Workplace violence is another common phenomenon with detrimental consequences to the safety and health of nurses,with 64.7% of nurses experiencing violent incidents [6].Overall,28% of healthcare workers who experience physical violence developed post-traumatic stress disorder (PTSD) [7].

The International Council of Nurses(ICN)has warned that there could be a shortage of 13 million nurses by 2030 [8],with this shortage further increasing the pressure on nurses.Overwork is the primary source of stress among nurses.A survey of 3.9 million registered nurses in the US found that 68%of nurses who left their jobs reported stress as a contributing factor.Other stressors include the high-level skills,teamwork,and emotional labor required,as well as nurse-patient relationships [9,10].Long-term night shift rotation is another leading cause of stress as it interferes with biological rhythms,resulting in sleep disorders and an increased risk of breast cancer and coronary heart disease [11].The occupational environment must be considered when assessing the mental health of nurses,creating the need for a specific tool for measuring the mental health of nurses that is different from that of the general population.This requires the development of a questionnaire designed to assess the mental health of nurses specifically.

Mental health is not just the absence of mental illness but a consideration of actual well-being according to the WHO’s definition of mental health [12].A highly stressful work environment makes nurses more prone to negative emotions such as depression and anxiety,but this does not mean they are mentally ill.We also need to pay attention to positive emotions and the cultivation of positive functioning in the nursing environment,and to create a positive psychological environment rather than dealing with negative emotions only.This suggests that tools based on psychiatric diagnostic protocols are not sufficient for assessing the mental state of nurses in the workplace,despite their widespread use over the past decade [13,14].

Well-being-related scales based on positive psychology are widely used to assess the mental health of nurses[15].Commonly used scales include the Memorial University of Newfoundland Scale of Happiness,the General Well-Being Schedule,the General Self-Efficacy Scale,and the Connor-Davidson Resilience Scale,for example.There is also a trend to develop more targeted measurements to assess the well-being of specific populations.Rosen et al.[16] developed a Female Sexual Well-Being (FSWB) scale to assess sexual well-being in sexually functional women.Giles et al.[17]proposed that it is necessary to develop a sport-specific measure of well-being to support the health and performance of athletes.Jarden et al.[18] proposed a conceptual model for well-being among working nurses,which includes healthy,authentic,meaningful,connected,and innovative factors,while Chung et al.[19] developed a conceptual model of nurses’ well-being that consists of“contentment” and “joyfulness”.

To mitigate these damaging effects and improve the sustainability of the long-term nursing workforce,there is an urgent need to develop tools to measure mental health in the nursing environment.The objective of this study was to develop a workplace mental health questionnaire for nurses,which was expected provide useful data for understanding nurses’ workplace mental health in the clinical setting,and facilitate the development of strategies for efficiently managing workplace mental health.

2.Methods

2.1.Theoretical framework

The theoretical framework of this study is based on the two continua model of mental illness and health proposed by Keyes[20].This framework states that complete mental health should consist of two dimensions: the mental illness continuum and the mental health continuum [21].The mental illness continuum is assessed using the mental illness diagnostic scale,while the mental health continuum includes emotional well-being,psychological well-being,and social well-being.Emotional well-being consists of life satisfaction and the balance of positive to negative affect.Psychological well-being consists of self-acceptance,positive relations with others,personal growth,purpose in life,environmental mastery,and autonomy.Social well-being consists of social coherence,social actualization,social integration,social acceptance,and social contribution [22].

Based on the theory of the two continua model of mental illness and health,this study focused on the psychological impact of the work environment on nurses,as well as the psychological coping abilities nurses need to deal with these challenges.

2.2.Development of the first item pool

The item pool was created in the following ways.First,based on a comprehensive review of the literature and pre-existing scales,initial items were generated.Relevant literature on mental health problems,the negative emotions that nurses are susceptible to,and theoretical models of mental health among nurses were highlighted.Some of the entries were based on the nurse work wellbeing model proposed by Jarden et al.[18] and the nurses’ wellbeing model proposed by Chung et al.[19].Second,informal interviews were conducted with three nurses to talk about the standards of nurses’ mental health,elements of nurses’ mental health,and new career paths.Distilled interview results served as the source for the items.

With a total of 164 items selected for the pool,an expert panel including professionals in clinical nursing,nursing management,and clinical psychology performed an initial item selection.A 5-point Likert scale was used for all items.Those items in the item pool were removed as follows: 1) items with the same or similar meaning;2)items that did not conform to the working conditions of nurses (judged by the clinical work experience of the project team and literature review conducted by the researchers).After item selection,an initial version of the questionnaire was formed,which contained six dimensions and 42 items.

2.3.Primary questionnaire development

We used two rounds of a modified Delphi method for scale construction and content validity testing.A pre-survey was used for item analysis and face validity.

2.3.1.The Delphi surveys

While the ideal number of participants varies according to the scope of the problem and the available resources,it has been recommended that more than 12 experts in different fields should be included in a Delphi survey[23].We included 17 experts in the first Delphi round and 16 experts in the second round (one expert did not reply to our reminder message on time).Thirteen (76.4%)experts were women,and the mean age of the experts was 49 years(range 37-64 years).The average work experience of the nurses was 28 years;four(23.5%)had a master’s degree,four(23.5%)had a doctoral degree,and 15 (88.2%) had senior titles.Five (30.0%) experts were from clinical medicine and 12(70.5%)were from clinical nursing.We then made the following adjustments: 1) two items with coefficient of variation(CV)＞0.250 and item-content validity index (I-CVI) ＜0.780 were removed;2) we revised an item according to the experts’ advice;3) three items were revised for clearing expression.After the second round of the Delphi method,a second version of the NWMHQ was developed,which contained six dimensions and 39 items.

2.3.2.Pilot survey

The questionnaire was tested among a smaller sample of respondents but with a sample size sufficient to perform item analyses (N ＜ 100) [24].A pilot survey was conducted by the convenience sampling method among 56 nurses with a qualification certificate.In the item analysis,item-total correlations had to be between 0.40 and 0.85 [25].Fifty-six nurses participated in the pilot survey,all of whom were currently working at the Guangdong Provincial People’s Hospital,China.In total,87.5% were women,44.6% were aged 20-30 years,87% had a bachelor’s degree,71.4%were married,53.6% had more than one child,83.9% were senior nurses,51.8% worked in intensive care units,and 51.8% worked on more than four-night shifts per month.The survey completion time was 6.47 ± 5.14 min.In the item analysis,item-total correlations were 0.362-0.824 (P ＜0.01).For item 19 “I am satisfied with my partner”(scored as“strongly disagree”if there was no partner),the item-total correlation was 0.283,which did not meet the criteria.Item 19 was therefore deleted.The third version of the NWMHQ contained six dimensions and 38 items.

2.4.Formal investigation

2.4.1.Participants

A sample size of 200 or above is considered a reasonable number of participants to assess the dimensionality of a scale via factor analysis [26].An online survey was conducted to test the reliability and validity of the NWMHQ.First-line clinical nurses from all parts of China were recruited through the Internet using an online questionnaire survey platform in June 2020.Inclusion criteria were as follows: 1) clinical nurse with a certificate of qualification;2) agreement to participate in the survey.

2.4.2.Ethical considerations

The study was approved by the research ethics committee of Guangdong Provincial People’s Hospital,Guangdong Academy of Medical Sciences (No.GDREC2019363H).Informed consent was provided by all participants involved in the study.

2.4.3.Instruments

The General Information Questionnaire contains 11 items relating to basic demographic data(gender,age,marital status,title,position,educational level,and night shift frequency).

The Self-Rating Depression Scale(SDS)is a 20-item Likert scale developed by Zung [27],with raw scores that range from 20 to 80 that are converted to index scores by dividing the sum of the raw scores by 80,and multiplying by 100.An SDS Index score of 50(raw score=40) suggests clinically significant symptoms.The SDS has fair internal consistency,with split-half reliability of 0.73 and Cronbach’s α coefficient of 0.68.In this study,we used SDS to measure the criterion-related validity of NWMHQ.

The NWMHQ is used to assess mental health among nurses across six dimensions: emotional status(6 items);psychological security(4 items);positive relationships(5 items);resilience(6 items);self-efficacy(6 items);subjective well-being(5 items).A higher score indicates a better emotional state.In other dimensions,a score of 1-5 represents“strongly disagree”,“disagree”,“general”,“agree”,and“strongly agree”,respectively.A higher score indicates better mental health.

2.5.Data collection

An online questionnaire was utilized for data collection in June 2020.The questionnaire was distributed through the Chinese online questionnaire platform,and researchers shared the web link with the director of the nursing department at each hospital unit.The questionnaire platform included a consent form,information about the study,contact details of the research team,and an anonymous questionnaire.By controlling the Internet protocol(IP)address,a nurse could only fill in the questionnaire once.The response time was controlled to within 30 min to ensure the authenticity of the data.The quality of the questionnaires was checked at the end of each survey,and anonymity and confidentiality were maintained throughout the study.

2.6.Data analysis

IBM SPSS version 25.0 and AMOS 24.0 (IBM Corp.,Armonk,NY,USA) software were used for data analyses.Missing values were confirmed through SPSS and replaced by the series mean (SPSSTransform -Replace Missing Values -add all variables -Choose series mean).The results showed that there were no missing values in the data.Continuous variables were represented by means and standard deviations (SD).Count and percentage values were calculated for categorical variables.

Regarding reliability was examined using Cronbach’s α coefficient for the NWMHQ and its subscales.Cronbach’s α coefficient greater than 0.700,0.800,and 0.900 indicated acceptable,good,and excellent internal consistency,respectively[28].We tested the test-retest reliability by randomly inviting 28 nurses to complete the questionnaire again one week after the first time and calculating the Pearson correlation coefficient of the total questionnaire.

Content validity was verified by the Delphi method,and Kendall’s rank-order correlation coefficient (W) was used to test consensus.Items were included simultaneously on the basis of the following criteria[29,30]:a)median score ＞3.500;b)CV ＜0.250;c)item content validity item(I-CVI)＞0.780(ratios of“very important”and “somewhat important” ＞78.0%).

The criterion correlation validity was tested by Pearson’s correlation coefficients between NWMHQ and SDS.The exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were used to test the structural validity of this questionnaire and the sample size needed for them is 5-10 times the number of items.We randomly divided the participant data into two parts (approximately 50%of all cases for each part).One part was for EFA and the other for CFA.We assessed the model’s fit with the following recommended goodness-of-fit indices [31,32]: 1) values of χ2/df ranging from 2.000 to 5.000;2) root mean square error of approximation (RMSEA) ＜ 0.080;comparative fit index(CFI) ＞0.900;3) a goodness-of-fit index (GFI) ＞0.850 and adjusted GFI(AGFI) value ＞0.800;and 5) values of the normed fit index (NFI) between 0 and 1 and ＞0.900.

3.Results

3.1.Demographic characteristics of participants

A total of 2,827 participants were surveyed and 2,815 valid questionnaires were collected (89 men and 2,726 women).Participants were from 20 (from a total of 34) provincial-level administrative regions in China,with most being from Guangdong(69.2%),Sichuan(25.0%),and Xinjiang(2.8%)Provinces.Participant age was 31.28 ± 8.20 years.Among all participants,1,749 (62.1%) were married,1,281(45.5%)had a bachelor’s-level education,760(27.0%)worked in the surgery department,741 (26.3%) worked in the internal medicine department,1,712(60.8%)had worked for no more than 10 years,and 931(33.1%)worked in night shifts(working over 9 h for each shift) more than four times per month.Demographic data (n=2,815) are presented in Table 1.

Table 1Demographic characteristic of participants (n=2,815).

3.2.Reliability

Cronbach’s α coefficient of the total scale was 0.948,with subscale values of 0.870,0.730,0.830,0.985,0.912,and 0.881.This indicates that the internal consistency of NWMHQ was good.The test-retest reliability of each scale was established using another small sample (n=28) by calculating Pearson’s correlation coefficients between the two test administrations for each scale.The intraclass correlation coefficient of the total questionnaire was 0.850.

3.3.Content validity and criterion-related validity

Content validity was verified by two rounds of the Delphi method.In the first round,Kendall’s W=0.457 and χ2=317.723.Median scores ranged from 3.529 to 4.882,CV ranged from 0.092 to 0.319,and the I-CVI ranged from 0.700 to 1.000.In the second round,Kendall’s W=0.575 and χ2=362.266.Median scores ranged from 4.125 to 4.882,CV ranged from 0.070 to 0.264,the ICVI ranged from 0.750 to 1.000 and the S-CVI/Ave was 0.906,indicating that the content validity was good.The SDS was used to test the criterion-related validity of the overall NWMHQ and subdimensions.The SDS score was negatively related to the NWMHQ score (r=-0.664,P ＜0.01).The Pearson’s correlation coefficients between the SDS and the dimensions of the NWMHQ were -0.710,-0.373,-0.396,-0.523,-0.391,and -0.602(P ＜0.01).

3.4.Construct validity

In EFA,it was possible to conduct factor analysis because Bartlett’s test of sphericity showed that the value of Kaiser-Meyer-Olkin (KMO) was 0.966;χ2=33317.335 (df=741,n=1402),P ＜0.001.In an initial analysis,eigenvalues for the seven factors were ＞1.However,the results of the matrix analysis after rotation showed that there were only two entries under two factors,and 12 entries under one factor.The Scree Plot shows that the line changes from steep to smooth at factors 5-6.Therefore,we limited the extraction to five and six factors,respectively.The results of the Rotated Component Matrix showed that when fixed at six factors,the factor distribution matched the dimensional distribution formed by the second version of the NWMHQ.Six factors explained 65.58%of the variance,with factor loadings between 0.39 and 0.80.Factor loading for item 8 (“I don’t worry about demotion or unemployment without reason”) was only 0.39,and this item was therefore deleted.The results of the EFA showed that item 7 belonged to a factor alone,and was also deleted.The dimensions to which item 39,item 27,and item 20 belonged did not conform to the second version framework of NWMHQ,and these items were therefore also deleted.After running the EFA four times,the rotated component matrix was consistent with the original framework.According to the rotated component matrix,item 13 (“I have the support of my team when things get tough”) belonged to the“Positive relationship”dimension.Factors A-F corresponded to the dimensions of emotional status,psychological security,positive relationship,resilience,self-efficacy,and subjective well-being,respectively,based on a total of 33 items (version 4).The factor loading for the six factors is presented in Table 2.

Table 2The factor loading and variance of exploratory factor analysis.

According to the modification index of CFA,we deleted item 17(“I am willing to discuss work problems with colleagues and leaders”).The final version (version 5) of the NWMHQ was then developed and included six dimensions and a total of 32 items representing emotional status (six items),psychological security(four items),positive relationship(five items),resilience(six items),self-efficacy(six items),and subjective well-being(five items).The model fit indices with a revised parameter specification yielded showed an acceptable fit (χ2/df=4.438,CFI=0.944,GFI=0.914,NFI=0.929,TLI=0.938,RMSEA=0.049),which support the NWMHQ’s six-dimensional structure.The CFA is depicted in Fig.1.

4.Discussion

This study attempted to develop a well-targeted,reliable,and valid workplace mental health scale for nurses.NWMHQ contains six dimensions: 1) emotional status,2) psychological security,3)positive relationship,4)resilience,5)self-efficacy,and 6)subjectivewell-being and a total of 32 items.The tool can be applied through email or other e-platforms and takes 3-5 min to complete.The scale was developed through a rigorous process,in which a panel of experts first established item pools and the original dimensions which were conceptually grounded in psychological scales and literature.Then 17 experts were invited to evaluate the importance of the entries and assess the content validity.In the second round,most of the experts reached a consensus,and some modifications were made to improve understanding.Content validity comprised I-CVI and S-CVI/Ave.With more than 10 experts,the I-CVI would be expected to be ＞0.780 and the S-CVI/Ave would be expected to be ＞0.900[33].In this study,I-CVI ranged from 0.750 to 1.000 and S-CVI/Ave was 0.906,which showed a good content validity.

The results of our study showed that NWMHQ had good consistency and stability.We used Cronbach’s α coefficient to measure the internal consistency and reliability of the NWMHQ.We report a Cronbach’s α coefficient of 0.948 overall and 0.730-0.985 for all sub-constructs,which shows good reliability.The test-retest reliability of the NWMHQ was 0.850,which is greater than 0.700,indicating that the NWMHQ has good stability.

We verified the construct validity and criterion-related validity of the NWMHQ,showing that the NMWHQ has good validity.In EFA,principal component analysis was used to investigate dimensionality.For good factor analysis,a KMO value greater than 0.600 is required[34].We adopted a maximum variance method and varimax with Kaiser normalization.EFA showed that KMO=0.966(＞0.6)and the six factors explained 65.58% of the total variance of the NWMHQ.CFA was performed to ascertain the goodness of fit of the measurement model of the NWMHQ.The results indicate that the hypothesized model consisting of the original theoretical framework generated through EFA was supported [35].The SDS was used to test the criterion correlation validity,and Pearson’s correlation coefficients were calculated for the total scores.Correlations can be classified as very strong(r ≥0.800),strong(r range,0.600-0.790),moderate (r range,0.400-0.590),weak (r range,0.200-0.390),and very weak (r range: 0.000-0.190).At least moderate correlation is required for criterion correlation validity.The Pearson’s correlation coefficients between the total SDS score and the NWMHQ were -0.373 to -0.710 (P ＜0.01),which means that SDS is moderately strongly associated with NWMHQ.Therefore,taken together,the NWHMQ questionnaire has been scientifically validated as a good tool for measuring the workplace mental health of nurses.

Choosing the right mental health assessment tool is important because different results may be obtained with instruments developed based on different theories[36].The currently available measurement tools for nurses’ mental health can be divided into mental illness diagnostic tools and tools based on positive psychology.The former,such as the Symptom Checklist (SCL-90),is one of the most widely used scales to assess the mental health of nurses over the past decade in China [14].An example of a psychological scale based on positive psychology is the Psychological Well-being(PWS).Currently,the use of one of these scales alone is insufficient to measure nurses’ mental health,and several studies have used multiple scales to measure the impact of nurses’ occupational environment on mental health [15,37].However,the questionnaire items tend to be complicated and time-consuming,which hinders the accuracy of psychological investigation to a certain extent.NWMHQ was developed based on the two continua model of mental illness and health,which assesses negative emotions and symptoms of psychological problems as well as positive social functions among nurses.As the scale items were based on the specific occupational environment of nurses,this helps an accurate understanding of the workplace mental health of nurses in the clinical setting.

Fig.1.Modified results of the confirmatory factor analysis of the Nurse’s Workplace Mental Health Questionnaire.

The main limitation of this study is that we did not conduct qualitative interviews with nurses owing to the COVID-19 pandemic;therefore,some important information may be missing in the construction of the entry pool.Additionally,the sample of participants was unevenly distributed geographically throughout China,with most respondents coming from Guangdong Province(69.19%).Therefore,although the study findings represent Guangdong Province,they may not be representative of the population of China as a whole.In addition,the number of experts in our study (n=17) is relatively small,although it exceeds the minimum requirement.Other studies have tended to include 20-30 experts[38],because a larger number of experts may elicit a different set of questions that are worth examining.However,there is also some evidence that a larger sample of experts may not lead to further response diversity [39].Finally,to maintain the participant’s attention,we did not mix negative and positive question wording,which may have increased the probability that participants completed invalid questionnaires.Further research should be conducted to adjust the expression of items and expand the sample,as well as include samples that are representative of all regions of China to establish a domestic norm.

5.Conclusion

Nurses undoubtedly are the healthcare providers most at risk of mental health issues.Mental health management is urgently needed,including regular mental health assessment and timely intervention.Measurement tools that are compatible with the occupational environment of nurses are a precondition of management.The NWMHQ tool was developed in this study to assess mental health among nurses.The NWMHQ has a good correlation with SDS,while psychometric testing showed that it also has good internal consistency,stability,content validity,structure validity,and criterion-related validity.This instrument may help assess the mental health status among nurses,and help nursing managers in developing appropriate psychological interventions.

CRediT authorship contribution statement

Jing Lai:Conceptualization,Methodology,Software,Validation,Formal analysis,Writing -original draft.Rong Zhang:Resources,Supervision,Writing -review &editing.Miaoxuan Hong:Visualization,Investigation.Nanyan Li:Data Curation.

Funding

This study was funded by the Medical Scientific Research Foundation of Guangdong Province,China(No.C2019046)and the Guangdong Health Economics Association (No.2020-WJMZ-23).

Declaration of competing interest

The authors have declared no conflict of interest.

Data availability statement

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Acknowledgments

We would like to thank the experts who participated in the Delphi method and all participants in this study.

Appendix A.Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.ijnss.2022.09.004.