Evaluation of undergraduate students’ nursing assessment and communication skills through an objective structured clinical examination within a high-fidelity simulation using a student-simulated patient?

Hao-Bin Yuan

School of Health Sciences and Sports, Macao Polytechnic Institute, Rua de Luis Gonzaga Gomes, Macao, China

Abstract: Objective: To evaluate nursing students’ assessment and communication skills through an objective structured clinical examination(OSCE) within a high-fidelity simulation using a student-simulated patient (SSP) and analyze the validity, reliability, degree of difficulty,and discriminability of the OSCE.

Keywords: communication ? high-fidelity simulation ? nursing assessment ? OSCE ? student-simulated patient

1. Introduction

Holistic human-centered care requires the competency to establish good therapeutic relationships with patients and provide safe nursing services. To face changeable and complex clinical situations, nursing graduates need the requisite knowledge, skills, and competency to deliver qualified nursing care. Nursing education is the basis for cultivating qualified nursing graduates who can provide safe patient care.1Simulation is a learning method to imitate or replace real experiences within a particular context through an interactive manner and help nursing students to acquire insight into the interconnected structures of professional knowledge under teachers’ guidance.2It is used for developing social–cognitive–emotional competency and team-work cooperation within a controlled simulated environment.3

The skills of nursing assessment and therapeutic communication are important in clinical teaching. The educators should find innovative ways to educate nursing students more effectively in health assessment and the therapeutic use of communication.4Simulation provides an appropriate method for observing and evaluating clinical skills. The simulation experience focuses on training in nursing assessment and empathy communication through the interaction with simulated patients(SPs). Such experience is highly beneficial for nursing students because it ensures that they can improve their level of confidence in clinical nursing care.3A metaanalysis reported that simulation using SPs in nursing education had a significant effect on cognitive, affective, and psychomotor learnings.5The use of SPsvs.manikins to teach cardiac and pulmonary examinations revealed that SPs were significantly preferred by nursing students as a learning method.6Student-simulated patients (SSPs) are the trained nursing students who can portray patient concerns in a realistic style.7Thus,SSPs are among the effective means to coach psychomotor skills, improve therapeutic communication,enhance clinical knowledge, and promote clinical judgment in undergraduate nursing education.8

Regarding student evaluation, SSPs usually focused on students’ shortcomings in competence and clinical performance.9However, the actual knowledge that students have gained was not taken into consideration.10To be clear and systematic, assessment needs to focus on objective criteria to ensure that SSPs can encourage students’ continuous learning process. The educators should consider the means by which evaluation of students’ competencies can be made feasible through adoption of the objective standardized approaches. The criterion-referenced assessments could enhance the reliability of measurements for achievement assessment of learning outcome.7As a summative assessment,objective structured clinical examination (OSCE) played a key role in the assessment of students’ abilities to complete specific skills and demonstrate holistic nursing for SPs. The OSCE was considered to be a reliable and valid assessment tool in assessing knowledge application and specific clinical skills. SSPs were also used as a learning assessment tool in nursing education.8However, the available literature on the use of SSPs in OSCE for nursing assessment and communication skills was limited. It was therefore considered necessary to create contextualized scenarios with a high-fidelity simulation and SSP for developing the objective assessment tools and explore their evaluation quality.

The objectives of this study were to evaluate students’ nursing assessment and communication skills through an OSCE within a high-fidelity simulation using an SSP and to analyze the validity, reliability, degree of difficulty, and discriminability of the OSCE.

Experience learning was defined as the learning process whereby knowledge was derived from it and continuously modified by experience. Knowledge was constructed through grasping it from learning-experiences which the work-environment offers; further, transforming that knowledge into skills that are of practical use is a challenge which can be addressed only through more experience. Simulation-based learning provides students a unique chance to learn through experience and to promote their in-depth reflection about learning.11The simulation-based learning centered on the opportunity for students to practice and learn in the environment that is as close to reality as possible. The closer a learning experience resembled a real event, the easier it was for a student to transfer gained knowledge and skills to real situations. Reflection was the process to reexamine the experience. Students may think about what comes to mind first, what knowledge they learned, and what needs to be improved. Students were expected to demonstrate successfully that they can transfer what they have learned and experienced from one situation to the next.5The clinical education focuses on facilitating knowledge application, accurate clinical judgment, and skill improvement.12

The OSCE involves performance-based testing.The OSCE assessment method consisted of a given number of “show how to do in simulated scenarios” stations at a given time.13It was used to assess students’ clinical competence and to test some knowledge,skills, judgment, and clinical reasoning through direct observations. The OSCE was used to evaluate the abilities to obtain and interpret data and handle unpredictable patient conditions in clinical examinations.14The OSCE was designed to evaluate the performances of the knowledge, skill, and cognitive/affective domains.It was a standard method of assessment in clinical competence, clinical skills, clinical thinking, problemsolving, and communication ability.13

The SP was a standardized patient, who was trained to act as a real patient to simulate a set of symptoms or problems. SPs were extensively used in health care education for students to practice clinical and communication skills.15The OSCEs in high-fidelity simulations—combined with SPs—were conducted for achieving the objective clinical performance assessment. Working with simulators and SPs helped the students to promote conceptual understanding, reasoning, and critical thinking skills. The students were observed and evaluated as they went through a series of stations in which they assessed the patient conditions, conducted the interventions, and treated the patients.13The OSCE within a simulated context not only provided students with a realistic clinical experience but also offered an opportunity for structured reflection on their performance.Students received comments openly from teachers and SSPs. The integration of feedback and reflection increased the students’ abilities to construct knowledge from multiple sources and make a safe and sound decision.16The OSCEs within high-fidelity simulations that deployed SSPs enhanced students’ understanding and respect of diverse cultures and religions and additionally minimized student and evaluator variation in the assessment of clinical skills.13

2. Methods

2.1. Design

A descriptive study was conducted at one nursing school in Macao, China.

2.2. Participants

This study involved a convenience sample of baccalaureate nursing students who had completed courses of fundamentals of nursing, health assessment, and medical-surgical nursing. A total of 54 students completed nursing assessment OSCE (Table 1).

2.3. The simulated scenario in OSCE

The OSCE was designed to evaluate students’ assessment ability, communication skills, and professional attitude. The OSCE included both generic nursing care skills as well as skills specific to each of the scenarios and involved comprehensive abilities to problem-solving and ethical consideration. Scenarios were designed using the high-fidelity simulator, which was a computercontrolled human patient simulator (HPS, SimMan). TheSimMan was a full-body manikin with a realistic upper airway, chest movement, variable cardiac and breath sounds, and a palpable pulse, with an ability to detect some verbal symptoms and respond to interventions.The simulated environments had wireless capabilities to run the simulations using a laptop computer. The scenarios progressed chronologically and were run sequentially. Students who passed the SP training were required to participate in enhancing the realism of interactions.SSPs were proficient in simulating a set of symptoms and emotional states while providing instant feedback and psychosocial interactions. They were expected to demonstrate an emotional connection, ask some appropriate questions, or answer questions from the “nurses.” SSPs used a checklist to record the details of the encounter for assessing student performance. Students provided care for the “patient” in two simulated environments. Station I (nursing assessment, 10 min) was for admitting the patient to the emergency department and conducting an assessment specific to patient conditions. Station II (patient instruction, 10 min) was for explaining the conditions and informing the precautions. Students were expected to assess patient conditions and recognize the abnormal signs and symptoms; meanwhile they should have dealt with the patient’s conditions and complaints,such as pain, breathlessness, vomiting, nausea, cough,and thirst. After reviewing the data from assessment and laboratory examination, students defined the problems which required nursing care and thereafter explained the conditions and informed the precautions to the “patient”.Finally, the debriefing was conducted following each scenario in terms of safe practice, priority setting, continuous assessment, and communication in the simulation laboratory.

Table 1. Characteristics of participants in nursing assessment OSCE (N = 54).

An example of simulated scenarios for OSCE is shown in Figure 1. The situation took place in a simulated emergency room. Mrs. Hong, a 25-year old female, was diagnosed with myocarditis and admitted to the emergency department for chest pain.

2.4. Instruments

2.4.1. Nursing assessment OSCE checklist (NAOC)

The evaluation indicators of nursing assessment in OSCE were grounded in a preliminary Delphi study which was 2 rounds of consultation with 20 nursing experts. The expert authority coefficient was 0.87. The coordination coefficient was 0.613. Based on those indicators, the NAOC was developed by the research team to assess students’ nursing assessment skills in terms of patient identification, history inquiry, symptom assessment, physical examination, and humanistic care. It was a 22-item observational measurement with a three-point Likert scale (2 = Achieved, 1 = Partly achieved, 0 = Not completed/Incorrect). A higher score indicated a better performance in nursing assessment.

Content validity (CVI) of the NAOC was established using 3 experienced nurse educators who were the experts in simulation, SP, and OSCE for baccalaureate nursing students. When the level of inter-rater agreement was reported as 0.99, the CVI value was reported as 0.98. The parameter of internal consistency reliability,Cronbach’s alpha, was 0.839. The inter-rater reliability should be established for minimizing the possibility that the scores would vary from rater to rater. The inter-rate reliability by two raters was 0.834.

Figure 1. The simulated scenario for OSCE. OSCE, objective structured clinical examination.

2.4.2. Communication evaluation rubric (CER)

Based on communication theory and principles of therapeutic communication, the CER was developed by the research team to assess therapeutic communication when students conducted nursing assessment and patient education exercises (Table 2). It was a 14-item observational measurement with a three-point Likert scale (2 = Achieved, 1 = Partly achieved, 0 = Not completed/Incorrect). The higher score indicated better communication between students and SSPs.

The CVI of the checklist was established using 3 experienced nurse educators who were considered experts in conducting simulation teaching, SP,and OSCE evaluation for baccalaureate-level nursing students. When the level of inter-rater agreement was reported as 1.00, the CVI of the CER was reported as 1.00. The parameter of internal consistency reliability,Cronbach’s alpha, was 0.896. The inter-rate reliability by 2 raters was 0.942.

2.5. Data collection and analysis

The audiovisual system was used to record the interactions among participants, the SimMan and SSPs.Each station was videotaped to enable the examiners to make the recording objectively during the OSCEs and to provide detailed feedback to the participants after OSCE. The NAOC and the CER were used by 2 raters to assess participants’ performance on nursingassessment and communication with SSPs in a specific simulated situation. Statistical analyses were conducted with SPSS 26.0 (Software package, Chicago, IL, USA).Descriptive analyses were performed to identify the characteristics of participants, to generate descriptive statistics on the Likert-type data, and to describe the central tendency and dispersion for the scores on the NAOC and the CER. Independent samples’t-test was carried out to compare the mean scores of the NAOC and the CER between male and female students, year 2 and year 3 students, and with or without the experience of simulations or SP interactions.

The degree of difficulty was calculated by dividing the mean score by the highest score. The difficulty factor was the exact opposite of the actual difficulty of the item. The greater the difficulty coefficient, the more the number of students who completed the item correctly,thereby implying that the easier the item was for students to answer. It was generally recommended that moderate difficulty could more objectively reflect the learning effects of students and should be distributed between 0.30 and 0.70. Furthermore, the degree of discriminability reflected the ability of items to distinguish students’ actual level. It was analyzed by the Pearson correlation coefficient (r-value) between the score of each item and the total score. It was defined as good discriminability(0.40 ≤r≤ 1.00), better discriminability (0.20 ＜r＜ 0.40),and poor discriminability (0.00 ≤r≤ 0.20).17

3. Results

The overall item mean score of nursing assessment was from 0.55 to 1.95 with the mean of 1.26 (SD 0.36). Students received high scores in conducting blood oxygen saturation and cardiac and lung auscultation but low scores in vomiting and diarrhea assessment. Difficulty coefficients of the exam were 0.63 while the discrimination index of the majority items of the NAOC (86.4%)was ＞0.20, showing a better to good discriminability(Table 3).

As shown in Table 4, the overall item mean score of communication was from 0.43 to 1.93 with the mean of 1.12 (SD 0.43). Students presented good communication skills in maintaining eye contact and listening, but culture assessment and explanation for visiting need to be improved. Difficulty coefficients of the exam were 0.56 for communication skills. The items of the CER had satisfactory indexes of item discrimination (from 0.38 to 0.84).

There were no differences based on gender, but year 3 students with a 30 week clinical practice (CP) had better assessment and communication skills than the year 2 student with 16 week-CP. The students who had experiences in simulations or interactions with SPs presented better nursing assessment and communication skills than the students without those experiences(Table 5). Nursing assessments were positively related to communication (r= 0.67,P= 0.000).

4. Discussion

Nursing teachers should continuously look for teaching methods to ensure that students are prepared to provide safe and competent nursing care.18SSPs accurately portrayed the patients as encountered in the real clinical situation. Using SSPs was definitely helpful and realistic for promoting clinical reasoning, critical thinking, and communication, all of which are essential skills for nursing assessment. This study showed that the students with experiences in simulation or SP interactions had better assessment and communication skills than the students without those experiences. A previous study reported that practical patient simulations offered the opportunity for students to practice the implementation of these skills with no risk to real patients.19Students engaged in role-playing to analyze real-life scenarios and practice communicating with SPs. This might provide them with the contextualization of information,which enhanced their learning experiences. Students viewed SP scenarios as being more useful than roleplaying when learning communication skills. The specified interactions with SPs emphasized the importance of communication skills for ensuring safe patient care.20The SP simulated experience increased the perceived learning in the nursing therapeutic communication process. The practical experience of communicating with an SP allowed students to ask and discuss communication skills and identify areas for improvement during the debriefing.21The application of SPs in nursing education enhanced communication skills, improved student self-confidence, promoted critical thinking, and helped students to overcome nervousness and anxiety in skill examinations.22

This study showed a positive relationship between nursing assessment and communication skills. As reported in previous studies, students preferred using SSPs for nursing assessment examinations because the experience was more realistic and also because it prepared them for doing physical assessments in real clinical settings.21Students repeatedly played a role in SP interaction, thereby ensuring that they mastered the technology learned from their own experience. Seeing the peers’ demonstrations on communication skills also provided these students with the necessary opportunities to reflect on their own communication skills. Comprehensive and meaningful reflections from SSPs can facilitate the development of psychomotor and communication skills. Feedback from peers and facilitators during the debriefing reduced their anxiety and meanwhileenhanced their self-confidence towards the task of communicating with patients. Students who interacted with SPs performed better on their performance evaluations as compared with students who practiced on peers.21

Table 3. Score description of nursing assessment skills in OSCE (N = 54).

In this study, students performed well in patient identification, blood oxygen saturation checking, heart and lung auscultation, and nonverbal communication behaviors (especially initiating and maintaining eye contact and listening). This may be attributed to the strengthening of knowledge and skill training in classroom teaching and CP. Students paid more attention to the assessment of chest discomfort but ignored the assessment of gastrointestinal reactions related to it,and neglected cultural assessment, the explanation for visiting reasons, and information reconfirmation. It was necessary to foster the nursing students’ cultural awareness and to train students to assess the patients’ feelings and ideas through a systematic clinical thinking mode in which the focus is on both the main clinical manifestations and related clinical manifestations.23Besides, nursing students had better communication skills through developing eye contac, listening, and empathy. Empathy denoted an understanding of the patient’s perception and feelings. Nursing students presented a sense of duty to the “patient” and empathy and advocacy for patients. Students integrated the intellectual understanding of empathy with their concrete behaviors to ensure that empathetic communication with other persons would become habitually ingrained behaviour. Such empathy and advocacy were noted to be essential for establishing rapport and safe patient care.24Students should demonstrate genuineness and honesty for facilitating therapeutic communication. They should not be proficient merelyin using open-ended and closed-ended questions, but also imperatively develop the nonverbal communication skills necessary in clarifying, interpreting, validating, and paraphrasing the information provided by the“patient.”25SPs provided the opportunity to engage the learner in active learning through the provision of a realistic scenario with a human interaction experience. SPs were useful in developing empathic skills,which would in turn facilitate nursing students to successfully implement therapeutic communication.26Students were expected to interact with SSPs and high-fidelity simulations as they would with the patients encountered in real clinical situations. The simulation experience provided an avenue for students to implement therapeutic communication within a controlled environment.27

Table 4. Score description of communication skills in OSCE (N = 54).

Table 5. Comparison of scores of nursing assessment and communication according to student characteristics.

As consistent and measurable evaluation methods, SSP-based simulations were used to assess students not only on the basis of objectively tested assessment and communication skills, but also on the basis of the intentionally inculcated critical-thinking and problem-solving skills. Working and being tested in SSPs improved the clinical skills, communication,and teamwork-abilities of students. As a formative assessment method, the collaborative testing method can help students reflect on their peers’ progress—according to specific criteria—when preparing for a summary assessment. Using standardized clinical simulated scenarios in OSCE was essential to conduct the direct comparison of the students’ assessment skills and communication skills.28The OSCE was designed as a structured checklist that facilitated the objective assessment. The trained SSPs presented the patient’s conditions and feelings reliably and consistently. OSCE standardized checklists were highly content-specific and provided objective ratings for measuring a range of important skills which were regarded to be necessary for nursing practice, including assessment, empathy, questioning, and initiating and maintaining eye contact and wording expressions in interactions with SPs.29Video recording was reported to be a reliable testing alternative that offered fairness in grading, evaluation, and debriefings.

5. Conclusions

SSPs were involved for enhancing the realism of interactions in scenarios. Nursing students can conduct nursing assessments specific to patient conditions,explain the relevant conditions to the patient, and inform the patient the precautions required to be followed.However, students’ cultural awareness and some communication skills need further improvement through training. It was worth noting that two assessment tools were tested for psychometric properties. With moderate difficulty and high discrimination index, OSCE showed satisfactory reliability and validity.

Limitations

A small convenience sample of students from one nursing school was used; for this reason, the findings were limited in their usefulness to be extrapolated to larger populations and the similar demographic profiles resulted in the difficulty in generalization of findings in other nursing programs. The assessment tools need to be further tested to verify their validities and reliabilities,especially when they are used in different contexts or among students with different cultural backgrounds in SSP-based simulations. A larger sample size and additional undergraduate nursing programs may contribute to the genesis of a more robust data set and are expected verify the findings to a greater extent. Moreover, student stress and anxiety in OSCEs also need to be explored in further studies.

Ethics approval

Ethics approval was granted from the research committee and administration council of the institute.Participations were voluntary. Confidentiality of data was maintained. Only aggregate data was reported.

Conflicts of interest

The author declares that there exist no conflicts of interest.