Evaluation of a simulation-based workshop on clinical performance for emergency physicians and nurses

Accident and Emergency Department, Kwong Wah Hospital, 25 Waterloo Road, Kowloon, Hong Kong, China

Evaluation of a simulation-based workshop on clinical performance for emergency physicians and nurses

Chi Ho Chan, Tung Ning Chan, Man Cheuk Yuen, Wai Kit Tung

Accident and Emergency Department, Kwong Wah Hospital, 25 Waterloo Road, Kowloon, Hong Kong, China

BACKGROUND:Simulation-based medical education has been growing rapidly and becomes one of the most popular teaching methods for improving patient safety and patient care. The Simulation Subcommittee of the Hong Kong College of Emergency Medicine organized an educational program emphasizing the team training, clinical decision-making and communication skills. This study aimed to evaluate the attitude of the participants toward a new training program and the change in the knowledge on clinical performance in emergency physicians and nurses after attending the educational program.

METHODS:A course evaluation form was f lled in by the participants at the end of the workshop. An assessment of 20 multiple-choice questions with 5 options was administered to the participants before and after the 2-day simulation-based training workshop.

RESULTS:A total of 72 doctors and nurses working in the Accident and Emergency Department were enrolled. The average pretest and posttest scores were 12 and 14.3 respectively. The percentage improvement in the mean score of the pretest and posttest was 11.5%. The Chi-square test showed signif cant improvement in the pretest and posttest score grading (P=0.00). Pairedt-test revealed signif cant difference between the mean scores of the pretest and posttest (P=0.00).

CONCLUSIONS:Participants had positive attitude toward this new training program. Significant improvement of the knowledge on clinical performance in healthcare professionals in the Accident and Emergency Department was observed after the participation in this simulation-based educational program.

Education; Medical; Decision making; Patient care team

INTRODUCTION

In recent years, simulation-based medical education has been growing rapidly and becomes one of the most popular teaching methods for the improvement of patient safety and patient care. There is growing evidence in the literature on the benef ts of clinical simulation.[1]Studies[2–4]have demonstrated the effectiveness of simulation in the teaching of basic science and clinical knowledge, procedural skills, teamwork, and communication as well as assessment at the undergraduate and graduate medical education levels. This method is particularly suitable for training medical personnel working in the Emergency Department (ED) due to the dynamic, complex nature of the working environment and its interdisciplinary nature. The Simulation Subcommittee of the Hong Kong College of Emergency Medicine (HKCEM) organized an educational program called "2-day simulation-based training workshop on clinical performance at emergency medicine" in the period of 2013–2014. The workshop put emphasis on the team training, clinical decision-making and communication skills. Acquisition of these nontechnical skills could avoid lots of medical blunders and errors.[5]This study was conducted to determine if this educational program was an effective means to enhance clinical performance for emergency personnel.

Binstadt et al[6]proposed the "clinical performancepyramid" which assumed that clinical performance hinged on 4 highly intertwined domains namely knowledge, decision-making, skill performance, and teamwork. Each domain was built on one another with medical knowledge at the base, which informed clinician's decision-making at the next level. The third level was technical skill and the highest state was achieved when a highly competent individual functioned effectively as a member of a team to accomplish a specif c outcome.

To achieve our goal of enhancing clinical performance of frontline ED staff, we designed the workshop content based on these 4 domains. After completion of the workshop, the participants were expected to 1) apply medical knowledge relevant to the specific emergency scenarios in emergency medicine (EM); 2) demonstrate procedural skills relevant to the specific emergency scenarios in EM; 3) compare different strategies of medical decision-making in EM; 4) apply different elements of teamwork in emergency scenarios. And the objective of this study was to evaluate the attitude of the participants and their change in the knowledge on clinical performance after attending this new training workshop.

METHODS

This was a cross-sectional prospective pretestposttest study. The subjects were enrolled from doctors and nurses working in the ED. They were nominated by the head of the department throughout the 16 Accident and Emergency Departments in Hong Kong. Five identical 2-day workshops were held in 2013–2014. The f rst 2 workshops were held in the Simulation Center of the Pamela Youde Nethersole Eastern Hospital. The last 3 workshops were held in the Hong Kong Jockey Club Innovation Learning Center for Medicine which was inaugurated in December 2013. This was a commissioned training program sponsored by the Hospital Authority. Each workshop enrolled 12–16 doctors and nurses currently working in the ED of Hong Kong. They were led by 5–6 facilitators who were doctors and nurses with formal training by professionally recognized training centers on using simulation as the teaching method.

At the beginning of the workshop, the participants were required to complete the pretest assessment which consisted of 20 multiple choice questions with 5 options (Table 1). The questions focused not only on the clinical knowledge about common acute medical problems in EM, but also teamwork, communication skills in diff cult situations and models of decision-making relevant to the unique working environment of the ED. Each question carried one mark. One mark was awarded for the correct answer. No mark was given to any wrong answer or question answered with more than 2 choices. The content validity of the pre- and post-test was accomplished by the tool that was devised and reviewed by a group of EM specialists and senior nurses who were experiencedtrainers in their own right.

Table 1. Multiple choice questions (The options of each question were omitted for simplicity)

Table 2. Example of case scenario

After the pre-test assessment, the participants were divided into groups of 4–6 and rotated through eight different scenarios which comprised of acute, highstake clinical situations encountered in daily practice of emergency medicine. One example of the eight scenarios is illustrated in Table 2. Each scenario was followed by a debrief ng session led by the facilitators. Each scenario session lasted 1 hour. Interactive lectures were conducted in between the scenarios. The topics included teamwork, medical decision-making and communication skills for difficult situations which were all essential components for forming an effective and efficient team in the ED. After the workshop, the participants were asked to complete the posttest assessment which was identical to the pretest assessment. The marking system was same as that of the pretest. The participants were also invited to f ll in a course evaluation form which consisted of 13 items with the lowest mark at 1 and the highest mark at 5. The course evaluation form was specially designed for this workshop. It would be used as a template for other similar simulation-based education programs organized by this center in the future.

Statistical analysis

The Statistical Package for Social Science (SPSS) version 22.0 for Windows was used for statistical analysis. Descriptive statistics, namely range of scores, mean scores and standard deviation, was used for thedescription of data distribution. Data on the scoring grades were tabulated using frequency and percentage. The Chi-square test was applied to test for association between various grading of the pretest and posttest. Paired t-test was used to assess any signif cant difference between the mean scores of the pretest and posttest.

Table 3. Ranking of the participants

Table 4. Year (s) of experience in the ED

RESULTS

In the 72 participants enrolled, 26 were Emergency physicians and 46 were nurses working in the ED. Their ranking and years of experience in the ED was listed in Tables 3 and 4. All the participants f nished the workshopand completed the pretest and posttest assessment. The average pretest and posttest scores were 12 and 14.3 respectively. The range of the pretest score was 4–19, and that of the posttest score was 7–20. The improvement in the mean of pretest and posttest scores was 2.3 (11.5%). Paired t-test revealed statistically significant difference between the mean scores of the pretest and posttest (P=0.00).

The grading of the participants was summarized in Table 5. The passing grade, as in many tests, was def ned as 50% of the total marks. Grade B was categorized as the third quartile of the total marks while grade A as 90% or above. Among the 15 participants who scored grade D in the pretest, 12 of them (80%) achieved the passing grade C or above after the workshop. It was observed that participants who were in lower grades in the pretestachieved higher grades in the posttest (Table 6). Those who were in higher grades in the pretest continued to be in higher grades in the posttest. The Chi-square test revealed that there was significant difference in the number of participants securing higher grades between the pretest and posttest (P=0.00).

Table 5. Distribution of score grading between the pretest and posttest

Table 6. Cross tabulation of the pretest and posttest scores

In the course evaluation form (Table 7), the scores varied from 2 to 5. The mean scores of all items were above 4 which was ranked as high by the participants. Most of the participants would recommend this workshop to their colleagues as ref ected by the highest socre among all the items. Most of them agreed that the debriefing experience allowed them to see their own mistakes (as shown as the second higheset score) and simulation-based training was more appropriate for them than non-simulation based training. Overall, they reacted postively toward the workshop.

DISCUSSION

The objective of this study was to evaluate the attitude of the participants to the new training program and the change in the knowledge on clinical performance. This study found the workshop could improve knowledge and the participants were positive about the course. The participants performed signif cantly better on the posttest assessment after attending the 2-day simulation-based training workshop on clinical performance as shown by the statistically significant result in the paired t-test analysis. Over 56% of the participants achieved a higher grade in the posttest.

Most of the participants in this group found that simulation-based education was more appropriate forthem and the debriefing session was a good way of reflective learning. Although not specified how this was more appropriate for them, it was believed that the characteristics of adult learning proposed by Knowles et al[7]explained why such simulation-based training method was well received by our participants. Adults needed to see the value and relevance of what they learnt. They liked to be actively involved in the learning process. Also, adults needed opportunity for reflection and feedback for improvement. This finding coincided with the survey of educators conducted by Paige et al.[8]They found that debrief ng, a specif c form of feedback, was the most important part of training using simulation, and a respondent called it 'heart and soul' of simulationbased training. The simulation-based teaching method has been used successfully in many other educational activities. Coupled with deliberate practice, it has been proven an effective means of educational method in terms of acquisition and retention of knowledge and skills.[9,10]Overall, they were satisfied with this format of teaching and would recommend others to join the workshop.

Table 7. Course evaluation form

As pointed out by Croskerry et al,[11]the unique operating characteristics, combined with the complex and myriad activities of EM, predicted vulnerability to a multitude of errors. No doubt that procedural skill sets were important for EM staff's clinical performance. Most of our time, especially for emergency physicians, was involved in cognitive process which included thinking and reasoning. The majority of diagnostic failures, probably over 75%, could be attributed to physician thinking failure.[12]Therefore we introduced the Dual Process Model for decision-making[13,14]and the concept of cognitive forcing strategies[15]in the interactive lectures. These topics were well received and provoked great interest to the participants as attested by their course evaluation scores and the feedback at the free text section. In our workshop, we were only able to introduce such concepts and highlighted their pertinence to our daily practice. More comprehensive coverage could only be achieved through other focused courses or by the participants' own pursuance.

We incorporated the framework of TeamSTEPPS[16]to train teamwork skills of our participants. It was a teamwork system designed for healthcare professionals to improve patient safety, communication and teamwork skills among healthcare professionals. Those nontechnical skills required in demonstrating effective teamwork simply could not come into operation in a vacuum. Clinical situations and certain challenging events would be needed to bring about the context for demonstrating such teamwork skills. Therefore, high fidelity medical simulation appeared to be a promising method for enhancing teamwork training.[17]Thanks to the newly-established and well-equipped Hong Kong Jockey Club Innovation Learning Centre for Medicine, the workshop could be run smoothly with the highfidelity simulation facilities and support from the welltrained staff.

Though the workshop was well-received by the participants on the whole, the pre-course reading materials were the course component which scored the lowest mark (average 3.7, range 2–5). The pre-course materials were not as welcomed as they were expected to be. The study was not designed to f nd out the reason. It would be related to intrinsic nature of emergency medicine which covered a broad spectrum of diseases. The scenarios were made up of a wide variety of critical clinical situations from different specialties. The pile of clinical knowledge together with non-technical skills contributed to an overwhelming amount of pre-course materials. The participants might not have enough time to go through the material before the workshop. Effort would be put on the redesign of the pre-course material in the coming workshop.

Simulation-based trainings are most successful when they are incorporated into the standard curriculum instead of an additional components.[18]It will be increasingly used as an assessment tool for accreditation for licensure and maintenance of certification.[19,20]It might be integrated into the formal curriculum of the specialist training for emergency physicians in Hong Kong in the future as it has already been incorporated in some specialties for high-stakes examinations in many other countries.[21]

There were several limitations in this study. First, the pretest and posttest instruments were the same, potentially introducing testing bias. Second, the format of the assessment was not comprehensive. The use of multiple choice questions as the instrument of assessment only tested the knowledge of the participants, but not the skills and behavior. The reliability and validity of the instrument was not addressed as well. In the future workshop, a variety of validated assessment tools could be incorporated in the pretest and posttest assessment to more truly ref ect the change in the knowledge, skills and behavior of the participants. Although no standardized and widely accepted methods for assessing non-technical skills were currently available, there were a number of commonlyused behavioral rating scores designed by different institutions for measuring the outcomes of teamwork training in medicine. For examples, the Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS),[22]the Anesthetists' Nontechnical Skills (ANTS) evaluation,[23]the Mayo High Performance Teamwork Scale (MHPTS).[24]The Ottawa GRS consists of six 7-point scales with descriptive anchors corresponding to f ve core CRM domains, namely leadership, problem solving, situational awareness, resource utilization, communication, and one overall performance domain. It has strong validity and inter-rater reliability for assessing crisis resource management skills and team leader performance in a simulated environment across a variety of clinical scenarios and training levels. This behavioral assessment tool could be used for rating the pre- and post-scenario test in the upcoming workshop. Third, the posttest assessment was carried out immediately after the workshop. The participants had fresh memory of what they had learnt in the workshop. However, the review conducted by Custers[25]suggested that in medical education approximately two-third to three-fourth of knowledge would be retained after one year, with a further decrease to slightly below fifty percent in the next year. Further assessment was necessary to test the retention of the knowledge and skills after the workshop in the future. Lastly, the sample size was small in this study. As the workshop continues to be run in the future, more participants can be accumulated to increase the data pool and validate the study.

In conclusion, this 2-day-simulation-based training workshop improved the knowledge of clinical performance of the participants and was well received. Future studies could focus on how the improvement of knowledge on clinical performance could lead to enhancement of the patient care and safety and how this type of training might be integrated into the formal curriculum of the specialist training for emergency physicians.

Funding:This commissioned training program was sponsored by the Hospital Authority in Hong Kong.

Ethical approval:Not needed.

Conf icts of interest:The authors have no competing interests.

Contributors:Chan CH and Chan TN are responsible for the study design, analysis and interpretation of data, and drafting. Tung WK and Yuen MC are responsible for revision of the intellectual content and f nal approval of the version. All the authors have read and approved the f nal version of the manuscript.

1 Issenberg SB, Mcgaghie WC, Petrusa ER, Gordon DJ, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Med Teach 2005; 27: 10–28.

2 Chakravarthy B, Ter Haar E, Bhat SS, McCoy CE, Denmark TK, Lotfipour S. Simulation in medical school education: review for emergency medicine. West J Emerg Med 2011; 12: 461–466.

3 McLaughlin S, Fitch MT, Goyal DG, Hayden E, Kauh CY. Simulation in graduate medical education 2008: a review for emergency medicine. Acad Emerg Med 2008; 15: 1117–1129.

4 Okuda YI, Bryson EO, DeMaria S Jr, Jacobson L, Quinones J, Shen B, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med 2009; 76: 330–343.

5 Leonard M, Graham S, Bonacum D. The human factor: The critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care 2004; 13: i85–i90.

6 Binstadt ES, Walls RM, White BA, Nadel ES, Takayesu JK, Barker TD, et al. A comprehensive medical simulation education curriculum for emergency medicine residents. Ann Emerg Med 2007; 49: 495–504.

7 Knowles M, Holton EI, Swanson R. The adult learner: the definitive classic in adult education and human resource development. Burlington, MA: Elsevier 2005.

8 Paige JT, Arora S, Fernandez G, Seymour N. Key elements of debrief ng for simulator training. Am J Surg 2000; 17: 516–517.

9 Ericsson KA, Charness N, Feltovich PJ, Hoffman RR. The influence of experience and deliberate practice on the development of superior expert performance. The Cambridge Handbook of Expertise and Expert Performance. Cambridge University Press. 683–703.

10 Wayne DB, Barsuk JH, O'Leary KJ, Fudala MJ, McGaghie WC. Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice. J Hosp Med 2008; 3: 48–54.

11 Croskerry P, Sinclair D. Emergency medicine: a practice prone to error? CJEM 2001; 3: 271–276.

12 Graber M. Diagnostic errors in medicine: a case of neglect. Jt Comm J Qual Patient Saf 2005; 31: 106–113.

13 Kahnemen D. Thinking fast and slow. Farrar, Straus and Giroux, New York; 2011.

14 Croskerry P. Clinical cognition and diagnostic error: applications of a dual process model of reasoning. Adv Health Sci Educ Theory Pract 2009; 14: 27–35.

15 Croskerry P. Cognitive forcing strategies in clinical decision making. Ann Emerg Med 2003; 41: 110–120.

16 Agency for Healthcare Research and Quality. TeamSTEPPS: national implementation. http://teamstepps.ahrq.gov/. Accessed 1st July, 2014.

17 Shapiro MJ, Morey JC, Small SD, Langford V, Kaylor CJ, Jagminas L, et al. Simulation based teamwork training for emergency department staff: does it improve clinical team performance when added to an existing didactic teamwork curriculum? Qual Saf Health Care 2004; 13: 417–421.

18 McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Medical education featuring mastery learning with deliberate practice can lead to better health for individuals and populations. Acad Med 2011; 86: e8–e9.

19 Buyske J. The role of simulation in certif cation. Surg Clin North Am 2010; 90: 619–621.

20 Steadman RH, Huang YM. Simulation for quality assurance in training, credentialing and maintenance of certification. Best Pract Res Clin Anaesthesiol 2012; 26: 3–15.

21 Levine AI, Schwartz AD, Bryson EO, Demaria Jr S. Role of simulation in U.S. physician licensure and certif cation. Mt Sinai J Med 2012; 79: 140–153.

22 Kim J, Neilipovitz D, Cardinal P. A pilot study using highfidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, And Crisis Resource Management Study. Crit Care Med 2006; 34: 2167–2174.

23 Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cogn Tech Work 2004; 6: 165–171.

24 Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, et al. The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc 2007; 2: 4–10.

25 Custers EJ. Long-term retention of basic science knowledge: a review study. Adv Health Sci Educ Theory Pract 2010; 15: 109–128.

Received August 9, 2014

Accepted after revision December 28, 2014

Chan Chi Ho, Email: cchhk@yahoo.com.hk

World J Emerg Med 2015;6(1):16–22

10.5847/wjem.j.1920–8642.2015.01.003