Subtle skills: Using objective structured clinical examinations to assess gastroenterology fellow performance in system based practice milestones

Marianna Papademetriou, Gabriel Perrault, Max Pitman, Colleen Gillespie, Sondra Zabar, Elizabeth Weinshel,Renee Williams

Abstract BACKGROUND System based practice (SBP) milestones require trainees to effectively navigate the larger health care system for optimal patient care. In gastroenterology training programs, the assessment of SBP is difficult due to high volume, high acuity inpatient care, as well as inconsistent direct supervision. Nevertheless,structured assessment is required for training programs. We hypothesized that objective structured clinical examination (OSCE) would be an effective tool for assessment of SBP. AIM To develop a novel method for SBP milestone assessment of gastroenterology fellows using the OSCE. METHODS For this observational study, we created 4 OSCE stations: Counseling an impaired colleague, handoff after overnight call, a feeding tube placement discussion, and giving feedback to a medical student on a progress note. Twentysix first year fellows from 7 programs participated. All fellows encountered identical case presentations. Checklists were completed by trained standardized patients who interacted with each fellow participant. A report with individual and composite scores was generated and forwarded to program directors to utilize in formative assessment. Fellows also received immediate feedback from a faculty observer and completed a post-session program evaluation survey. RESULTS Survey response rate was 100%. The average composite score across SBP milestones for all cases were 6.22 (SBP1), 4.34 (SBP2), 3.35 (SBP3), and 6.42 (SBP4)out of 9. The lowest composite score was in SBP 3, which asks fellows to advocate for cost effective care. This highest score was in patient care 2, which asks fellows to develop comprehensive management plans. Discrepancies were identified between the fellows’ perceived performance in their self-assessments and Standardized Patient checklist evaluations for each case. Eighty-seven percent of fellows agreed that OSCEs are an important component of their clinical training,and 83% stated that the cases were similar to actual clinical encounters. All participating fellows stated that the immediate feedback was “very useful.” One hundred percent of the fellows stated they would incorporate OSCE learning into their clinical practice. CONCLUSION OSCEs may be used for standardized evaluation of SBP milestones. Trainees scored lower on SBP milestones than other more concrete milestones. Training programs should consider OSCEs for assessment of SBP.

Key words: Objective structured clinical exams; Medical education; Medical error;System based practice; Milestones; Gastroenterology

INTRODUCTION

Medical education assessment in the United States is currently based on six competencies as defined by the Accreditation Council for Graduate Medical Education (ACGME) in 1999: Patient care (PC), medical knowledge, practice-based learning and improvement, professionalism, interpersonal and communication skills(ICS), and system based practice (SBP)[1].

The SBP competencies require trainees to effectively recognize and navigate the larger healthcare system for optimal PC. While these are important skills to build,they are difficult to define and assess in a standardized way in daily clinical encounters. For gastroenterology (GI) fellowships in the United States, the high acuity of inpatient consultations with little time for complete direct observation necessitates focus on tools to evaluate all milestones in addition to SBP[2-5]. Simulation based medical education such as the objective structured clinical examinations (OSCEs) are now a standard methodology for assessing clinical skill and knowledge in medical education[1]. OSCEs are the foundation of Step 2 CS of the United States Medical Licensing Examination. They have the advantage of assessing large groups of learners across a range of skills by recreating situations where there is limited opportunity for supervision and feedback with high validity and reliability[6]. While many studies have corroborated the effectiveness of OSCEs for learner assessment, there are few papers that have evaluated this setting as an effective teaching tool as well[2,7].

Furthermore, SBP milestones are particularly difficult to evaluate objectively and reproducibly in everyday clinical encounters and may be even more difficult to teach in the course of typical clinical encounters[8].

The purpose of our program was to assess first year GI fellows’ skills in SBP milestones utilizing OSCEs that created opportunity for medical errors. We hypothesized that the OSCE would be an effective tool for assessment of SBP.

This is the first paper to our knowledge to focus on these milestones in GI trainees.

MATERIALS AND METHODS

OSCE station development and implementation

Four cases were developed to assess several ACGME milestones (PC, medical knowledge, ICS, SBP, professionalism). Medical education and GI content experts reviewed all 4 cases prior to implementation.

Impaired colleague:The fellows were asked to give sign out to a co-fellow demonstrating emotional and substance-related impairment. Participants were not forewarned of the impaired colleague, but were expected to screen for depression, life stressors, and substance use based on verbal and behavioral cues. This case was adapted from prior use for assessment with faculty and internal medicine residents.

Overnight handoff:The fellows were asked to give handoff to a senior fellow on an acutely ill patient and were in part evaluated on use of best practices from Illness severity, patient summary, action list, situation awareness, synthesis (I-PASS), a validated handoff tool mnemonic that stands for Illness Severity, Patient Summary,Action List, Situation Awareness, and Synthesis by the Receiver[9]. Fellows were given I-PASS resources in the days prior to the OSCE but were not required to review them.We adapted this case's checklist from a previously utilized scenario[10].

Percutaneous endoscopic gastrostomy placement:The fellows were asked to discuss percutaneous endoscopic gastrostomy (PEG) tube placement with the health care proxy of a patient in a persistent vegetative state. The proxy was aware that the patient had expressed wishes not to have a feeding tube placed prior to becoming ill.This case was newly developed for this OSCE.

Note feedback:The fellows were asked to meet with a medical student and give feedback on a consult note which contained multiple documentation errors. This case was redesigned from a previously validated case for internal medicine residents.

Actors with prior OSCE experience performed the roles of the impaired colleague and the patient’s health care proxy in the PEG case. A senior GI fellow and a medical student performed the necessary roles in the overnight handoff case and in the note feedback case, respectively. Collectively these individuals were referred to as standardized persons (SPs). The SPs underwent a 2-hour training session with scripts and role-play to ensure standardization of case portrayals and assessment. The program was held in 3 sessions with different fellows participating in one of each of the three sessions over a 2-year period. The OSCEs were run during the late fall or winter period both years in order to ensure all fellows had spent similar time in fellowship training at time of participation. First year GI fellows were recruitedviaemail to their program directors, and participation was voluntary. All participating fellows encountered identical case presentations with the same SPs and faculty observers and had 12 min to perform each scenario, followed by a 3-minute feedback session. The exception was the senior fellow SP, where two different fellows played this role based on availability. The SPs completed OSCE checklists scoring each fellow immediately after each encounter.

Assessment toolsWe utilized validated checklists completed by the faculty observers and SPs and a post-session program evaluation tool completed by participants[11]. The checklists also included case specific questions that highlighted milestones of SBP and were reviewed by content experts prior to implementation[3,4,12,13].

The checklist items were correlated with specific ACGME milestones and were rated on a 3-point scale of “not done” (the fellow did not attempt the task), “partly done” (the fellow attempted the task, but did not perform it correctly), and “well done” (the fellow performed the task correctly). The score for each milestone was converted from this three-point scale to a composite milestone score across all the cases. For example, a participant’s score for that milestone (ex. PC 1) across all the cases was divided by the total possible score which yielded a number less than 1. This number was then multiplied by 9 to get the composite score for each milestone. This allowed us to compare score between different milestones.

A report with individual and composite scores was generated and forwarded to program directors to utilize in formative assessment (Supplemental Figure 1).Following each session, fellows had a debriefing session to review the teaching points,discuss the experience and provide open-ended feedback. They also completed an exit survey assessing pre- and post-OSCE perceptions and beliefs about competencies,educational value of the experience, and case difficulty. We collected and managed the data using Research Electronic Data Capture, a secure, web-based application[14].

This program was considered an educational performance improvement project by the New York University School of Medicine Institutional Review Board and was not considered for IRB approval.

RESULTS

Twenty-six first year fellows from seven GI training programs in New York City participated. Survey response rate was 100%, however, 2 surveys contained missing data and responses for those questions were not included in the calculation.

Milestone scores

The average composite milestone scores for all 26 learners across all four OSCE cases are reported in Figure 1. A wide variation in performance was noted across the evaluated milestone metrics. The lowest composite score was in SBP 3 (identifies forces that impact the cost of health care, and advocates for, and practices costeffective care), where the mean score was 3.64 out of 9 points. In comparison, the highest composite score was seen in PC2 (develops and achieves comprehensive management plan for each patient), with a mean score of 7.25.

Perceived performance

We evaluated how participants felt they performed in each case (Tables 1 and 2). This can be contrasted with the scores given by the SPs. SPs documented whether particular course objectives were met, which were individualized for each case.Fellows were provided each case objective during the debriefing. In the PEG case,88% of the fellows felt they were well prepared for the case and 66% felt they did well achieving the case objectives. Conversely, only 11% of fellows were rated by the SP to have engaged in shared decision making and no fellows were noted to have assessed the SP’s basic understanding of risks, two points which were identified as case objectives. No fellow fully evaluated for depression, suicidal ideation or alcohol use in the Impaired Colleague case. Only 33% screened for depression and collaborated on identifying next steps. Participants indicated they felt more prepared for these scenarios after the OSCE than before. This difference was most striking for the PEG case and Impaired Colleague case.

Program evaluation

Participants were asked about their preparedness and performance on the cases in an exit survey. These results are summarized in Tables 1, 2, 3 and 4. The fellows rated their performance highly in the Handoff and note feedback cases, with 92.3% of respondents stating their performance was either “fine” or “pretty good.” Selfassessment scores were lower for the PEG discussion and Impaired Colleague case,with 34.6% of respondents stating their performance “could have been better,” in both cases (Tables 1 and 2). The participants’ overall views of the OSCE are reported in Table 4. In general, the participants responded favorably: Eighty-seven percent agreed that OSCEs are an important component of their clinical training, and 83% stated that the cases were similar to actual clinical encounters. All of the respondents stated that the immediate feedback was “very useful,” and 100% of respondents stated they would incorporate OSCE learning into their clinical practice. For each case, a majority of respondents stated they would feel more comfortable in a similar situation after the OSCE than they did before.

Figure 1 Average composite scores for each milestone across all cases.

DISCUSSION

Our main objective was to assess SBP milestones using novel OSCE cases. We were also interested to see if participants felt the experience was helpful as a teaching tool,although we could not assess teaching utility in this observational study.

We highlighted the following SBP milestones in our handoff and feedback cases:SBP1 “works effectively within an inter-professional team”; SBP3 “Recognized system error and advocated for system improvement”; and SBP4, “Transitions patients effectively within and across health delivery systems.”

A recent report calculated medical errors as the third leading cause of death within the United States[15]. Handoffs are vulnerable to communication failures that can compromise quality and safety[16-18]. A 2007 analysis of malpractice claims involving trainees found that 70% of the errors involved the lack of supervision and hand-off errors. These types of errors were disproportionately more common amongst traineesvsfaculty[19]. Studies have demonstrated that standardizing handoffs leads to better communication between physicians and, ultimately, to safer PC[20,21]. Specifically, the mnemonic I-PASS has been shown prospectively to reduce medical errors[9]. In this OSCE, participants were given learning materials explaining the I-PASS format and were provided immediate feedback about critical communication. More fellows agreed with feeling prepared to give adequate hand off after the OSCE than did beforehand.

Graduate medical trainees also have important responsibilities to supervise and teach junior learners, yet rarely receive formal instruction on how to deliver feedback effectively. A survey of 50 graduate medical education programs found that trainees had a lower perception than staff in the “communication and feedback about error”domain, indicating trainees do not feel they obtain regular feedback when an error occurs[22]. After our feedback case, the number of fellows who strongly agreed they felt comfortable giving constructive criticism doubled from 35% to 70%.

We found discrepancies between how fellows felt they met case objectivesvshow they were scored, most apparent in the PEG and the impaired colleague scenarios.These cases highlight complex situations, which we believe early trainees have not often experienced. Our PEG discussion case allowed participants to demonstrate SBP3: Competency advocating cost effective care. The case focused on sensitive issues of end of life counseling and discussing risks and benefits of a procedure - skills which fellows were least likely to demonstrate when compared to other PC skills[13].Similarly, in this OSCE, fellows scored the lowest on the PEG case and the lowest average composite score in SBP3.

The case involving sign out to an impaired colleague is especially relevant with the current focus on physician depression, burnout and suicide. Fellows had to recognize that having an impaired colleague care for patients would directly compromise patient safety. While this situation is rare, the risks are high and the issues are complex. Singhet al[19]also found that 72% of medical errors involved an error in judgment and 58% a lack of competence, both of which would be present in an impaired physician. Twenty four percent of fellows felt they were not prepared for this case, the most of any of the four cases. The number of fellows who strongly agreed they could recognize signs of substance abuse and depression in a colleague more than doubled after the OSCE.

SBP milestones are difficult to assess objectively during GI training programs, as fellows may not be directly observed in situations where these competencies are required. Importantly, fellows’ self-reported comfort levels increased after every OSCE case. Fellows reflected positively on the experience in the post participation survey. They universally felt that the immediate feedback was useful and would improve their clinical skills. All participants stated they would recommend this OSCE as an assessment and training tool. At the conclusion of our study, each fellow was provided a comprehensive report card documenting their performance, which could be utilized by the training program (Supplemental Figure 1). SBP milestone was shown in Supplemental Table 1.

The limitations of our OSCE study are inherent to studies involving subjective assessment, although our instructors and standardized patients were trained prior to the OSCE on scoring and the same individuals scored all 26 participants. Secondly,our study was small, as each OSCE was resource intensive. Lastly, as this was not a longitudinal study, we could not assess whether participation in and feedback from the OSCE improved performance in SBP. An interesting future direction could include the repeat assessment of the participating fellows at the end of their training to assess change in performance over time and to compare them to fellows who did not participate in the OSCE in their first year. In conclusion, OSCEs can be utilized to assess SBP milestones in high risk scenarios linked to medical errors.

Table 1 Fellow reported preparedness for each case, n (%)

Table 2 Fellow reported performance for each case, n (%)

Table 3 Fellows’ perceived skills and beliefs about clinical practice as reported in post-examination survey, n (%)

Table 4 Fellows’ views of the objective structured clinical examination based on post-examination survey, n (%)

ARTICLE HIGHLIGHTS

assessment tool for the evaluation of SBP milestones. We aimed to see if new clinical scenarios commonly encountered by GI trainees would be useful in this assessment. We also sought to evaluate how trainees felt about the experience.

Research methods

We developed four cases to help assess the Accreditation Council for Graduate Medical Education milestones with a focus on SBP. Trainees went through these four simulations with standardized patients and were evaluated by faculty experts using standardized checklists. Their performance from the checklists were aggregated and used to produce a scorecard which was sent to program directors at the conclusion of the OSCE. The trainees were then given direct feedback from the standardized patients and the faculty observer. Finally, the trainees were asked to complete a survey on the experience.

Research results

We ran three OSCE sessions involving 26 GI trainees. Scorecards indicated that, on average,trainees scored lower on SBP milestones than on other milestones categories. We identified and reported discrepancies between how well trainees believed they achieved objectives, and how they were rated by the standardized patients and faculty observers. Overall, trainees reflected positively on the experience in the post participation survey. They universally felt that the immediate feedback was useful and would improve their clinical skills. All participants stated they would recommend this OSCE as an assessment and training tool.

Research conclusions

In this study we demonstrated that OSCEs may be utilized to assess SBP milestones in an objective manner. Since SBP milestones may be difficult to assess in day-to-day activities in the hospital or clinic setting, training programs may want to utilize this type of standardized casebased simulation for assessment. Likewise, trainees reflected positively on the experience and felt they would incorporate feedback into their daily practice.

Research perspectives

Future studies are needed to assess if OSCEs may be useful teaching tools for SBP milestones.This would require repeat assessment with the same OSCE at the GI fellows’ completion of training and comparison of this group to a group who did not participate in the initial OSCE in their first year.