Student self-assessment versus preceptor assessment at the midpoint of a family medicine clerkshipa

William Huang, Larissa Grigoryan

Student self-assessment versus preceptor assessment at the midpoint of a family medicine clerkshipa

William Huang1, Larissa Grigoryan1

Objective:To study how student self-assessment compared with the faculty’s assessment on our family medicine clerkship and to explore the effect of demographic factors on the ratings.

Methods:Students and their faculty preceptors assessed the students’ achievement of clerkship objectives at mid clerkship. We performed Mann—Whitney U tests to compare student ratings and faculty ratings for each clerkship objective. We performed linear regression analyses to investigate the effect of medical school year and student sex on student ratings and the effect of sex concordance or sex difference of the faculty—student pair on faculty ratings.

Results:Two hundred one students completed the family medicine clerkship between July 2015 and June 2016. Faculty ratings were higher than student ratings for all 12 clerkship objectives (P＜0.05 for all comparisons). Third-year students rated themselves higher than second-year students for nine of the clerkship objectives. There was no difference in student ratings between female students and male students and no difference in faculty ratings whether there was a sexconcordant or a sex-different preceptor—student pair.

Discussion:Our fi ndings add to the knowledge of the mid-clerkship feedback process offered by different clerkships. Further study is needed to investigate how students use this feedback to improve for the remainder of the clerkship.

Education; medical; undergraduate; clinical clerkship; student self-assessment

Introduction

It is important that medical students learn to self-assess their knowledge and skills accurately, as their ability to self-assess their knowledge and skills may affect their choice of learning activities and contribute to the quality of care they provide in the future[1—3]. Practicing physicians have been found to have “a limited ability to accurately selfassess” [4]. Therefore medical schools must provide an opportunity for medical students to learn to self-assess their knowledge and skills.

Studies have compared medical students’self-assessment with external measures such as their performance on standardized patient examinations [5, 6] and national board subject examinations [7—9], and have found that medical students’ self-assessment does not correlate well with these external measures.For students on clinical clerkships, there also have been studies of their self-assessment compared with their faculty’s assessment, and the results have been mixed. On clerkships of other disciplines, some studies found that students’ self-assessments did not correlate well with their faculty’s assessments [7, 10, 11], while other studies have reported good correlation between students’ self-assessments and faculty’s assessments [12]. A small study of a family medicine clerkship showed that student self-assessments and faculty assessments were similar [13], but otherwise there have been no recent studies comparing student self-assessment and faculty assessment on a family medicine clerkship.

In its standard on formative assessment and feedback, the Liaison Committee for Medical Education (LCME) stipulates that faculty should provide mid-clerkship feedback to students so that “each medical student is assessed and provided with formal formative feedback early enough during each required course or clerkship four or more weeks in length to allow sufficient time for remediation. Formal feedback typically occurs at least at the midpoint of the course or clerkship” [14]. Recent studies compared student self-assessments and faculty assessments performed as part of the mid-clerkship feedback process;however, the results are mixed. One study of a surgery clerkship reported less frequent agreement in student and faculty assessments at mid clerkship [15], but another report of a surgery clerkship reported frequent agreement at that point in time [16].

In striving for our clerkship to adhere to LCME guidelines and to enable students to receive better formal feedback from their faculty preceptors on our family and community medicine clerkship, we introduced a revised mid-clerkship process that included both students and faculty completing the same assessment of the student’s achievement of key clerkship’s objectives. As we planned this student self-assessment and faculty assessment mid-clerkship process, our objectives were to:

1. compare student self-assessments with faculty assessments regarding their achievement of clerkship objectives at the midpoint of the clerkship;

2. investigate if the year of medical school or sex affects students’ self-assessment ratings;

3. investigate if sex concordance between the preceptor and student or sex difference between the preceptor and student affects how precepting faculty assess their students.

Methods

The Institutional Review Board of Baylor College of Medicine and its affi liated hospitals approved this educational research study.

In the 4-year medical school curriculum at Baylor College of Medicine, students spend the fi rst 1.5 years in the preclinical curriculum and spend the remaining 2.5 years in the clinical curriculum.

The family and community medicine clerkship at Baylor College of Medicine is a required 4-week clerkship for clinical students. Most students take the family and community medicine clerkship during their third year of medical school,but some students take it during the second half of their second year, when they begin clinical rotations. Similarly to other family medicine clerkships, students spend most of the clerkship time in the office of a community-based family physician preceptor, learning through participation in clinical ambulatory care.

In September 2014, we introduced a self-assessment form for students and an assessment form for faculty preceptors to complete at mid clerkship. The forms asked the student and the preceptor to rate the student’s achievement of 12 key clerkship objectives, using the same rating scale: 1—9 Likert scale[1—3, working toward; 4—6, competent; 7—9, excels (on the preceptor form) or confi dent (on the student form)]. (See Table 1 for the list of 12 clerkship objectives being assessed.)

At the midpoint of the clerkship, students were asked to self-assess their performance using their form. Afterward,preceptors met their assigned students for formal feedback sessions in which they completed the faculty assessment form.Students submitted both the student self-assessment form and the faculty assessment form to the clerkship administrators for review.

Statistical analysis

For each of the 12 clerkship objectives, descriptive statistics were calculated. We performed Mann—Whitney U tests to compare student self-assessment ratings and faculty assessment ratings for each objective.

We performed linear regression analyses to investigate the effect of medical school year and student sex on student self-assessment ratings. We also performed linear regressionanalyses to evaluate the effect of sex concordance or sex difference of the faculty—student pair on faculty assessment ratings.

Table 1. Student self-assessment of key clerkship objectives compared with preceptor assessment at the midpoint of the clerkship

IBM SPSS Statistics (version 22; IBM, Armonk, New York,United States) was used to perform the statistical analyses.

Results

Two hundred one medical students completed the family and community medicine clerkship during the study period between July 2015 and June 2016. All submitted a completed student self-assessment form and a preceptor assessment form.Twenty-nine (14.4%) were second-year students and 172(85.6%) were third-year students. Ninety-four (46.8%) were female students and 107 (53.2%) were male students. Eightynine (44.3%) preceptor—student pairs were of the same sex and 112 (55.7%) preceptor—student pairs were of different sexes.The median student self-assessment rating and the median faculty assessment rating for each clerkship objective are given in Table 1. The median faculty assessment on all items was 7.0 or more for all items, indicating students were in the“excels” range for all 12 clerkship objectives. The median faculty assessment ratings exceeded the median student selfassessment ratings on all 12 clerkship objectives (P＜0.05 for all comparisons).

Linear regression analyses for each of nine clerkship objectives showed that the self-assessment ratings of thirdyear medical students were higher than those of secondyear medical students (Table 2). For the remaining three clerkship objectives (professionalism, identifying learning needs, and describing the role and identity of a family physician), there were no significant difference in ratings between second-year medical students and third-year medical students.

For the 12 clerkship objectives, linear regression analyses demonstrated no significant differences in self-assessment ratings between female students and male students. In addition,for all of the objectives, linear regression analyses demonstrated no significant differences in faculty assessment ratings when there was a sex concordance between the preceptor and the student or when there was a sex difference.

Table 2. The effect of year of medical school on students’ ratings for the clerkship objectives: significant results

Discussion

As more clerkships adopt the LCME standard on formative assessment and feedback, our fi ndings add to the growing number of studies investigating the feedback process that occurs at mid clerkship.

As many studies have reported poor correlation between student self-assessment and faculty assessment [7, 10—11, 15],our fi ndings also indicate poor correlation between student self-assessment and faculty assessment of students’ achievement of 12 key clerkship objectives. A few of the studies indicated that the poor correlation was due to female students’ [11]or all students’ [15] self-assessments being lower than their faculty’s assessment. We also found that student self-assessments were lower than faculty self-assessments for all key clerkship objectives.

It is not surprising that second-year students reported lower self-assessment ratings than third-year students. One meta-analysis reported that “students earlier in their medical education were less accurate than students in later years” [17]. The fi ndings of this meta-analysis as well as our own fi ndings suggest that students’ self-assessment skills improve as they advance through medical school and become more experienced, although further studies are needed to confi rm this fi nding.

One study reported that female students are more likely to underrate themselves compared with their faculty assessments,while male students are more likely to overestimate themselves compared with their faculty assessments [11]. However, another study found no difference in student self-assessments on the basis of sex [10]. A meta-analysis of multiple studies concluded that “female students underestimate their performance more than male students,” but also noted that some studies found no difference and many other studies did not perform a sex analysis in their fi ndings [17]. In our study, we did not fi nd a significant difference in the self-assessment ratings between female and male students. Many of the past studies did not perform a multivariate analysis to identify sex as an independent factor, and the question of whether female students and male students selfassess themselves differently remains without a clear answer.

In considering how sex concordance between the preceptor and the student or sex difference affected mid-clerkship faculty assessments, previous studies have demonstrated that sex concordance or difference in the preceptor—student pair leads to differences in students’ learning experiences on family medicine clerkships [18, 19]. We were not able to fi nd a report on how this factor affected faculty assessments at mid clerkship, but one study of an ambulatory care clerkship reported that female students with male preceptors received a higher clinical grade at the end of the clerkship than male students with female preceptors or students with preceptors of the same sex [20]. In our study, which investigated the assessment at mid clerkship, rather than at the end of the clerkship, we did not fi nd that there was a difference in how our faculty preceptors assessed students of the same sex or the opposite sex.

The intent of mid-clerkship feedback is to inform students of their strengths and areas of needed improvement so that they will know what they need to work on for the remainder of the clerkship. There have been few studies on how mid-clerkship feedback changes students’ behavior. One family medicine clerkship reported that mid-clerkship feedback can at least change the types of clinical encounters students saw for the remaining of the clerkship [21]. However, our fi ndings do not directly address how student self-assessment and preceptor feedback at the midpoint of the clerkship affects students’learning behavior for the remainder of the clerkship. More research is needed to understand how the mid-clerkship feedback process can be used to change the behavior of students and improve their performance by the end of the clerkship.

This study has limitations in that the fi ndings represent those of one medical school class on a family medicine clerkship at one medical school, and they may not be generalizable to other clerkships or institutions. In addition, this self-assessment exercise was a required assignment on our family medicine clerkship, but was not graded. This may have affected how students completed their self-assessment of clerkship objectives. Finally, student self-assessments were compared with a subjective standard (faculty assessments). Our results demonstrate that faculty highly rated students’ achievement in all clerkship objectives in the “excels” range. It is difficult to determine whether the study population was an outstanding group of medical students who deserved these high ratings or whether faculty overrated their students. Further study to compare faculty assessments with more objective measures such as examination performance may help us learn the significance of the high ratings that faculty gave their students.

Our results add to our understanding of how student selfassessment compares with the faculty assessment at the midpoint of the clerkship, especially in investigating if there is an effect of the student’s sex or year of medical school on the student’s self-assessments and whether sex concordance or sex difference in the preceptor—student pair affects faculty assessment. As more schools incorporate the mid-clerkship formative assessment and feedback standard required by the LCME,there is an opportunity to further study how student selfassessment compares with faculty assessment at mid clerkship and explore these and other factors that affect the assessments.Ultimately, the goal of this research is to understand how to help students use this mid-clerkship self-assessment and faculty assessment to improve their performance in the remainder of the clerkship. Further comparisons of mid-clerkship assessments by students and faculty and end-of-clerkship assessments (student self-assessments, faculty assessments, student examination performance) may help us understand the progress students make in the second half of their clerkship.

Acknowledgment

The authors thank Elvira Ruiz, coordinator of the family and community medicine clerkship, for creating and organizing an electronic spreadsheet that included all student self-assessment ratings and faculty assessment ratings.

Conflict ofinterest

The authors declare no conflict ofinterest.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profi t sectors.

1. Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med 1991;66(12):762—9.

2. Westberg J, Jason H. Fostering learners’ ref l ection and selfassessments. Fam Med 1994;26(5):278—82.

3. Fitzgerald JT, Gruppen LD, White CB. The inf l uence of task formats on the accuracy of medical students’ self-assessments. Acad Med 2000;75(7):737—41.

4. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. J Am Med Assoc 2006;296(9):1094—102.

5. Gruppen LD, Garcia J, Grum CM, Fitzgerald JT, White CA,Dicken L, et al. Medical students’ self-assessment accuracy in communication skills. Acad Med 1997;72(10 Suppl 1):S57—9.Erratum in: Acad Med 1997;72(12):1126.

6. Coutts L, Rogers, J. Predictors of student self-assessment accuracy during a clinical performance exam: comparisons between over-estimators and under-estimators of SP-evaluated performance. Acad Med 1999;74(10 Suppl):S128—30.

7. Woolliscroft JO, TenHaken J, Smith J, Calhoun JG. Medical students’ clinical self-assessments: comparisons with external measures of performance and the students’ self-assessments of overall performance and effort. Acad Med 1993;68(4):285—94.

8. Edwards RK, Kellner KR, Sistrom CL, Magyari EJ. Medical student self-assessment of performance on an obstetrics and gynecology clerkship. Am J Obstet Gynecol 2003;188(4):1078—82.

9. Weiss PM, Koller CA, Hess LW, Wasser T. How do medical student self-assessments compare with their fi nal clerkship grades?Med Teach 2005;27(5):445—9.

10. Herbert WN, McGaghie WC, Droegemueller W, Riddle MH,Maxwell KL. Student evaluation in obstetrics and gynecology:self- versus departmental assessment. Obstet Gynecol 1990;76(3 Pt 1):458—61.

11. Lind DS, Rekkas S, Bui V, Lam T, Beierle E, Copeland EM 3rd.Competency-based student self-assessment on a surgery rotation.J Surg Res 2002;105(1):31—4.

12. Peterson LN, Eva KW, Rusticus SA, Lovato CY. The readiness for clerkship survey: can self-assessment data be used to evaluate program effectiveness? Acad Med 2012;87(10):1355—60.

13. Alnasir F, Grant N. Student self-assessment in a community based clinical clerkship in family medicine: a preliminary report.Educ Health 1999;12(2):161—6.

14. Liaison Committee for Medical Education. Functions and structures of a medical school. Standards for accreditation of medical education programs leading to the M.D. degree, March 2014.

15. Hochberg M, Berman R, Ogilvie J, Yingling S, Lee S, Pusic M,et al. Midclerkship feedback in the surgical clerkship: the “Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills” application utilizing learner self-assessment.Am J Surg 2017;213(2):212—6.

16. Torres MB, Cochran A. Accuracy and content of medical student midclerkship self-evaluations. Am J Surg 2016;211(6):1153—7.

17. Blanch-Hartigan D. Medical students’ self-assessment of performance: results from three meta-analyses. Patient Educ Couns 2011;84(1):3—9.

18. Carney PA, Dietrich AJ, Eliassen S, Pipas C, Donahue D. Differences in ambulatory teaching and learning by gender match of preceptors and students. Fam Med 2000;32(9):618—23.

19. Levy BT, Merchant ML. Differences in clinical experiences based on gender of third-year medical students in a required family medicine preceptorship. Acad Med 2002;77(12 Pt 1):1241—6.

20. Wang-Cheng RM, Fulkerson PK, Barnas GP, Lawrence SL.Effect of student and preceptor gender on clinical grades in an ambulatory care clerkship. Acad Med 1995;70(4):324—6.

21. Delzell JE Jr, Chumley HS, Clarkson AL. Mid-clerkship feedback is effective in changing students’ recorded patient encounters. Fam Med 2011;43(8):586—8.

1 Department of Family and Community Medicine, Baylor College of Medicine, Houston,TX, USA

aA portion of these fi ndings were presented at the 49th Society of Teachers of Family Medicine Annual Spring Conference, Minneapolis, Minnesota, on May 2,2016.

William Huang, MD

Department of Family and Community Medicine, Baylor College of Medicine, 3701 Kirby Drive, Houston, TX 77098, USA E-mail: williamh@bcm.edu

19 January 2017;Accepted 2 March 2017