Adopting and adapting the mini-CEX as an undergraduate assessment and learning tool

Page created by Alan Fields
Adopting and adapting the mini-CEX as an undergraduate assessment and learning tool

                        Adopting and adapting
                        the mini-CEX as an
                        assessment and learning
                        Faith Hill and Kathleen Kendall , School of Medicine, University of Southampton, UK

                            n Southampton, we intro-          Medicine1 and evaluated by           examiners rate student perfor-
                            duced an innovative clinical      John Norcini.2,3,4 This paper de-    mance along several predeter-
                            assessment to the final year      scribes how we adopted and           mined dimensions, and provide
                        of our BM programme in 2004.          adapted the mini-CEX for use in an   immediate feedback to students.
                        This was designed to replace the      undergraduate context. We            This process enables observation
                        long-case assessment at the end       address the feasibility and          and feedback by different exam-
                        of attachments, and to increase       acceptability of introducing this    iners in a variety of settings,
 Examiners rate         the opportunities for formative       new assessment tool into an          across a range of patient prob-
                        and summative assessment of           established programme.               lems. For the mini-CEX at South-
        student         students in clinical settings. The                                         ampton, students are assessed on
   performance          innovation was modelled on the            The mini-CEX involves a short,   three different occasions in each
  along several         mini-CEX (clinical evaluation         focused observation of students      of five different clinical attach-
 predetermined          exercise) developed by the            interacting with patients. Using     ments, across a wide range of
                        American Board of Internal            scannable standardised forms,        clinical problems of varying com-

244  Blackwell Publishing Ltd 2007. THE CLINICAL TEACHER 2007; 4: 244–248
Adopting and adapting the mini-CEX as an undergraduate assessment and learning tool
student comments. Student par-       We were aiming
                                                                             ticipants in the evaluation were     to improve not
                                                                             recruited through e-mail invita-
                                                                             tions, and purposive sampling was
                                                                                                                  only our
                                                                             used in order to ensure that a       summative
                                                                             diverse range of participants was    assessments
                                                                             included from across the region,     but also the
                                                                             and from a range of attachments.     learning
                                                                             Staff were also sampled purpo-
                                                                             sively to include course co-ordi-
                                                                                                                  experience of
                                                                             nators and others with               our students
                                                                             responsibility for student assess-
                                                                             ment, both in the NHS and in the
                                                                             School of Medicine. An initial
                                                                             interview schedule, containing
                                                                             questions about experiences and
                                                                             perceptions of the mini-CEX was
                                                                             piloted and revised in response to
                                                                             feedback. The interviews, lasting
                                                                             between 20 and 75 minutes, were
                                                                             conducted by an independent and
                                                                             experienced researcher and were
                                                                             professionally transcribed. The
                                                                             student comments were collected
                                                                             from feedback forms following
                                                                             each attachment. The data was
plexity. The setting tends to be     based on a wide range of data           analysed systematically using the
ward-based (68 per cent in           sources, including:                     framework approach. This ap-
2004/5) and the patients repre-                                              proach outlines a well-defined
                                     • 2,340 completed forms (156            procedure for data analysis that
sent a broad spectrum of ages.
                                       students) (2004/5)                    involves classifying and organis-
Apart from Obstetrics and Gynae-
cology, there is only a small        • 1,146 completed forms (177            ing data according to key themes
                                       students) (2005/6)                    and issues. It is recognised as a
gender disparity (in 2004/5, 54
                                                                             method particularly suited to
per cent of patients were male).
                                     • 33 tape-recorded, semi-struc-         applied research.5
                                       tured individual interviews
    Students have a different
                                       (2004/5)                              ADOPTING AND ADAPTING
examiner for each assessment and
are observed with different          • 4 tape-recorded group inter-          THE MINI-CEX FOR
patients. Students are expected to     views with both staff and             UNDERGRADUATE
initiate the assessment, but each      students (2004/5)                     ASSESSMENT
attachment centre is responsible
                                     • 19 tape-recorded individual
for providing examiners and                                                  Since the mini-CEX was a new and
                                       interviews with staff and stu-
selecting patients. Pre-carbonised                                           unfamiliar assessment tool, the
                                       dents (2005/6)
multi-copy forms allow both                                                  school recognised that its intro-
examiner and student to retain a     • Students’ comments on feed-           duction would be likely to provoke
copy of the assessment, and for        back forms                            some concern among both stu-
additional copies to be forwarded                                            dents and staff. Indeed, a number
                                         This article focuses on the
by the centres to the School                                                 of staff were initially sceptical
                                     qualitative aspects of the evalua-
Office. The forms are then scanned                                           about the introduction of the
                                     tion, describing the key issues
centrally and the results entered                                            mini-CEX. Therefore, prior to its
                                     identified during the adoption of
into the students’ records.                                                  implementation, the school em-
                                     the mini-CEX and how the mini-
                                     CEX was adapted in response to          phasised that an evaluation would
    We have made a number of                                                 be undertaken to identify prob-
                                     these issues. The quantitative
changes to the mini-CEX since its                                            lems as they arose. In addition, a
                                     data, including an analysis of
introduction in response to the                                              videotape about the mini-CEX,
                                     reliability, will be reported in full
findings of a rigorous evaluation                                            which included mock consulta-
                                     in a separate paper.
conducted in 2004/5 and 2005/6.                                              tions, was produced and circu-
The evaluation used both quali-                                              lated. Information about the
                                        The qualitative data included
tative and quantitative methods,                                             mini-CEX was also presented at a
                                     student and staff interviews, and

                                                               Blackwell Publishing Ltd 2007. THE CLINICAL TEACHER 2007; 4: 244–248 245
Many of the         number of meetings attended by        and feedback was indicated more       inations. In addition, some stu-
       concerns         both staff and students.              clearly in the guidelines.            dents reported that the mini-CEX
                                                                                                    has increased their overall level of
   expressed by             Many of the concerns ex-              The mini-CEX forms used in        study time and motivation to learn
       staff and        pressed by staff and students were    2004/5 and the calculations for       because of the recurrent nature of
  students were         practical – including, for example,   an overall pass mark also raised      three mini-CEX’s in each attach-
        practical       the availability of examiners as      some issues. There were concerns      ment. It is no longer feasible for
                        well as the time and organisation     that the 9-point rating scale         students to miss any stage of an
                        required. However, in practice,       being used was too broad. Objec-      attachment, or to rely heavily on
                        these problems did not arise. For     tions were also raised about the      library-based learning at the ex-
                        example, it proved possible to        agreed calculations for an overall    pense of work with patients.
                        find enough clinical examiners for    pass mark that would exempt           Examples of the students’ com-
                        all the students, even in the first   students from further mini-CEX        ments are given in Table 3.
                        year. In addition, time and or-       assessment in their final examin-
                        ganisational issues were found to     ations. There was a perception            A particularly important influ-
                        be less, rather than more, prob-      that the assessment criteria were     ence on students’ learning was
                        lematic than with long-case           too lenient, possibly allowing a      the opportunity to be observed
                        assessment.                           small number of students to pass      and receive immediate feedback
                                                              their mini-CEX’s inappropriately.     on multiple occasions from a
                            Staff and students also ques-     We therefore modified the form        range of examiners. Students felt
                        tioned the reliability of using a     (see Table 1) and strengthened        that the feedback from the
                        single examiner in the mini-CEX       the criteria for passing (see         examiners on their clinical skills
                        compared to the paired scheme         Table 2). These changes appear to     helped them to identify their
                        that had been used for long-case      have been successful. Examiners       strengths and weaknesses, and to
                        assessment. Examiners working on      and students prefer the new           use this information to improve
                        their own might be reluctant to       6-point scale. The number of          and enhance their skills. The
                        give a poor mark, particularly as     students failing to earn exemp-       mini-CEX also provides a useful
                        they had to give immediate feed-      tions (and therefore repeating        opportunity to identify student
                        back to students. There was also      their mini-CEX’s in their final       problems at an early stage within
                        concern about the standard ex-        exams) has increased to the level     an attachment. Students valued
                        pected, and students felt there       we would normally expect.             highly the one-to-one ‘teaching’
                        was a poor correlation between                                              associated with the mini-CEX and
                        the comments to students during       THE MINI-CEX AS A                     felt this was unlikely to have
                        feedback and the written mark.        LEARNING TOOL                         occurred without the assessment.
                        Finally, some staff and students
                        felt that there was a great deal of   In introducing the mini-CEX at            Concerns were expressed by
                        variation across examiners and        Southampton we were aiming to         some examiners that the focused
                        attachments in the time spent         improve not only our summative        history taking in the mini-CEX
                        observing students and giving         assessments but also the learning     might encourage students to view
                        feedback.                             experience of our students            patients in a narrow, de-contex-
                                                              through formative assessment.         tualised way. The importance of
                            In response to these concerns,    We knew from the literature that      taking a more holistic approach to
                        renewed emphasis was placed on        assessment drives student learn-      patient problems was expressed in
                        staff development, and each of the    ing,6 and were expecting that         particular by some examiners from
                        speciality co-ordinators devised      introducing this innovative form      the child health attachment.
                        plans for disseminating good          of assessment would lead to           At Southampton, we are intending
                        practice in the use of the assess-    important changes in the way our      to address this concern by the
                        ment. Increased attention was         students learn. The qualitative       increased use of student portfolios
                        paid to quality control, and more     evaluation showed this to be the      to record full patient histories.
                        detailed guidelines were provided     case. For example, students are
                        for examiners. The expected stan-     now very aware that they will be      CONCLUSIONS AND
                        dard was described more explicitly.   observed closely and marked on        RECOMMENDATIONS
                        The guidelines now clearly state      their history taking, physical
                        that students should achieve the      examination and interaction with      We are continuing to monitor and
                        level that would be expected of       patients. For many students, this     evaluate the mini-CEX. On the
       Renewed          a safe and competent doctor at        means paying far more attention       basis of our findings to date, we
  emphasis was          the start of the Foundation Pro-      to their clinical skills and spend-   believe that this assessment is
 placed on staff        gramme. Finally, the time that        ing more time practising focused      feasible and acceptable to both
                        should be allowed for observation     history taking and physical exam-     students and staff. Despite initial

246  Blackwell Publishing Ltd 2007. THE CLINICAL TEACHER 2007; 4: 244–248
Table 1. Revised mini-CEX form                                                                                There was a
                                                                                                              perception that
                                                                                                              the assessment
                                                                                                              criteria were
                                                                                                              too lenient

                                                                                                              Students valued
scepticism, almost all students     learning opportunities provided     ing in a clinical setting. In par-    highly the
and examiners at Southampton        by the mini-CEX are considerable;   ticular, there is immediate feed-     one-to-one
are now in favour of this innova-   and students are provided with      back to students who have been
tive form of assessment. The        valuable opportunities for learn-   observed interacting with pa-         ‘teaching’

                                                           Blackwell Publishing Ltd 2007. THE CLINICAL TEACHER 2007; 4: 244–248 247
ensure that the wide range of
                         Table 2. Revised mini-CEX marking criteria                                      examiners are marking to the same
                         Completing the form – using the scale:                                          standard. It is also necessary to
                         The full range of the rating scale will be used. Comparison will be made with   ensure that chosen assessment
                         a safe and competent doctor at the start of the F1/PRHO year. It is expected    criteria can discriminate effec-
                         that some rates below ‘meet expectations’ will be in keeping with your level    tively between competent and
                         of experience, and you should be aware that a borderline score represents a     under-performing students. Final-
                         referral, rather than a fail.                                                   ly, it is very important to consider
                         Referral (requires repeat mini-CEX at the BM Final exam in the specialties in   the mini-CEX in the context of the
                         which you are referred):                                                        overall assessment of students: it
                                                                                                         offers a valuable addition to tra-
                         Five or more borderline or below expectation scores in any of the seven         ditional assessments but is not
                         elements across 3 mini-CEX assessments within an attachment
                                                                                                         designed to assess the complete
                         or                                                                              range of knowledge, skills and
                         Three borderline or below expectation scores in any one element within an       attitudes required of a modern
                         attachment (to identify students who have a difficulty in a particular area)    medical graduate.


                         Table 3. Student comments                                                       1. American Board of Internal Medicine
                                                                                                            Clinical Competence Program. The
                         ‘As learning tools they have been fantastic. As an actual occasion to get
                                                                                                            mini-CEX: a quality tool in evaluation.
                         someone... to sit down and properly listen to you and give you genuinely           guidelines and implementation strate-
                         informed feedback... it’s some of the best teaching I’ve had on any of my          gies from program directors. Work in
                         attachments.’                                                                      progress. Philadelphia, PA:ABIM Clin-
                                                                                                            ical Competence Program, June 2002.
                         ‘I think it makes me do a lot more clinical work on the wards – a lot more –
                         because I know somebody is going to be watching me examine people, so I         2. Norcini J, Blank L, Arnold G, Kimball
                         had better know how to do it properly.’                                            H. The mini-CEX (clinical evaluation
                                                                                                            exercise): a preliminary investigation.
                         ‘I found that one of the best features of them actually is that you know once      Ann Intern Med 1995;123:795–799.
                         a week or whenever you get them, you will have a good teaching session and
                                                                                                         3. Norcini J, Blank L, Arnold G, Kimball
                         it’s useful being watched because the criticism they give you is useful and
                                                                                                            H. Examiner differences in the mini-
                         there’s not so many opportunities you get a Consultant to watch you doing a
                                                                                                            CEX. Adv Health Sci Educ 1997;2:
                         full examination on something.’                                                    27–33.
                                                                                                         4. Norcini J, Blank L, Duffy D, Fortna G.
                                                                                                            The mini-CEX: a method for assessing
                        tients. As a summative assess-            adapted in response to local con-         clinical skills. Ann Intern Med
                        ment, it is much preferred by both        cerns and specific settings. In our       2003;138:476–481.
                        staff and students when compared          experience, any school thinking of     5. Ritchie J, Spencer L. Qualitative data
                        to the long-case assessment used          undertaking the mini-CEX with             analysis for applied policy research.
                        in the past.                              undergraduates needs to consider          In: Bryman A., Burgess RG, eds.
                                                                                                            Analyzing qualitative data. London:
                                                                  a number of issues. First, to ensure
                                                                                                            Routledge 1994:173–194.
                           We are therefore clearly in            consistency it is essential to
                        favour of adopting the mini-CEX.          invest in an extensive staff           6. Newble DI, Jaeger K. The effect of
                                                                                                            assessments and examinations on the
                        However, in order for it to be            development programme for po-             learning of medical students. Med
                        adopted successfully in an under-         tential mini-CEX examiners. Staff         Educ 1983;17:1.
                        graduate context, it needs to be          need very clear guidelines to

248  Blackwell Publishing Ltd 2007. THE CLINICAL TEACHER 2007; 4: 244–248
You can also read