Whither the Part 2 exam?
Dr Peter Gibson
Dept of Anaesthesia Westmead Hospital And Children’s Hospital at Westmead.
The aim of this presentation is to put forward ideas about the future of assessment in the College and the place of final exam in that future. It does not reflect any formal decisions made by College office bearers or committees or wish to pre-empt such decisions.
The objective of the FANZCA training program is to produce specialist anaesthetists with the attributes of Medical expert, Communicator, Collaborator, Manager, Health advocate, Scholar, Teacher and Professional as defined by the Canadian Medical Education direction for Specialists, adopted by the Australian Medical Council (AMC) and the Medical Council of New Zealand and incorporated into our curriculum. Each of the attributes has components of knowledge, clinical skills, attitudes and behaviours. The practical purpose of the Final FANZCA exam is to determine if candidates are able to demonstrate sufficient mastery of the knowledge skills and attitudes and behaviours required of a specialist anaesthetist as defined in the FANZCA curriculum to be considered for admission to Fellowship of the College. Apart from the primary exam it is the only form of summative assessment undertaken by trainees.
Health consumers and licensing bodies assume that the FANZCA training and assessment processes produce competent anaesthetists. Given the safety record and high reputation of Australasian anaesthetists we must be doing something right. We however must be also seen to be applying appropriate standards to our training and assessment and strive to improve what we do according to sound educational principles. We have increasing accountability to bodies such as the AMC and to the ever-increasing numbers of trainees presenting for assessment.
There are obviously many aspects of the FANZCA curriculum that cannot be assessed within the format of the current Final exam and indeed in any practical future revision of the exam. There is no summative assessment of procedural skills or assessment of the performance of trainees in real life clinical situations. The medical viva only provides a brief highly structured and artificial assessment of clinical encounters and behaviours. We can restructure our final exam, tinker with the medical viva, add actors and OSCES to test aspects of the curriculum not currently tested and make it an even bigger production than it is or we can reorient our assessment process to emphasise in-training assessment (ITA).
The final exam drives trainee learning to an extraordinary degree. Trainees spend a large part of the year leading up to the exam in study groups researching past MCQ questions, practising Short Answer Questions (SAQ’s) and attending viva practice. Many trainees have their practice viva sessions videotaped for feedback on performance. Some seek the help of professional counsellors to help them cope with the stress of the examination. As well as spending an extraordinary amount of time learning to pass the final exam our trainees are working less hours and are exposed to less clinical material than previously. We need to ensure that our assessment processes drive learning appropriately.
Our final exam was developed prior to development of the modular curriculum and based on but not confined too published objectives of training. The modular curriculum replaced the objectives of training but it did not specify assessment methods for its different domains, relying on the methods already developed for the final. Ideally curriculum and assessment should have been developed together so that assessment was “blueprinted” to the curriculum. It is hard to do this in retrospect but it should be integral to any curriculum revision.
The modular nature of the current curriculum lends itself to assessment related to each of the modules. It would seem logical that summative in-training assessment tools be developed for each of the modules. This would drive learning appropriately within the modules. It would make each module a more intense learning experience. A modular based MCQ exam testing the core knowledge required for each of the modules could be a secure web-based summative ITA. The summative test could be attempted after a set number of formative attempts. Similarly SAQ banks could be developed for each of the modules. The College is investigating tools to provide more formal; perhaps summative in-training assessment of patient encounters (mini-cex) and procedural skills (DOPS). These again can be modular based. This is an appropriate learn-as-you-go approach to training and assessment. I wonder how many of our trainees have been assessed in such skills as rapid sequence induction, failed intubation drills and advanced life support before they are left as the sole anaesthetist in an institution at night. The place for assessing this knowledge and these skills is not at the Final FANZCA but in the first year of training!
If we have modular based summative in-training assessment where does this leave the final exam? It could be argued that once robust appropriate in-training assessment tools are in place there is little role for a final FANZCA exam. I think there would still be a role for case-based anaesthesia vivas as in our current exam. These would provide a way of confirming our ITA processes and provide a way of checking some uniformity of training scheme standards. These vivas would assess what we expect of consultant anaesthetists to be able to do ie integrate knowledge and skills to solve complicated clinical problems. There may still also be a role for a final MCQ exam drawing on well performing questions asked during ITA to check retention of knowledge.
One of the great strengths of the current final exam is that practising anaesthetists develop the material emphasizing points they think important for safe anaesthesia practice. This is particularly shown in the anaesthesia vivas. We have, without the benefit of a formal blueprint, tried to base the exam on the published curriculum. These aspects ensure the validity of the viva exam. The large number of vivas ensures reliability.
The weakest sections of the exam are in the areas that either require more time and educational expertise to prepare such as the MCQ questions or where logistic concerns make valid and reliable examination difficult - the medical viva. In the short term the medical vivas are to be separated in time and place from the anaesthesia vivas to solve logistic problems with patient availability and to allow development of a more valid and reliable assessment. In the longer term pre-anaesthetic clinics provide a rich resource for in-training assessment of medical skills with the development of appropriate tools.
MCQ’s have many assessment advantages but good clinically relevant MCQ’s are difficult and time consuming to write. Many submitted MCQ’s require extensive rewriting and checking. Many MCQ’s are on esoteric points of dubious clinical relevance. There is an online site devoted to the final MCQ bank and some candidates seem to spend an inordinate amount of time debating minutiae. The MCQ component of the exam is very dependant on the continued availability and work of a few individuals. It is time to re-look at how we develop and maintain and use the MCQ bank. As well as developing a MCQ bank based on the modules the MCQ bank has to be in a form that College support staff can maintain and update. Those in charge of developing the MCQ bank need to be able to access the bank remotely. Given the time and effort and expertise needed to develop a robust MCQ bank, the College should look at funding expert groups, not exclusively current examiners, to develop a bank of MCQs for each module. The medical experts trained in MCQ writing should write the MCQ’s but the support staff within the educational unit should maintain the computer database and conduct analysis of questions. When a sufficiently robust bank of questions is developed it can be used for modular based ITA as described previously.
The final FANZCA is here to stay. But it in the future it may be a leaner final exam. Rather than adding to the final exam to I think the future lies in paring it down to the parts that work best and developing in-training assessment to more fully cover our curriculum.
Time of Presentation
1530
