Controversies in high – stakes competency assessment for anaesthetists
ISSUES IN HIGH STAKES PERFORMANCE ASSESSMENT
Michele Joseph, Brendan Flanagan
The Alfred Hospital, Melbourne, VIC
High stakes performance assessment continues to be a controversial topic for the medical profession, the government, the media and the community at large. Several high profile Australian examples of patient harm and organisational dysfunction have once again highlighted the need to ensure safety for every patient, every day across our healthcare system.
The issues for individuals surrounding poor performance, recertification and maintenance of standards have been around for a long time but it is only in the last few years that programmes have been developed for the identification, assessment and remediation of underperforming doctors-they exist in UK, Canada, USA and here in Australia, the NSW Medical Board (NSWMB) Performance Assessment Programme has been running for almost 5 years. The aim of the programme is to assist the anaesthetist to return to safe practice by identifying the problem areas and the appropriate means of improving performance. The programme consists of a referral process, interview, site visit, practice observation and a tailored simulation assessment, which includes debriefing. Assessors were recruited by the NSW Medical Board after consultation with ANZCA. They are all practicing anaesthetists with extensive experience as trainers, supervisors, and examiners and in College processes which produce policy documents, accreditation guidelines and regulations. Each assessment involves two assessors and they are indemnified by the NSWMB for the duration of the assessment. The findings of the performance assessment are documented in a narrative report and all simulations are videoed and sent with this document to the medical board. The material is considered by a Performance Review Panel, which is convened independently of the assessors. This panel makes recommendations for the remediation process but can also recommend deregistration. A performance review is undertaken after 12 months of the remediation process and again considered by the NSWMB performance assessment panel.
Experience in this program has raised a number of issues in designing and implementing high stakes performance assessment. The purpose and the regulatory structure of the assessment must be explicit and conveyed to all involved so that the boundaries and consequences are well defined. The process has to address performance rather than competence-what doctors actually do in practice rather than what they can do under controlled circumstances. It also has to be reliable and valid. This requires multiple sources of information from the simulation and the site visit and the cooperation of the staff and management of the organization concerned.
The assessment of anesthetists who are more than 20 years from qualification presents some challenges with respect to:
- The varying scope of practice
- Currency of practice
- Maintenance of professional standards
- Age and ability to learn
- Illness
- Confidence, self-regulation, isolation
- Working environment, culture
The assessment process has to be tailored to the individual’s daily practice but still be able to assess the ability to respond to potential problems, which are rare, but may be life-threatening.
The process has to be accepted by all stakeholders: consumer/patient, regulators and employers and doctors. It needs to be practical and the goals achievable in a given time frame. Assessors must be skilled at interview techniques, have some experience in simulation and be able to come to agreement on rating performance in the workplace and the simulation centre. The use of simulation in this assessment process has attracted supporters and opponents. The simulation is used as an integrated component of the assessment and it always includes briefing, discussion and debriefing. The debriefing component is not common to simulation training programmes around the world, yet it provides an excellent opportunity to explore reasoning, decision-making and insight.
The performance in the actual simulation scenario acts as a trigger for this discussion-they may make errors as we all do, but we are interested in whether they recognize the error, know how to manage the consequences and prevent recurrence. It’s only possible to interpret the behaviour observed and hence introduce bias, if there is no discussion following a performance.
The major issue that has arisen is the relationship between the evidence collected, the criteria applied and the expected standards of performance. While technical skills can be assessed against described parameters, we don’t have structured standards against which to judge behaviours and attitudes. Yet, the experience with the NSWMB programme has shown that it is in these areas that most of the problems arise. Some frameworks such as Anaesthetic Non-technical Skills, which was developed by industrial psychologists and anaesthetists in Aberdeen, offer a useful guide to the critical observable behaviours, which ensure safe practice. The complex problem of performance standards will be explored using a video trigger and interactive discussion during the session.
Time of Presentation
Sunday 14 May 2006 - 1030-1200
