Implementation of the accreditation council for graduate medical education outcome project in ophthalmology

Key Points

The changing requirements of working in a modern health-care delivery system have created new incentives for transforming the "apprenticeship model" to a "competency-based" model of education. External stakeholders including the public, the government, and the third-party payers demand evidence not only of teaching but also of proof of learning.^1-4 The Accreditation Council for Graduate Medical Education (ACGME) has proposed six competencies to assist educators in understanding and developing tools both to teach and assess resident and fellow competency.

These "competencies" include: 1) patient care, 2) medical knowledge, 3) practice-based learning and improvement, 4) interpersonal and communication skills, 5) professionalism, and 6) systems-based practice. In addition, surgical specialties like ophthalmology also must teach and assess "surgical competency."¹

Ophthalmology has been one specialty that has been active in developing and testing methods and tools for the competencies. In the process of educational innovation several specific methods have emerged as particularly useful and might form the basis for a "starter kit" for all programs to standardize their assessment process (e.g., direct observation, written and oral examinations, 360° feedback, self-reflection exercises, chart audits, and portfolio).

Ophthalmology opportunities

Ophthalmology is a relatively small specialty and, thus, has an opportunity to implement a select set of methods using defined tools that can be standardized across programs, customized to our specialty needs, and then tested in a multicenter fashion for validity and reliability.

For example, one direct observation method developed for ophthalmology is the Ophthalmic Clinical Evaluation Exercise (OCEX). The OCEX uses a scoring rubric and direct observation of a clinical encounter by the resident. Prior work has demonstrated acceptable reliability and validity for the OCEX.^5-7

A chart audit tool for ophthalmology called the On Call Assessment Tool (OCAT) also has been used to define objective measures of timeliness, appropriateness, or accuracy of ophthalmic on-call consultations.⁸ Several investigators have demonstrated at least face validity for direct observation and structured scoring rubrics for wet lab performance and surgical skills (e.g., objective structured assessment of technical skills [OSATS], objective structured assessment of cataract surgical skills [OSACSS], and Eye Surgical Skills Assessment Test [ESSAT]).^9-10

Another method for assessing the competencies is the "360°" evaluation. In the "360," multiple observers provide a 360° view of the learner, including supervisors (e.g., faculty members or the chief resident), patients, peers (e.g., fellow residents and fellows), subordinates (e.g., medical students, technicians, nurses), and the residents themselves (i.e., self-evaluation). Resident ophthalmic journal club linked both to a structured checklist and to downstream chart audits can verify behavioral modifications in practice-based learning, evidence-based medicine, and practice-based improvement.^11-12

Finally, self-reflection, evidence of life-long learning, and active participation in learning projects can be included in a learner portfolio. Residents can participate, design, implement, and test projects to improve the quality of their own care, reduce medical error in their practice, improve system efficiency, reduce costs, or increase patient satisfaction.

All of these teaching and learning encounters likely will need to demonstrate the following criteria to be in full compliance: 1) structured (rather than ad hoc) activities that include all learners, 2) defined faculty oversight, mentorship, role modeling, and active participation by all, 3) formative, bi-directional, ongoing feedback to and from faculty, 4) explicit, written, quantitative scoring rubrics with explicit behavioral anchors and defined remediation plans for poor performance or nonparticipation, and 5) aggregate benchmark data that are linked to external outcome measures of improvement.