Psychiatry Training

Clinical Skills Assessment FAQ?s

AADPRT Clinical Skills Assessment Task Force

1. Definitions
2. Competencies
3. Parameters of the Clinical Skills Assessment
4. Forms
5. Examiners
6. Remediation
7. RRC

1. Definitions

a. What are the definitions of Clinical Skills Assessment, Clinical Skills Verification, and credentialing?

A Clinical Skills Assessment is an observed patient interview done by a resident which meets the parameters described below. Clinical Skills Verification refers to the documentation of competency in clinical interviewing skills required by the ABPN to be eligible to take the ABPN Boards. Credentialing refers to eligibility to sit for the ABPN Boards. Therefore, a resident who successfully completes a series of Clinical Skills Assessments which results in Clinical Skills Verification is credentialed to take the ABPN Boards.

2. Competencies

a. What competencies does the Clinical Skills Verification cover?

The competencies assessed are focused on Patient Care and Interpersonal Communication Skills. The following three areas will be addressed: 1) the doctor-patient relationship, 2) conducting a clinical interview and data collection, including a mental status examination, 3) case presentation. The specific elements of each area are described in the anchor points on the attached AADPRT Clinical Skills Verification forms.

b. May program directors add additional competencies to the exam?

Program directors may add additional components to the clinical examination but these are not part of the Clinical Skills Verification process. The ABPN only requires that the three areas noted above be formally assessed. However, if other competencies are useful for purposes of the education and training of the residents, or assessment for a program, they may be added. The Task Force suggests, however, that these be kept separate, as documentation of passing the required ABPN Clinical Skills Assessment is necessary for a resident?s file.

c. What are the essential data that should be collected in the clinical interview?

Residents should be expected to ascertain the essential current symptoms the patient is experiencing. In addition, they should be able to elicit other psychiatric symptoms from the present and past history. Residents should obtain pertinent past psychiatric history, medical history, family and social history, and perform a mental status examination. They should also inquire about current treatment.

d. What is the standard of competency? How high should the "bar" be?

The competencies evaluated in this examination include the sound establishment of a respectful doctor-patient relationship; ease and fluency in conducting a psychiatric interview and gaining pertinent knowledge; and an ability to present a clear, concise mental status examination and history. While a residency graduate would also be expected to perform a detailed formulation, differential diagnosis and treatment plan, these competencies are not included in the Clinical Skills Verification.

It is understood that skills in the doctor-patient, clinical interviewing, and case presentation continue to evolve later in training and following graduation, but the minimum level expected of a practicing psychiatrist should be achievable by residents by the end of the PGY2 year of training. If some programs wish to set the "bar" higher, they should define "advanced" competencies, and expect these later in training.

3. Parameters of the Clinical Skills Assessment

a. When should the assessments be administered? PGY1, PGY2, PGY3 year?

Residents learn the clinical skills of relationship development, interviewing and case presentation during the PGY1 and 2 years, and should achieve competency by the end of the PGY2 year. This will vary from program to program, and between individual residents, but the skills should be in place and be evaluated during the later PGY 1, PGY 2 year and early PGY3 year. The ABPN does not mandate when the assessments should take place. The AADPRT Task Force recommends that the examination of these skills take place during the PGY1 and PGY2 years. We discourage programs to wait until the PGY3 year to complete the assessments because it does not allow adequate time to identify residents who need remediation.

b. What types of patients may be used for the exam?

Any type of patient, including children or adolescents, may be used for the Clinical Skills Assessment. Standardized patients should not be used. The Task Force recommends that patients have a wide variety of diagnoses and demographic characteristics.

c. What are the possible settings for the clinical skills assessment? What are some examples of settings?

There are no specific requirements for the setting in which the evaluation may take place. The evaluation could appropriately be done anywhere a practicing psychiatrist would routinely evaluate a patient. Possible settings include an adult or child inpatient unit, consultation/liaison service, general adult or child outpatient clinic, specialty outpatient clinic, or a psychiatric emergency service.

Some settings might be especially advantageous. For example, an adult inpatient unit has the advantages of convenience for faculty and residents, ready availability of patients, a wide range of psychopathology, varying degrees of patient cooperation, and the option that the assessment could be incorporated into the routines of patient care. In contrast, an outpatient clinic would have the advantages of more nuanced psychopathology, more cooperative patients, and the option of faculty selection of specific patients to broaden the resident?s experience.

Alternatively, the assessment might be embedded in an interviewing seminar in which the residents practice observed interviews, are videotaped, and receive detailed feedback regarding their interviewing skills. In this case, formal recognition of residents having successfully completed their observed interviews would be an appropriate conclusion to the seminar.

d. Should the patients be known to the residents, e.g. their inpatient, outpatients, or be "new?"

The ABPN has specified that the Clinical Skills Assessments be done with new patients.

e. How should the examination be folded into the daily workings of a resident?s training? Should it be separate from the daily work of the resident?

There is no ABPN requirement regarding the relationship of the assessment to the routine of clinical work in which the resident is involved. The evaluation may involve interviews that are entirely within the context of clinical care, or are entirely separate from that work (as most mock board exams are now).

There are, however, several advantages to faculty incorporating the evaluation into routine clinical care. Observation of interviews performed during on-going clinical care avoids the anxiety-provoking "high-stakes exam" model of the current board format, ensures that patients are truly representative of those seen in practice, and evaluates residents in the same settings in which they are being trained and have gained clinical skills.

From the faculty perspective, incorporation of the assessment into clinical care is an efficient use of time, encourages frequent repetition of observed interviews, provides additional opportunities for constructive feedback to residents and lowers the stakes for not passing a marginal resident on the evaluation.

For all of these reasons the AADPRT Task Force recommends that the assessments be incorporated into routine clinical care.

f. Do the clinical assessments have to be live? Is it possible to use audio- or videotapes?

So far, the ABPN has discussed live patient interviews as the basis for the Clinical Skills Assessments. They have not taken up the issue of whether there might be circumstances in which assessments are done using videotapes of resident interviews.

g. How many times may the resident take the examination?

There is no requirement for how many times residents can take the Clinical Skills Assessment. Each resident must pass three examinations, and must pass all three parts of each examination. Some programs will elect to offer the assessments very frequently, eg. during the course of every rotation, while some other programs will offer fewer opportunities.

h. How long should the assessment be?

The ABPN requires that the interview take "about" 30 minutes, and the presentation and feedback take at least 15 minutes. The Task Force regards 30 minutes as an absolute minimum to effectively assess interview skills, and believes that interviews between 30 and 45 minutes may provide a better opportunity. However, we recognize that more scarce faculty resources may favor 30 minute interviews.

Reasons to favor a longer interview are the opportunity for more extensive interaction and data collection, more extended observation by the examiner, and closer reflection of the real world clinical situation. Favoring a 30 minute interview are efficiency in terms of time and resource expenditure, greater likelihood of frequent examination because of ease of administration, and possible gains in reliability from a standard time frame. The Task Force recommends that the total time for the interview, case presentation, questions, and feedback session will take a minimum of 60 minutes when 30 minute interviews are conducted, and 75 minutes or more for longer clinical interviews.

i. How can the clinical assessment process be as educational as possible? Should feedback be given?

The clinical assessment process should serve a primarily educational function. It is an opportunity for residents to be closely observed by faculty members, and receive specific and focused feedback. Whatever the format a program chooses to use, the Clinical Skills Assessment should include detailed feedback to the resident on all three competencies with specific reference to the observed interview. Sufficient time is required for feedback; the more specific and detailed the faculty member?s feedback, the better. Residents should have adequate time to ask questions about the interview and presentation, and about the feedback they have received. Residents should be able to review the Clinical Skills Verification forms used by their program. They should have the opportunity to review their performance on the required series of Clinical Skills Assessments with the program director, or faculty member designated as responsible for Clinical Skills Assessments.

4. Forms

a. What forms are available for Clinical Skills Verification?

Clinical Skills Assessments are documented using Clinical Skills Verification forms. The ABPN has made two forms available and has approved two additional forms created by AADPRT. These forms document the assessment performed, and are used to verify competency.

The AADPRT forms were designed for ease of use, to enhance inter-rater reliability, and to facilitate subsequent evaluation of the reliability and validity of the assessment process. A training session for faculty raters is planned for the 2009 AADPRT annual meeting.

b. What are the differences between the ABPN and AADPRT forms?

The forms are similar in that they all include the three basic areas of competency ¡ patient relationship, conduct of the interview, and presentation of the case. They all use an 8-point scale and require a passing score of 5 on the global items.

The AADPRT forms differ from the ABPN forms primarily in that they include anchors for each of the categories and subcategories to assist evaluators in making assessments of resident performance. In addition, the forms differ in the subcategories that are incorporated into the 3 global items.

The two AADPRT forms are different from one another in the length and description of the anchor points. The longer of the two forms was designed to facilitate training in the assessment and to enhance inter-rater reliability. For faculty who have become skilled at doing the assessments, the shorter form is more compact and faster to use.

c. Can a program create its own form/model for clinical assessment?

Yes, programs have the option of designing their own forms and submitting them to the ABPN for approval. The three basic areas of competency (patient-physician relationship, psychiatric interview, and case presentation) must be included, but the subcategories, scoring system, and anchor points may be designed by the training program. Newly-designed forms submitted for approval should be sent to Larry Faulkner, MD, Executive Vice President of the ABPN.

5. Examiners

a. Who can examine?

Any ABPN-certified psychiatrist may be an examiner for a Clinical Skills Assessment. Rotation attendings or volunteer faculty may be utilized for these assessments. At least two different examiners must be utilized for the required three successful assessments.

b. How many examiners need to be in each exam?

One or more examiners need to be present for each exam. One examiner may verify a maximum of two of the required three exams.

c. How will examiners be trained?

Training examiners is an aspect of the Clinical Skills Verification process that is yet to be determined. At this point the ABPN has left it to each program to determine how examiners will be trained and vetted. It is anticipated that the content, method, and process of training examiners will be fleshed out over the next year or so.

AADPRT, and other professional organizations, will be actively involved in working with the ABPN to develop the approach to training examiners. The Task Force strongly endorses training for examiners to best assure standardization, inter-rater reliability, and an educational focus. The Task Force anticipates that AADPRT will take the lead in developing a national model(s) of training, and that there will be a training session for program directors at the 2009 AADPRT Annual Meeting. Over the course of the year, AADPRT will keep members informed about the evolution of this critical aspect of clinical skills verification.

6. Remediation

a. What kind of remediation is appropriate for residents having difficulty passing three clinical assessments?

To optimize the learning potential of the assessment process, the AADPRT Task Force recommends that residents be given frequent opportunities to take and pass these assessments. Offering the assessments early and often will allow programs to identify potential problems quickly and develop plans for remediation. There is no ABPN requirement to report the number of times a resident needs to pass the Clinical Skills Assessment, and it is left to each program to determine preferred methods of remediation and to meet the specific educational needs of individual residents. Remediation may include methods such as additional observed clinical interviews, standardized patients, and review of videotaped interviews performed by the resident.

b. Is passing the Clinical Skills Assessment required for promotion in residency, or for graduation from residency training?

Clinical Skills Verification is a pre-requisite for individuals to be eligible to take the ABPN examination in psychiatry, similar to other credentials required by ABPN for eligibility to take the examination e.g. documentation of 4 months of primary care, 2 months of neurology, etc. While the ABPN does not require passing the clinical skills assessment for promotion or graduation, new RRC guidelines (in effect 4/12/08) do require this for graduation.

7. RRC

a. Will the RRC requirements change to include the clinical skills assessment?

The current RRC program requirements state that all programs must conduct a clinical skills examination annually (lines 1352 ? 1354). They also state that "there must be at least three evaluations at any time during the program in which residents must demonstrate satisfactory competence in establishing an appropriate doctor/patient relationship, psychiatric interviewing, including mental status examination and case presentation. Satisfactory demonstration of the competencies during the three required evaluations is required prior to completing the program" (lines 1366-1376). Thus, the RRC has now incorporated the ABPN Clinical Skills Verification as defined by the ABPN policy, and added that all residents must successfully complete three assessments, at any time during training.

The additional requirement for a clinical skills exam annually means that for those residents who successfully complete three observed interviews prior to the PGY-3 or PGY-4 year, an additional Clinical Skills Assessment would need to occur in each of the following years.

One way to approach these requirements would be to structure ABPN Clinical Skills Assessment interviews in the PGY-1 and PGY-2 years with a more comprehensive exam in the PGY-3 and PGY-4 years (e.g. full DSM differential, case formulation, treatment, etc.).

b. Does the clinical skills credentialing process meet the RRC?s standards for one clinical assessment per year?

As described in a. above, a minimum of three passed Clinical Skills Assessments meets the requirement for the clinical skills verification process. For those residents who are credentialed (i.e. successfully completed three Clinical Skills Assessments) prior to the PGY-3 or PGY-4 year, an additional clinical skills exam must be done in each of those years. This allows programs to have a stepwise process with the ABPN competencies demonstrated first, and additional competencies added in the PGY-3 and PGY-4 years.

c. Are there ways of reducing the bureaucratic and administrative burden of conducting and documenting these assessments, and meeting both the ABPN and RRC requirements?

The primary way of reducing bureaucratic and administrative burden is to embed the observed exams into the residents? current clinical rotations as per FAQ #3e above.

AADPRT Clinical Skills Assessment Task Force
David Goldberg, MD, Co-Chair
Richard Summers, MD, Co-Chair
Eugene Beresin, MD
Michael Jibson, MD
David Kaye, MD
Dorothy Stubbe, MD