How should we evaluate teaching in institutions of higher education?  Despite the deep relevance of this question today, the truth is that we are only beginning to have that discussion and in many places, fitfully at best.  A couple of years ago I was charged to lead a committee to think about this question.  We had two basic issues we wanted to address: (1) we knew that we over-relied on student evaluations; and (2) we did not have much of a culture around open discussions of teaching.  Nothing in this report is particularly innovative; the goal was to chart a practical way toward change or a discussion thereof based on current best practices across the profession.  Although three academic units participated, our final report was, to my knowledge, ignored by all three and the university at large.  I reproduce it below, in slightly edited form, with the hope that somebody, somewhere, might find at least a little part of it slightly useful.

Report of Ad Hoc Committee on Teaching Evaluation

Effective Teaching

There is great and legitimate diversity in what can be considered “effective” teaching. And, as we are all aware from wide public discourse, there is little agreement on how effective teaching is best to be measured in terms of its most important goal, student learning outcomes. Nevertheless, we believe that a series of core criteria can be identified as being important for the assessment of effective teaching. They are:

1. Professionalism
   • Comes to class regularly; prepared; conducts class in a professional manner
   • Accessible to students
   • Grade work fairly and responsibly
2. Curriculum Design
   • Makes an effort to balance needs of units, students, and personal commitments and interests
   • Creates and revises courses when appropriate
3. Course Design
   • Syllabi clearly state goals and requirements for the course; required resources and activities; and class schedule
   • Course content can be justified according to acceptable disciplinary standards
   • Assignments and assessment techniques are appropriate to the course goals and to advancing student learning
4. Effectiveness of Teaching
   • Students feel that they received effective instruction
   • Clearly explains material when appropriate to learning goals
   • Utilizes pedagogical approaches most appropriate for the topics covered
   • Provides timely and helpful oral and written feedback
   • Formal recognition (e.g., awards) of commitment to an effectiveness of teaching
5. Professional Development
   • Self-critically reflects on own teaching
   • Engages in professional development activities
   • Contributes to the Scholarship of Learning and Teaching
   • Attempts new approaches/methods in classes
6. Leadership
   • Coordinates large classes and provides mentorship to TAs
   • Supports the learning of colleagues (e.g., helps assess; coordinates workshops)
   • Involved in wider efforts to improve teaching
   • Involvement in teaching at external institutions

A few explanatory comments are important:

• Every faculty member cannot be expected to do each of these things well. Indeed, some faculty members will not and perhaps should not engage in some of them (e.g., junior faculty members should not be expected to take leadership roles, but should be rewarded should they choose to do so). All faculty members might be expected to fulfill the rather straight-forward expectations in 1, 2 and 3. The other criteria are meant to help develop a more holistic picture of the faculty member as a pedagogue.
• We omit some criteria that are often found in departmental standards, such as “enthusiasm.”  We do not feel that it is appropriate to tie evaluation to the emotional response of a faculty member or a student. It is, for example, entirely possible to imagine a teacher who so clearly conveys a topic so that a student realizes that s/he is not interested in it to be very effective.
• Recognizing the difficulty of ascertaining actual student learning outcomes, we have emphasized here instead teaching as a self-reflective practice. We have great difficulty accurately measuring student learning in our classes, but we can gauge whether a faculty member is concerned about student learning and creates a course that maximizes opportunities for student learning.

Evidence

Teaching is a mufti-faceted activity that can best be assessed only with the help of multiple kinds of data and perspectives. In order to assess a candidate according to the specific criteria articulated above, the following four sources of data are most useful:

Student Evaluations. Many academic units rely heavily – arguably too heavily – on student evaluations as an assessment tool. Research has found that answers to global questions (e..g, “Rate the effectiveness of the instructor”) have been found, in general, to correlate with student learning, but we feel that students, particularly at the end of a harried semester, are not always the best judges of what they really learned. Student evaluations, we feel, are best used to elicit information pertaining to the following criteria indicated above: 1; 3 (partially); and 4 (partially).  Below is a list of more specific suggestions:

• End of semester student evaluations might be more tightly designed primarily for purpose of evaluation rather than eliciting feedback useful to the instructor. This allows the evaluations to be targeted to eliciting information useful for the assessment process; long evaluations tend to lose student interest. Such evaluations might, for example, include only one general space for free-form answers.
• Faculty members are strongly encouraged to administer a separate instrument for gathering information useful for improving teaching. The results of these instruments are not formally used for assessment purposes, but selections might of course be included as part of a teaching portfolio. This might even, and perhaps ideally, take the form of a mid-semester evaluation whose results can be applied quickly.
• Each unit would do well to revisit its student evaluation form in order to ascertain that it most effectively and efficiently elicits the information that is needed for proper assessment while eliminating questions that do not. Every question should be correlated with a specific criterion (or criteria) for assessment, and should substantively contribute to the evaluative process.

Classroom Observation.  Our recommendation is that those units that do not already make use of collegial review begin to so. The decision to begin collegial observation is often informed in part by a desire to provide the best possible support for junior faculty, both in terms of documenting their effectiveness as teachers, and by way of providing opportunities for conversations about teaching and classroom practices outside of a strictly evaluative framework.

We believe that a combination of formative and summative assessments is most likely to both foster ongoing conversions about teaching effectiveness and innovation, and to help us advocate effectively for our colleagues by making it possible to produce persuasive and robust descriptions of their teaching.

Teaching Portfolio. The teaching portfolio is a collection of materials and documents compiled by the instructor to catalog the strengths and accomplishments of a professor in the classroom.   It thus allows the faculty member to reflect on and frame her or his or her activities as a teacher. While not exhaustive, the portfolio ideally suggests the scope and quality of the instructor’s teaching. It typically contains a reflective statement on teaching as well as representative syllabi, course evaluations (either in full or in summary), sample assignments, a list of awards or other recognition of teaching accomplishment, and documentation of any participation in workshops or programs related to teaching. The portfolio may also contain material from others, such as observations or evaluations of one’s teaching by peers or colleagues. There is no single template for what a portfolio ought to contain, and as such, they differ from colleague to colleague and according to discipline.

Colleagues and Chair. Colleagues within an academic unit, or in related units, are often in good position to judge the academic soundness of a course; a faculty member’s willingness to develop an appropriate curriculum; and the faculty member’s contribution to the pedagogical development of their colleagues.

Peer review, in which outside referees are requested to evaluate a candidate’s teaching, present another opportunity for assessment. Outside evaluation of a full teaching dossier can provide a less biased perspective of a candidate’s self-reflective practices as well as valuable feedback on the academic soundness of course design (e.g., are the readings for the topic appropriate?). Instituting a protocol of outside evaluation of teaching, however, raises several complex issues, and while we feel that it is worth further discussion we are not ready to recommend it as a regular feature of tenure and promotion reviews.

It is also an occasional practice to solicit letters from former students and then include selected comments from these letters. We recommend that this practice stop, as it adds little to the assessment and is intrinsically biased.

Implementation

Adopting any or all of the suggestions in this report requires careful thought about efficient and fair implementation. Ideally, we feel the following considerations should guide implementation:

• Changes should be made with an eye first and foremost toward helping the unit develop a strong and supportive environment in which we can all grow and thrive as teachers. The purpose of these changes is not to divide the unit, impose additional work seen as “unnecessary”, or to critique or otherwise burden junior faculty. Senior faculty should take the lead in implementation.
• Any changes should be made over time and fairly. In units not accustomed to classroom observation protocols, for example, observation might take place for two or three years before being used for evaluative purposes. Since at least three years of observation reports would be constitute a sufficient record, this would mean that the reports would be included as part of the files of faculty members who come up for renewal, tenure, and promotion only in and after the trial period.

One structured path toward implementation by each unit could be:

1. Clarifying what the unit means by “effective teaching”. Create a clear list of criteria to be used for purposes of evaluation;
2. Determining what evidence can and should be used to evaluate which criteria:
3. Revising student evaluation forms, as necessary, in light of (1) and (2);
4. Deciding whether to implement or revise a policy of classroom observation;
5. Deciding whether a teaching portfolio should be required or recommended and, if so, how it is to be used;
6. Create a timetable for implementing changes, if any.