Education

At most colleges and universities, the granting of tenure and promotion is based on achievement in three key areas: scholarship, teaching, and service.  Measuring scholarship and service is not exactly straight-forward, but is still a relatively clear process that usually involves some combination of tallying things up and soliciting input from both external reviewers and those who can comment on a candidate’s contributions.  Measuring teaching, though, is particularly difficult.  Student evaluations, upon which many such recommendations are based, are after all one dimensional and notoriously difficult to parse.  Is there a better way?

Many institutions of higher education have grappled with this problem.  Some time ago I led an ad hoc committee commissioned by three academic units at Brown to consider how we might better, more fairly, and more transparently evaluate teaching.  I posted that report here.  It took a long time for those recommendations to wind themselves down into policy but recently one of the commissioning units of the original report did exactly that, revising its “Standards and Criteria” document (each academic unit has its own set of criteria for tenure and promotion) to accord with the ad hoc report.  In the interest of transparency (and should this be useful to anyone else) the revised section (very slightly redacted) is below:

B. Teaching
Assistant professors will be assigned an on going mentor with whom to discuss issues of pedagogy and personal development as a teacher. We recognize that excellence in teaching can take different forms. All candidates for promotion are expected to address in their statements their own goals, definitions of teaching excellence, and how they assess and reflect upon their effectiveness.
The following evidence will be used by the review committee to evaluate teaching effectiveness:
1) A teaching statement which includes the candidate’s teaching philosophy, classroom approach, and reflections on effectiveness;
2) Completed course evaluations from all the courses taught by the candidate since the date of hire or the last promotion;
3) Syllabi for all the courses for which there are course evaluations;
4) Teaching observation reports, to be completed at least once each year. These will follow a protocol: the faculty member whose teaching will be evaluated will attend a class meeting of the senior faculty member conducting the evaluation; an initial conversation between the two faculty members follows, including consultation about which course and course meeting to observe; then a review of the syllabus before the class meeting, the observation itself, the writing of the report, and the sharing of the report with the faculty member being evaluated;
5) A teaching portfolio that documents teaching effectiveness (including the candidate’s teaching statement, comments on student papers and revised papers, and annotated syllabi; Sheridan Center guidelines for the preparation of a portfolio might be consulted).
In their determination of the value of the candidate’s work submitted for evidence, the review committee will consider such matters as:
1) Quality of course design: Do the courses have appropriate learning goals and are they constructed in a manner that advances those goals? Do they demonstrate knowledge of the state of the field and do they include appropriate assessment techniques? The main evidence used for determination is class syllabi.
2) Contributions to curricular development: Does the candidate seek to create courses that contribute to our curriculum and/or that students might find attractive? Is the candidate willing to teach beyond her/his specialty if necessary? The main evidence used for determination is the candidate’s history of offerings and responsiveness to annual reviews.
3) Advising and mentoring: Has the candidate been willing to advise students on whichever levels are most appropriate? Has the candidate been an effective mentor and thesis advisor (where relevant)? Does the candidate mentor graduate teaching assistants? The main evidence used for determination is institutional data (i.e., the number of advisees); student evaluations; and the candidate’s statement.
4) Professionalism: Does the candidate demonstrate appropriate behavior in the classroom (e.g., comes to class prepared; does not belittle students)? Does the candidate treat graduate teaching assistants with respect? The main evidence used for assessment is student evaluations, classroom observation reports, and the candidate’s statement.
5) Professional development: Is the candidate self-reflective about his/her teaching and does she/he seek to improve it? The main evidence used for assessment is the candidate’s statement and (if provided) the teaching portfolio.
6) Effectiveness of teaching: We recognize that effective teaching and learning takes many forms and that it is very difficult to evaluate, especially in a relatively short time frame. Nevertheless, to the extent possible we would like to know whether students are learning in the candidate’s courses and the extent of that learning. Does the candidate teach in a manner that is inclusive? The main evidence used for assessment is the candidate’s statement; observation reports; student evaluations; and (if provided) the teaching portfolio.
7) Contributions to scholarship on pedagogy: This is not expected of all candidates but active participation in the scholarship of teaching and learning (SOTL) counts to the candidate’s credit.
8) Leadership: Does the candidate help to support and foster an environment that supports the learning and assessment of colleagues? While not expected of all candidates, activities that demonstrate leadership might include helping to observe others and/or organizing sessions about pedagogy.

It should be noted that in my opinion, at least, the goal of evaluation is not strictly evaluation: it is to help to  support a community of practice in which educators support each other in our common mission.  Teaching is hard and involves, in ways that scholarship and service don’t, making oneself vulnerable (a reflection on that here) and ideal work environment is one in which faculty feel free to discuss their failures and well as successes and to help each other to become better.  Evaluation, at its best, is simply a natural outgrowth of sustained reflection and practice.

How should we evaluate teaching in institutions of higher education?  Despite the deep relevance of this question today, the truth is that we are only beginning to have that discussion and in many places, fitfully at best.  A couple of years ago I was charged to lead a committee to think about this question.  We had two basic issues we wanted to address: (1) we knew that we over-relied on student evaluations; and (2) we did not have much of a culture around open discussions of teaching.  Nothing in this report is particularly innovative; the goal was to chart a practical way toward change or a discussion thereof based on current best practices across the profession.  Although three academic units participated, our final report was, to my knowledge, ignored by all three and the university at large.  I reproduce it below, in slightly edited form, with the hope that somebody, somewhere, might find at least a little part of it slightly useful.

Report of Ad Hoc Committee on Teaching Evaluation

Effective Teaching

There is great and legitimate diversity in what can be considered “effective” teaching. And, as we are all aware from wide public discourse, there is little agreement on how effective teaching is best to be measured in terms of its most important goal, student learning outcomes. Nevertheless, we believe that a series of core criteria can be identified as being important for the assessment of effective teaching. They are:

1. Professionalism
   • Comes to class regularly; prepared; conducts class in a professional manner
   • Accessible to students
   • Grade work fairly and responsibly
2. Curriculum Design
   • Makes an effort to balance needs of units, students, and personal commitments and interests
   • Creates and revises courses when appropriate
3. Course Design
   • Syllabi clearly state goals and requirements for the course; required resources and activities; and class schedule
   • Course content can be justified according to acceptable disciplinary standards
   • Assignments and assessment techniques are appropriate to the course goals and to advancing student learning
4. Effectiveness of Teaching
   • Students feel that they received effective instruction
   • Clearly explains material when appropriate to learning goals
   • Utilizes pedagogical approaches most appropriate for the topics covered
   • Provides timely and helpful oral and written feedback
   • Formal recognition (e.g., awards) of commitment to an effectiveness of teaching
5. Professional Development
   • Self-critically reflects on own teaching
   • Engages in professional development activities
   • Contributes to the Scholarship of Learning and Teaching
   • Attempts new approaches/methods in classes
6. Leadership
   • Coordinates large classes and provides mentorship to TAs
   • Supports the learning of colleagues (e.g., helps assess; coordinates workshops)
   • Involved in wider efforts to improve teaching
   • Involvement in teaching at external institutions

A few explanatory comments are important:

• Every faculty member cannot be expected to do each of these things well. Indeed, some faculty members will not and perhaps should not engage in some of them (e.g., junior faculty members should not be expected to take leadership roles, but should be rewarded should they choose to do so). All faculty members might be expected to fulfill the rather straight-forward expectations in 1, 2 and 3. The other criteria are meant to help develop a more holistic picture of the faculty member as a pedagogue.
• We omit some criteria that are often found in departmental standards, such as “enthusiasm.”  We do not feel that it is appropriate to tie evaluation to the emotional response of a faculty member or a student. It is, for example, entirely possible to imagine a teacher who so clearly conveys a topic so that a student realizes that s/he is not interested in it to be very effective.
• Recognizing the difficulty of ascertaining actual student learning outcomes, we have emphasized here instead teaching as a self-reflective practice. We have great difficulty accurately measuring student learning in our classes, but we can gauge whether a faculty member is concerned about student learning and creates a course that maximizes opportunities for student learning.

Evidence

Teaching is a mufti-faceted activity that can best be assessed only with the help of multiple kinds of data and perspectives. In order to assess a candidate according to the specific criteria articulated above, the following four sources of data are most useful:

Student Evaluations. Many academic units rely heavily – arguably too heavily – on student evaluations as an assessment tool. Research has found that answers to global questions (e..g, “Rate the effectiveness of the instructor”) have been found, in general, to correlate with student learning, but we feel that students, particularly at the end of a harried semester, are not always the best judges of what they really learned. Student evaluations, we feel, are best used to elicit information pertaining to the following criteria indicated above: 1; 3 (partially); and 4 (partially).  Below is a list of more specific suggestions:

• End of semester student evaluations might be more tightly designed primarily for purpose of evaluation rather than eliciting feedback useful to the instructor. This allows the evaluations to be targeted to eliciting information useful for the assessment process; long evaluations tend to lose student interest. Such evaluations might, for example, include only one general space for free-form answers.
• Faculty members are strongly encouraged to administer a separate instrument for gathering information useful for improving teaching. The results of these instruments are not formally used for assessment purposes, but selections might of course be included as part of a teaching portfolio. This might even, and perhaps ideally, take the form of a mid-semester evaluation whose results can be applied quickly.
• Each unit would do well to revisit its student evaluation form in order to ascertain that it most effectively and efficiently elicits the information that is needed for proper assessment while eliminating questions that do not. Every question should be correlated with a specific criterion (or criteria) for assessment, and should substantively contribute to the evaluative process.

Classroom Observation.  Our recommendation is that those units that do not already make use of collegial review begin to so. The decision to begin collegial observation is often informed in part by a desire to provide the best possible support for junior faculty, both in terms of documenting their effectiveness as teachers, and by way of providing opportunities for conversations about teaching and classroom practices outside of a strictly evaluative framework.

We believe that a combination of formative and summative assessments is most likely to both foster ongoing conversions about teaching effectiveness and innovation, and to help us advocate effectively for our colleagues by making it possible to produce persuasive and robust descriptions of their teaching.

Teaching Portfolio. The teaching portfolio is a collection of materials and documents compiled by the instructor to catalog the strengths and accomplishments of a professor in the classroom.   It thus allows the faculty member to reflect on and frame her or his or her activities as a teacher. While not exhaustive, the portfolio ideally suggests the scope and quality of the instructor’s teaching. It typically contains a reflective statement on teaching as well as representative syllabi, course evaluations (either in full or in summary), sample assignments, a list of awards or other recognition of teaching accomplishment, and documentation of any participation in workshops or programs related to teaching. The portfolio may also contain material from others, such as observations or evaluations of one’s teaching by peers or colleagues. There is no single template for what a portfolio ought to contain, and as such, they differ from colleague to colleague and according to discipline.

Colleagues and Chair. Colleagues within an academic unit, or in related units, are often in good position to judge the academic soundness of a course; a faculty member’s willingness to develop an appropriate curriculum; and the faculty member’s contribution to the pedagogical development of their colleagues.

Peer review, in which outside referees are requested to evaluate a candidate’s teaching, present another opportunity for assessment. Outside evaluation of a full teaching dossier can provide a less biased perspective of a candidate’s self-reflective practices as well as valuable feedback on the academic soundness of course design (e.g., are the readings for the topic appropriate?). Instituting a protocol of outside evaluation of teaching, however, raises several complex issues, and while we feel that it is worth further discussion we are not ready to recommend it as a regular feature of tenure and promotion reviews.

It is also an occasional practice to solicit letters from former students and then include selected comments from these letters. We recommend that this practice stop, as it adds little to the assessment and is intrinsically biased.

Implementation

Adopting any or all of the suggestions in this report requires careful thought about efficient and fair implementation. Ideally, we feel the following considerations should guide implementation:

• Changes should be made with an eye first and foremost toward helping the unit develop a strong and supportive environment in which we can all grow and thrive as teachers. The purpose of these changes is not to divide the unit, impose additional work seen as “unnecessary”, or to critique or otherwise burden junior faculty. Senior faculty should take the lead in implementation.
• Any changes should be made over time and fairly. In units not accustomed to classroom observation protocols, for example, observation might take place for two or three years before being used for evaluative purposes. Since at least three years of observation reports would be constitute a sufficient record, this would mean that the reports would be included as part of the files of faculty members who come up for renewal, tenure, and promotion only in and after the trial period.

One structured path toward implementation by each unit could be:

1. Clarifying what the unit means by “effective teaching”. Create a clear list of criteria to be used for purposes of evaluation;
2. Determining what evidence can and should be used to evaluate which criteria:
3. Revising student evaluation forms, as necessary, in light of (1) and (2);
4. Deciding whether to implement or revise a policy of classroom observation;
5. Deciding whether a teaching portfolio should be required or recommended and, if so, how it is to be used;
6. Create a timetable for implementing changes, if any.