As longtime readers of this blog know, I have been thinking for a long time about issues of evaluation: How do we know if we, as individuals and collectives, are doing a good job? Within the academy, what are the best practices when it comes to evaluating the effectiveness of our teaching (and here) or a job candidate (here, here, and here)? Much of my own thinking has been influenced by Daniel Kahneman’s book, Thinking, Fast and Slow, which is particularly helpful for identifying our biases and developing strategies for working around them.
One of the most pervasive components of an academic file, be it for a job, fellowship, or promotion, is the recommendation letter. Yet few documents are as subject to bias, both by writers and readers. In this short piece, I try to outline in general terms how I read, and write, such letters. My way is certainly not the only way, but my goal is to contribute to a conversation about how we use such letters.
The following opinion was published in Inside Higher Ed.
What a professor considers when reading recommendation letters (opinion)
The season of writing recommendation letters is coming to an end, and the season of reading them will soon begin. For those being evaluated—particularly for a fellowship, an academic job or admission to a graduate program—the entire process of how those decisions are made can be mysterious. And perhaps nothing about it is as baffling as the role that recommendation letters play.
The content of the letters is itself a bit of a mystery, attenuated by the need for the applicants to waive all rights to ever see what is in them. (And that really is necessary if the letter is to be taken at all seriously.) While, earlier in my career, I used to ask for many such letters—and continue to do so, but less frequently—I more often find myself writing and reading them. Academics use no single approach to the way they read letters, but perhaps by explicitly stating how I read them, I can help to dispel for the applicants some of the apprehension and anxiety inherent in the process.
When I read recommendation letters, I look for three things: 1) confirmation, 2) context and 3) insight. Let me unpack each one.
When I have to evaluate any fellowship, job, promotion or graduate admission application, I begin with the applicant’s statement or cover letter and then move, if available, to that person’s other written materials, usually a writing sample or proposal. Those are the primary documents from which I form an impression. I then turn to the rest of the information—maybe a transcript, test scores, previous education and letters—to see if I can confirm that impression. In the best applications, everything lines up; all the materials present a coherent image. When they do not, I need to re-evaluate my initial reaction and think a bit harder about which evidence to value more and which to value less. Almost always, when it comes down to a discrepancy between items written by the applicant and the recommendation letters, the letters lose.
The primary scenario where the letters become more important is when they can provide greater context for the applicant’s work. Applicants (and, in fact, not just applicants—I frequently find myself in such a situation) sometimes are too close to their material to put it into a context that somebody not in their immediate field fully understands. I often find myself wondering about the significance of a proposal in an area with which I have limited familiarity and look to the letters to provide the lay of the land. While I do not really fault an applicant—especially a junior one—for being unable to articulate the significance of what they are doing, I do depend on the recommendation letters to provide that information.
Finally, my ideal candidate for almost anything—and this may speak to my own idiosyncrasies—is confident but has some humility. I always delight in learning something small but new from a letter, an accomplishment that the applicant may have forgotten to mention. I am very cautious about discussions of character traits because they are so prone to unconscious biases, but fun and positive facts are always welcome.
Note what I do not take very seriously: value judgments and the stature of the letter writer. Letter writers often have a direct interest in the success of a student. I know that I feel a little proud when someone I recommend gets a notable acceptance. But I find the praise routinely found in recommendation letters—most egregiously in letters for tenure and promotion—mostly vapid and unhelpful. “Best student in my career” or “pathbreaking scholarship” are meaningless terms unless they are backed up with evidence, as they so often are not.
If a writer says that the applicant was the “best student” he ever had, for example, I want to know what that means and how often that writer has used the term for other students. “Pathbreaking scholarship” requires a serious elucidation of the specific intervention made by the work, how it is “pathbreaking” and the specific ways in which the writer thinks it will be influential. It also demands a candid evaluation of whether, when it comes to prediction, the writer has previously demonstrated their ability to be more accurate than not.
When I look at who wrote the recommendation letter, I often think back to the other letters that I have read by this writer for comparison. (The fields I work in are small enough that this happens more often than one might think.) Is the writer always critical, or is every scholar “the best of her cohort”? I have neither the time nor the inclination to actually dig up those past letters, but I am left with distinct impressions about which letter writers are honest and which are not. And, ultimately, a letter from a giant in the field at a most selective college or university, in and of itself, is not worth more to me than one from a junior scholar at a less selective one.
How I read letters has also informed how I write them. I do not think that I have ever written a letter for a person who has superhuman intellectual powers or who is without room for professional improvement, and I never try to write a letter that suggests otherwise. I also try to keep my letters succinct, because I know how much work these evaluations require. I try to be helpful to colleagues who are laboring in good conscience to make incredibly difficult and consequential decisions. And I hope that my colleagues have equal consideration for me.
I wish I could say whether most scholars share my approach to the reading of these letters. From my own limited experience, I know that at least some do not, and probably most scholars, afraid of endangering the chances of a candidate they are recommending, are more effusive than I am when they write such letters. The very fact, however, that I do not know how my colleagues read these letters is part of the problem: we rarely have an honest and open discussion about how we evaluate. One of my goals in this essay is to advance that conversation.
This might be cold comfort to applicants. The application process in academia is as messy and subjective as it is in almost every other field. When I say that recommendation letters on their own and the support of the “big guns” are unlikely to land a fellowship, position, promotion—as other letters are unlikely to torpedo such chances—I know that I do not speak only for myself. I also know that most selection committees are made of smart, hardworking people who are genuinely torn by difficult decisions. I would like to think that most of us would rather you invest your energy and time in your own work rather than cultivating letter writers, and that you have some faith in the evaluation process.
Michael L. Satlow is professor of religious studies and Judaic studies at Brown University and the author of How the Bible Became Holy [1] (Yale University Press, 2015).
Links
[1] https://yalebooks.yale.edu/book/9780300171921/how-bible-became-holy