Letters of Recommendation: Are They Useful for Graduate Admissions?

Among admissions professionals, three key data sources are almost universally used to predict student potential within graduate programs: test scores (e.g., GMAT), undergraduate GPA, and letters of recommendation. Of these three sources, we know quite a bit about the usefulness of test scores and GPA in predicting academic success. For example, a large 2007 meta-analysis found that both undergraduate GPA and GMAT total scores were moderately (.35 for UGPA) to strongly (.47 for GMAT Total) correlated with first-year MBA grades. However, in contrast to the clear support for UGPA and GMAT scores, very little is known about the usefulness of letters of recommendation in academic settings. Yet, if prior research on letters of recommendation from the literature on employee selection is at all relevant to academic settings, the news is not very good. In short, letters of recommendation come with a host of evaluation problems ranging from low reliability (i.e., consistency across evaluators) to low objectivity and validity. As I/O psychologist Michael Aamodt is known for saying, letters of recommendation say more about peculiarities of the letter writer than they do about the applicant!

We thus recently read a 2014 meta-analysis by a Kuncel, Vannelli and Ones with great interest. These authors examined the usefulness of letters of recommendation in predicting important outcomes strictly in academic settings for both undergraduate and professional graduate programs (including medical school). This study represents one of the first attempts to empirically summarize research findings, though it should be noted that the total number of studies included is rather small (ranging from 3 to 16 studies for a given outcome), reflecting just how little research has been undertaken relative to how long letters have been used by admissions offices.

Good News, Bad news

The results of this study mirror previous research on letters of recommendation in employment situations. Key findings include that letters of recommendation:

  • show no practically meaningful correlation with graduate GPA (.13);
  • are highly correlated with applicant personal statements (.41), suggesting that letter writers are likely reflecting an applicant’s self-evaluation;
  • add less than 1% incremental validity to the prediction of graduate GPA and faculty-rated student performance beyond verbal and quantitative test scores and prior GPA;
  • significantly improve the prediction of degree attainment beyond test scores and prior GPA by some 6%.

First the bad news. Letters of recommendation do not appear to provide unique information beyond prior GPA and standardized test scores in predicting performance in graduate school – the key outcome of concern in admissions. This should be very troubling to admissions professionals because it suggests that the return on the investments in collecting, reading, and evaluating letters of recommendation is not likely to be net positive. Based on these results alone, we could only endorse the discontinuation of the use of recommendation letters in the admissions process.

Now for the good news. The one optimistic finding in the study is that letters of recommendation provide unique information in predicting degree attainment or persistence to graduation. Couple this finding with the facts that (a) test scores and GPA are only modestly related to degree attainment and (b) degree attainment is very difficult to predict accurately, and we have reason to suggest the continued use of recommendation letters. That recommendation letters are related to degree attainment suggests they may be best at capturing applicant motivation and conscientiousness – capabilities required to persist in graduate school but are not well captured by more cognitively-based indicators like test scores.

Improving the Utility of Recommendation Letters

As the study’s authors discuss, there is wide variance in the findings indicating that some methods of recommendation letters are likely better than others. In that regard, the authors offer some excellent suggestions for improving the usefulness of recommendation letters. We briefly elaborate on these below:

1. Ask letter writers to rate a small number of applicant capabilities or characteristics and focus those characteristics on motivation and persistence. Almost all recommendation forms we’ve seen ask for ratings of applicant “academic potential” or “intellectual curiosity.” Avoid including these factors on a recommendation form because they are better represented by prior GPA and test scores and, when included, are likely to reduce the overall validity of the form.

2. Use letter writer numerical evaluations, not letter readers. In general, the authors found that letter writer ratings are more predictive of success than letter reader evaluations (i.e., admissions’ personnel). In other words, don’t ask letter writers to rate applicants and then use their judgments to come to your own judgments, simply aggregate the writer’s evaluations.

3. Eliminate the use of open-ended narrative letters. A narrative letter captures what the letter writer believes is important. Admissions professionals should not leave it up the letter writer to make such a determination. Therefore, we suggest explicitly eliminating an open-ended letter in favor of a briefer and more specific set of questions for letter writers to address.

4. Use a structured coding format or rubric to score letter narratives. Decades of research have shown that unstructured and holistic evaluations are likely to reduce the effectiveness of selection decisions. Any narrative response should be coded using a structured numerical procedure such as the behavioral consistency method.

5. Don’t “read between the lines.” It is simply impossible to know what letter writers meant by omission or faint praise so avoid trying to find the message within the message.

What are your experiences with recommendation letters? Let us hear from you at BEI@depaul.edu

Trackbacks

  1. […] Click here to view the rest of the article via Letters of Recommendation: Are They Useful for Graduate Admissions? – The Business Education Insid…. […]

%d bloggers like this: