The following information is mentioned in or related to the reviewer orientation:
Helpful hints
If you have been asked to review one or more proposals for NSF, please watch the reviewer orientation video before you begin to read the proposals. Some of helpful hints are listed here. The context for them is provided in the video.
-
Read the merit review criteria before you read the proposal(s), decide how you will apply the criteria, and stick to them.
-
Do not shift your criteria as you go from evaluating one proposal to the next, and do not include extraneous data or criteria.
-
Take notes when reading the proposal.
- Do not include a lengthy summary of the proposal in your review!!!
- Be constructive in your feedback; is this the type of review you would like to receive?
- List strengths and weaknesses with respect to the review criteria.
- In the summary section of your review, tell us whether or not you believe the proposal is competitive and why.
- Include concrete examples from the proposal in support of the points in your review.
- Look for signs of the impact of cognitive biases in what you write and strive to mitigate these.
- If you are reviewing multiple proposals, are your reviews consistent and objective?
- Think of alternative views and consider whether they are justified based on facts.
- Play a devil’s advocate to your own assessment.
- Review your notes.
- Take time, pause, and reflect on your recommendation.
- Critically read each review after you have written it; ask yourself whether each judgment is clearly justified in the text of the review.
- Be accountable to yourself and imagine justifying your decision to others.
Additional resources
Books by Researchers Intended for a General Audience
D. Kahneman (2011). Thinking, fast and slow. Farrar, Straus & Giroux.
M. Banaji and A. Greenwald (2013). Blind Spot: Hidden Biases of Good People. Delacorte Press.
Research Articles
R.S. Nickerson (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175- 220.
R.E. Nisbett and T.D. Wilson (1977). The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology, 35(4), 250- 256.
C.A. Moss-Racusin, J.F. Dovidio, V.L. Brescoll, M.J. Graham, and J. Handelsman (2012). Science faculty’s subtle gender biases favor male students. Proc. Nat. Acad. Sci., 109, 16474 – 16479.
R.E. Steinpreis, K.A. Anders, D. Ritzke (1999). The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: A national empirical study. Sex Roles, 41, 509-528.
C. Goldin and C. Rouse (2000). Orchestrating Impartiality: The Impact of "Blind" Auditions on Female Musicians. Am. Econ. Review, 90, 715-741.
M. Bertrand and S. Mullainathan (2004). Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. Am. Econ. Review, 94, 991-1013.
J.F. Dovidio and S.L. Gaertner (2000) Aversive Racism and Selection Decisions: 1989 and 1999. Psychological Science, 11(4) 315 – 319.
Lev-Ari, S. & Keysar, B. (2010). Why don't we believe non-native speakers? The influence of accent on credibility. J. Expt. Social Psych. 46, 1093 – 1096.
