DEB Virtual Office Hour: Merit Review and How to Get Involved with NSF

Virtual Office Hours: Merit Review and How to Get Involved with NSF

On Monday, June 10^th, 1 – 2pm ET DEB held its Virtual Office Hour: Merit Review and How to Get Involved with NSF. DEB Program Officers discussed what to expect when you are a panelist or reviewer and opportunities to get involved with NSF that don’t involve grant writing. Upcoming DEB Virtual Office Hours are announced ahead of time on DEBrief, so we suggest you also sign up for blog notifications. 

If you couldn't make it to this or any future office hours, don’t worry! Come back to the blog afterwards, as we post recaps and the presentation slides of all office hour sessions. Alternatively, visit our Office Hours homepage for slideshows and recaps of past topics. Slides are made available below. 

Virtual Office Hours are on the second Monday of every other month from 1 – 2pm ET. 

Upcoming Office Hours and Topics: 

August 12: Graduate Research Fellowship Program

October 21: Dear Colleague Letters

December 9: TBD

Presentation outline

Division of Environmental Biology NSF staff in attendance today: 

• Jeremy Wojdak (host) - Population and Community Ecology
• Maureen Kearney – Systematics and Biodiversity Science
• Sam Scheiner – Evolutionary Processes
• Jason West – Ecosystem Science

Resources:

• DEB Virtual Office Hour DEB Office Hours: second Monday of each month, 1-2pm Eastern
• DEB Blog posts upcoming topics, registration, and recap posts: https://debblog.nsfbio.com/office-hours/ 
• Sign up for BIO News and Updates
• Volunteer to review DEB: https://www.surveymonkey.com/r/DEBexpertise 
• “The art and science of reviewing proposals” NSF Video
• Find recent BIO funding opportunities

Learning the “culture” of NSF

Formal documentation 

• Proposals & Awards Policy and Procedures Guide (PAPPG)
• Solicitations
• Dear Colleague Letters
• Research.gov/ Fastlane systems 

NSF outreach

• Blogs
• Virtual Office Hours
• Conferences
• Individual meetings with program officers 

Participation

• Proposal submission
• Receiving reviews/ panel Summaries
• Award management including reports, requests, supplements, etc. 
• Writing ad hoc reviews 
• Panelist service

Integrity of the review process

• Conflicts of Interest - Your responsibility to identify COIs, both institutional and individual. Also, even if no official conflict exists, be honest and open if you don’t feel you can be objective. 
• Confidentiality – Do not share proposal information.
• Bias – Many flavors, many origins... but awareness is first step in mitigation.

Reduce cognitive biases 

• Actively reflect on your own thought process
• Think of alternative views 
• Play a devil’s advocate
• Take time with your decision

Merit Review Process Overview

1. Review panelists and ad hoc reviewers 
2. Panel discussion
3. Panel summary and panel recommendation
4. Division directorate review 
5. Business review and processing
6. Notification of award decision

Merit Review Criteria

• Intellectual Merit: The intellectual merit criterion encompasses the potential to advance knowledge;
• Broader Impacts: The broader impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes. 
• Keep in mind: 
  • Is this exciting science? 
  • Will it work? (is the risk/reward appropriate?)
  • Can this team do this work?

Basic Reviewer Guidelines

• Use a structured format, highlighting strengths and weaknesses in intellectual merit and broader impacts
• Focus on the most important aspects... Not every detail like in a manuscript review.
• Make sure the narrative review and proposal rating align: 
  • Poor: Proposal has serious deficiencies
  • Fair: Proposal lacking in one or more critical aspects; key issues need to be addressed 
  • Good: A quality proposal, worthy of support 
  • Very Good: High-quality proposal in nearly all aspects; should be supported if at all possible 
  • Excellent: Outstanding proposal in all respects; deserves highest priority for support 

Advice on Broader Impacts 

• There is no single formula or checklist. Proposals should provide a rationale for chosen broader impacts, describe measurable outcomes, and match the resources (time and money) allocated. Go above and beyond what principal investigators are already paid to do.
• Budget should reflect the proposed broader impacts activities
• Use existing infrastructure, as appropriate, but give, as well as take 
• Assessment of outcomes is ideal though not strictly required. 

Consult Center for Advancing Research Impact in Society (ARIS) for additional resources: https://www.researchinsociety.org/ 

Reviewing Traps 

• Mentally re-writing the proposal yourself 
• Summarizing the proposal
• Giving a pass to underdeveloped plans by great/experienced PIs
• Favoring charismatic systems
• Commenting on things you don’t understand/know about
• Favoring quantity over quality in broader impacts
• Expecting novelty in broader impacts
• Piling on the negatives
• Describing, rather than evaluating

Special criteria

In addition to intellectual merit and broader impacts, sometimes reviewers must consider solicitation-specific criteria:

• Vary by solicitation, but may constrain scope, content, and participants (e.g., RUI, CAREER, OPUS, LTREB). 
• You just want to measure the proposal by the correct standard.
• NSF will provide guidance to reviewers for each solicitation.

As a reviewer, you help shape the community norms… 

NSF defines the “boundaries” for parts of a proposal, while the community helps construct the norms through their perspectives in reviews: 

• Broader impacts – Which are most valuable? How much is “enough”? What are the emergent best practices? How should broader impacts be weighed versus intellectual merit? 
• What constitutes a safe and inclusive field plan (SAIF)? How should data be managed and shared (DMP)? How are postdocs best mentored (PMP)?

Reviewing outside of DEB 

Opportunities for DEB panel and reviewer service are limited by the number of proposals/panels we have.

It is important to look outside of DEB as well, including: 

• Graduate Research Fellowship Program (GRFP) 
• Division of Biological Infrastructure
  • Postdoctoral Research Fellowships in Biology (PRFB)
  • Research and Mentoring for Post-bacs (RaMP)
  • Other infrastructure stuff
  • Postdoctoral Research Fellowships in Biology (PRFB)
• Directorate for STEM Education
  • Historically Black Colleges and Universities - Undergraduate Program (HBCU-UP)