NSF invests in reviewer development to advance innovation

The U.S. National Science Foundation's merit review process helps the agency identify high-quality proposals that merit NSF funding. This winter, the NSF Division of Civil, Mechanical and Manufacturing Innovation (CMMI) is expanding a pilot that trains experts who participate in NSF merit review through review panels.

The program called the Game Changer Academies for Advancing Research Innovation aims to improve group dynamics during panel discussions, increase awareness of bias and identity, and enhance understanding of high-risk, high-reward ideas. Once trained, "Panel Fellows" will bring enhanced skills and awareness when they participate in panel discussions during the NSF merit review.

"The Game Changer Academies are part of our effort to make sure that CMMI supports the nation's economic prosperity and safety and taps into the diversity of engineering and science researchers we have across the country," said CMMI Division Director Rob Stone.

Researchers from academia, industry and national laboratories at all career stages can apply to join the 2022 cohort; applications are due by Oct. 15, 2021. The Game Changer Academies trained 150 researchers in the 2021 cohort and expects to train 200 more in the 2022 cohort.

Merit review

Each year, NSF receives thousands of proposals to fund research and education. To ensure that proposals are evaluated in a fair, competitive, transparent and in-depth manner, NSF uses a rigorous merit review system.

Nearly every proposal is evaluated by a minimum of three independent reviewers consisting of scientists, engineers and educators who do not work at NSF or for the institution that employs the proposing researchers. NSF selects the reviewers from among the national pool of experts in each field, and their evaluations are confidential. Reviewers come together in a panel and discuss the strengths and weaknesses of proposals.

"The Game Changer Academies do not provide training in how to be a panel reviewer; NSF and our program directors do a great job of explaining that role already," said Stone. "Game Changer Academies training addresses how to improve and elevate the panel review discussion so that we are focused on finding those diamonds in the rough that are the high-risk, high-reward research we should fund."

Game changers

CMMI Game Changer Academies (CGCA) grew out of a 2019 NSF-funded pilot project aimed at increasing reviewer risk tolerance. This pilot identified five key levers associated with the success of panel discussions:

• Defining high-risk, high-reward ideas.
• Promoting effective group dynamics.
• Mediating cognitive bias.
• Navigating social identity.
• Engaging in difficult conversations and conflict.

As the need for innovative and collaborative science grows, reviewers need more tools to enable broad thinking and to identify potentially transformative ideas. In response to this need, CGCA launched in 2021.

At the core of the program is the awareness that, without deliberate attention to grant review, discussions can sometimes fail to engage the full perspective and experience of panelists. CGCA develops the capacity of panel discussions to benefit from high-quality interaction across differences in perspective, identity, experience and expertise.

"Better discussions contribute to better decisions. In order to respond with the speed and efficiency needed in today's world, we need processes that function at their best. In CGCA, we're looking at the process of group deliberation," says Diana Kardia, lead CGCA presenter and designer.

Kelly Mack, CGCA co-presenter and designer, adds, "We are persuaded that if we improve how we hold the space for productive discussions, then our panels will be more able to significantly advance our disciplines."

CGCA uses a flipped classroom methodology aimed at harnessing the capacity of scholars to apply theory and research to practice. After reviewing material associated with that month's topic on an asynchronous content site, Panel Fellows meet with a trained discussion leader in a small group format to discuss that topic and its relevance to panel discussions. The main sessions then pull the full cohort together to provide further elaboration and application through case studies, simulations and other pedagogical tools.

The 2022 CGCA program will launch on Dec. 10, 2021. Discussion sections and main sessions will then be held monthly from January through May 2022.