Q&A: Recently retired NCSES Director John Gawalt on the importance of surveys and statistics

The National Center for Science and Engineering Statistics (NCSES) is the nation's leading source of data on the U.S. science and engineering (S&E) enterprise, including its workforce, educational pathways, research and development (R&D) funding and performance, and innovation and outcomes. NCSES is a division within the National Science Foundation's (NSF) Social, Behavioral, and Economic Sciences Directorate and is one of the federal government's 13 principal statistical agencies.

John Gawalt served as the director of NCSES from 2012 until his retirement in early April and worked in various roles at the center for 30 years.

Q: What one thing has changed the most in the 30 years since you have been at NSF?

A: This may seem obvious, but the internet really changed the way people access NCSES data and interact with it. When I first started here, the only way you could get the data was through the mail, through a library, or by phone if you had the number for one of our analysts. A limited set of individuals used our data: academicians, policy makers and journalists. Now, with the internet, access to reports and non-sensitive data is wide open. This allows a much broader swath of the public to be involved and interested in statistics about the science and engineering enterprise. This is a good change. We were quite excited to see NCSES data used in a recent "Good Morning America" feature about the organization Black Girls Code.

This new, broader audience has in turn changed how NCSES presents data so that non-specialized users can understand and use the data for their own needs. This means an increased focus on transparency and clarity about what the numbers mean.

Q: How are NCSES data used? What is the value of NCSES' work?

A: The science policy community wants and needs access to accurate, representative data in order to develop good policy based on strong evidence. NCSES is the source for information on the science and engineering enterprise. For example, if policy makers want to increase investment in R&D, or increase federal research spending, or encourage workforce training activities, they need to access NCSES data first to understand baseline numbers and areas of need. The importance of this foundational data is evidenced in the latest omnibus appropriations bill that explicitly cites NCSES analysis as a reason to increase federal research funding for science and engineering. State and local lawmakers will also look at our data to think about the "where" and "how" of investing in research infrastructure and the contribution this makes to local and regional economies.

Q: What do you wish more people knew about federal statistics services?

A: First is the care we take with the data. Quality, transparency and privacy are deeply important to us. We take great care when we design a data collection. We think about the cost to implement a survey, the time demands on our respondents and the potential utility of the survey results. And we evaluate whether there is a need for a collection.

We develop our methods carefully and in stages. The questions we ask are all tested before they are put into the field, and it is important to understand that there is a scientific method behind developing questions. Also -- this can't be underscored enough -- we value the public as a partner and participant. When we make a promise that we are going to keep something confidential, we really, really mean that. We collect data for statistical purposes only, and we are very careful not to allow access to anyone (even other federal authorities) seeking to use our data for anything other than statistical purposes.

Q: Why are there so many federal statistical agencies?

A: We are occasionally asked why the nation has 13 principal statistical agencies distributed across the government. A few of the statistical agencies are large and have broad missions that cover a very wide swath of the U.S. economy and its population -- think of the Bureau of the Census and the Bureau of Labor Statistics. Others, such as NCSES and agencies like the Bureau of Transportation Statistics or the Bureau of Justice Statistics, are more narrowly focused and more closely aligned with the mission of their parent agency or department.

Q: What do you wish people knew about NCSES specifically?

A: There is a strong analytic component to NCSES' work, and the benefit of NCSES within NSF is that our analysis can be informed by needs that are important for our data users. There is a fair degree of nuance and terminology, particularly in S&E fields, and having a solid understanding of the context yields a substantial positive impact on the relevance of the data that get collected and presented. The value of having NCSES at NSF can be seen clearly in the development of the Science and Engineering Indicators report, where we work closely with the National Science Board to understand the most important topics and areas of focus.

I want people to know that NCSES is an approachable organization. If you have questions about what you see on our website or want to know more about our data and processes, reach out. We are always happy to connect with users.

Q: What major challenges do you see facing NCSES in the next five to ten years?

A: People who work in survey research know that response rates are declining. That decline is driving up costs. Additionally, we need to make sure that we have sufficient responses and that the responses are representative of the population we are trying to measure. To do that, we need to work harder in the face of increased reluctance of participants and often with constrained resources.

At the same time, the internet, information technology data systems, and the rise of big data and data science have led to an increase in the availability of other types of data, including nontraditional data and administrative data. These other data sources are generally less costly and can be quicker to procure (once data sharing agreements are in place). In combination, these two forces -- the increasing costs of survey operations and the increasing availability of other data -- are putting pressure on traditional survey agencies to do more with less or, perhaps, to do things differently.

Q: How does NCSES approach this evolving data landscape?

A: The thing many people don't understand is the difference in quality between these data sources. As I mentioned previously, at NCSES we think first about the need for new information, and then we think about the best way to collect it. Then we test that method of collection before launching a question on a survey. It takes time to be confident in this process. Alternate data sources can be faster, but it can be hard to have confidence that you are really measuring what you want to be measuring.

The methods used in survey research have been developed over decades, and they and the properties of the estimates they produce are very well understood. These newer data sources and associated analysis and data integration methods are not as well understood, and we have much work to do so that we can have confidence in the results and how to use them.

Examining the value and potential of these nontraditional data sources is, I truly believe, a fruitful area for research. Such research may yield new and better methods that will allow us to take advantage of nontraditional data sources, but that work must be done before our surveys can be replaced.

Q: What advice do you have to students or those just starting out in their careers in data and statistics?

A: This is an extremely interesting time to be a statistician or someone who is quantitatively inclined. The amount of information and data available to analyze today is so much greater than it has ever been in the past. And that information drives lots of decisions throughout the economy. Leaders in business and government understand the value of the data but may not yet have the ability to extract that value. Having the skill to responsibly make sense of data and extract meaning is really valuable to individuals and to society. If you like to work with numbers, to reason and to ask questions of data sets, this is a terrific time. There are many, many new opportunities.

NCSES' work is divided into several broad categories:

• The Human Resources Statistics Program conducts surveys of students and graduates related to STEM education and the S&E workforce.
• The Research and Development Statistics Program conducts surveys of establishments and organizations across R&D sectors, including business, federal and state government, academia and nonprofit organizations.
• NCSES prepares and publishes two congressionally mandated biennial reports: Science and Engineering Indicators (under National Science Board guidance) and Women, Minorities, and Persons with Disabilities in Science and Engineering.
• NCSES disseminates a broad range of reports and data on the S&E enterprise, which are available in several formats, including public use data files, data profiles and interactive tables.
• NCSES supports a grants program to enhance its efforts in advancing analytic and methodological research in support of its surveys.