Mobile video game submerges users into ocean depths
NSF-supported project is part of a broader effort to leverage AI and machine learning to increase our knowledge and help monitor the health of the ocean ecosystem
Exploring most of the world's oceans is technologically challenging, however, researchers are making progress and collecting more data about oceans than ever before. Cameras on advanced underwater robots have helped scientists discover new species, observe marine environments and monitor ocean health.
Organizations such as the Monterey Bay Aquarium Research Institute (MBARI) have amassed an enormous archive of deep-sea video. However, converting underwater visual data into species identifications and population estimates is a daunting task. It can take expert researchers three to five times longer to review the images than it takes to collect them. When assessing the data needs of the ocean exploration community a few years ago, researchers estimated that U.S.-based organizations have collected hundreds of thousands of hours of underwater video, but very little has been processed.
With the help of artificial intelligence, science enthusiasts and video gamers can expand the knowledge of the ocean exponentially.
Swimming in scientific data
AI can help researchers analyze this deluge of visual data, which can be used to inform scientists, policymakers and ocean enthusiasts everywhere. But before AI can contribute to the knowledge of the ocean, it needs to be trained.
To address this challenge, researchers developed FathomVerse. The mobile game combines immersive imagery, gameplay and cutting-edge science, allowing players to interact with real underwater images. By engaging ocean enthusiasts worldwide to review and label images, the team hopes to improve the AI that helps researchers study marine life and assess ocean health.
"There are very few experts — ones that can look at an image and tell you exactly what that animal is — so we have to reach out and encourage participation from the spectrum of people with knowledge and interest," said Kakani Katija, a principal engineer at MBARI who led the development of FathomVerse. "The question was: How do we bring that expertise into the fold and contribute to this process? FathomVerse is our way to engage the enthusiast audience, and players are contributing directly to ocean science by evaluating and verifying AI proposals. Data from the game is then shared with the expert community and are subsequently used to train new models."
To develop FathomVerse, MBARI researchers collaborated with Netherlands-based game studio &ranj Serious Games, Barderry Applied Research, who provide expertise in human-AI interactions, and Internet of Elephants, a nature tech enterprise based in Kenya focused on rekindling relationships between people and wildlife. The team spent 18 months developing FathomVerse. After a beta launch in 2023 that drew nearly 1,500 players from around the globe and generated about 450,000 annotations, the app launched publicly on May 1, 2024. One month after the launch, FathomVerse had welcomed 8,000 players from 100 countries who logged more than 3 million annotations.
"We are trying to promote and create an inclusive culture around sharing of data in a community that needs to do this better," Katija said. "Game users see FathomVerse and what we're trying to do with the data to discover and observe ocean life, and they see this as a good use for AI and want to support that."
NSF helping tackle challenges related to the blue economy
FathomVerse is one of three software tools developed by MBARI, CVision AI, Ocean Discovery League and Purdue University as part of the Ocean Vision AI program supported by the U.S. National Science Foundation to take advantage of the massive amounts of images and video to study marine life and assess ocean health.
In addition to FathomVerse, Ocean Vision AI has released FathomNet, a database of images and machine learning models that can be used to identify ocean animals; the Portal, an online, collaborative tool for end-to-end AI-assisted processing of ocean imagery, is scheduled for its debut later in 2024.
Ocean Vision AI was one of six multidisciplinary projects funded under the $30 million Phase 2 of NSF Convergence Accelerator Track E: Networked Blue Economy. The focus of the track includes interconnecting the blue economy and accelerating convergence research across ocean sectors.
With NSF funding, MBARI was able to bring in strategic partnerships and new perspectives that strengthened the organization's approach and subsequent solutions. MBARI established partnerships with institutions that collect visual data, ones that focus on data governance, and with organizations such as Monterey Bay Aquarium to help engage audiences.
"In addition to NSF funding, the Ocean Vision AI team receives intensive training and support from the NSF Convergence Accelerator program team to support teaming activities and to ensure end users are engaged in developing an innovative project such as FathomVerse," said Aurali Dade, NSF Convergence Accelerator Track E: Networked Blue Economy program director. The Ocean Vision AI team fully leveraged the program features to develop outstanding products during the course of the three-year award."
"The pressure to not only be able to observe ocean life and fill observational gaps comes from climate change and the expansion of activities around the blue economy," Katija said. "As more industries and institutions look to utilize the ocean — traditionally we think of fishing and oil and gas, but more recently aquaculture, offshore wind and even deep-sea mining are increasingly cited — there's an increased need to understand how these growing activities impact ocean life."
And FathomVerse has been built for growth: With feedback from players and data contributions from organizations and groups collecting visual data in the ocean, more minigames, data interactions, animal groups and features can be incorporated into future game versions. Players share findings, animals and feedback with researchers and other fellow players on the FathomVerse Discord server.
"A lot of people want to help the ocean," Katija said. "I'm really excited that FathomVerse invites people to directly help the ocean. This is especially important as people are looking more to the ocean for solutions to environmental problems and economic opportunities. For these new uses to be effective and sustainable, we need to better understand their impacts on the environment and whether the potential economic benefits are worth it. FathomVerse players are helping us create AI to help monitor the ocean so that resource managers can make informed decisions on what is sustainable and what is not."
