Talmo Pereira *21’s Software Tracks the Motion of Living Things

Inside the towering walls of the Salk Institute for Biological Studies in San Diego, a maze-like corridor leads to dark rooms chock-full of glowing screens. Some contain video streams of animals overlaid with colorful points, others contain streams of code. Students at the workstations are eager to explain what they are doing, and what the cryptic images and symbols mean. “It says a lot about the lab culture that the people who start as research assistants often stay on to become grad students,” says Elise Davis, a Ph.D. student who followed that very path.

Leading the whole operation is assistant professor Talmo Pereira *21, whose contributions to neuroscience are incongruously weighty for someone who seems young enough to be a student himself. After a childhood spent playing computer games to avoid the crime in his neighborhood in Brazil, Pereira has gone on to develop software that precisely and automatically tracks the motion of living things, with applications ranging from fighting climate change to characterizing early ALS.

“Behavior is a lens through which we can view and interpret activity in the brain,” says Mala Murthy, director of the Princeton Neuroscience Institute and one of Pereira’s Ph.D. advisers. “And Talmo has played a pivotal role in the development of methods to quantify behavior.”

As an undergraduate at the University of Maryland, Baltimore County, Pereira was selected to be part of the prestigious Meyerhoff Scholars Program, which Pereira says is “one of, if not the most successful program aimed at expanding access to STEM careers.” The program guided him toward internships at the leading edge of neuroscience and computing at the Broad Institute and Caltech. As a result of the internships, “I knew exactly what I wanted to do — use machine learning to create new ways of quantifying animal behavior — before I knew where I would go to graduate school,” says Pereira, who ultimately chose Princeton because “this Hogwarts-looking place had seemingly unlimited resources, a super positive and open culture, and faculty with forward-thinking approaches and perspectives on neuroscience.”

When Pereira began graduate school in 2015, motion capture was mainly done by placing reflective markers on things like joints. “To animate Gollum in The Lord of the Rings, Andy Serkis probably wore a suit with the reflective markers at the joints. Fruit flies are not huge fans of motion capture suits,” Pereira jokes, referring to the first animals he studied. Pereira and an undergraduate in the lab, Diego Aldarondo ’18, now a senior applied scientist at the Amazon company Fauna Robotics, created a deep learning software, LEAP Estimates Animal Poses, that can identify animal parts and poses from video. (A later version, SLEAP, can do this for multiple animals in a video.) “They realized they could create a smaller, faster algorithm that required far less training data than, say, what Google was doing with human pose tracking,” says Josh Shaevitz, a biophysics professor at the Lewis-Sigler Institute for Integrative Genomics, and one of Pereira’s Ph.D. advisers. “To track animal body parts with minimal training time was a breakthrough that made the technology accessible for widespread scientific use.”

In November 2021, Pereira became a Salk Fellow, which let him immediately set up his own lab and skip years as a postdoctoral researcher, the most common next step for a scientist after graduate school. In collaboration with Salk’s Harnessing Plants Initiative, SLEAP has been used to see which genetic variations in plants result in deeper, more massive, and widespread roots that can capture more carbon dioxide. Pereira describes the current setup as a “super-scale pipeline, with video of tens of thousands of plants to extract over a thousand traits per plant.” Pereira goes on to explain that if we could double the amount of carbon dioxide captured by the main agricultural crops, this would exceed the total amount currently emitted worldwide.

Pereira’s lab is also using SLEAP to detect the earliest signs of the devastating and fatal disease ALS. With funding from Project ALS, Pereira’s lab has now amassed over 100,000 hours of video of mice and can tell around one month of age whether a mouse will go on to develop the disease. “Their caretakers — people who have worked with these mice their entire careers — have told us they cannot tell the difference between the sick and the healthy mice until months later, and even then, it’s not through their daily behavior, but through specific assays, like testing their grip strength, for example,” says Davis. This ultra-early detection in mice has implications for the search for ALS drugs: “This is going to enable us to do large-scale screening of therapeutics in a way that wasn’t possible to do before,” says Pereira.

He says that what we really need is “a system that can represent the biological structures and the [brain] computations that give rise to movement,” but linked to a living thing. Such systems are called digital twins, and Pereira believes this is the future of neuroscience — testing hypotheses and situations on virtual animals. “In a very deep sense, I am still that 11-year-old tinkering with computers and building agents that can level up in the game,” he says.

Pereira’s self-deprecation and easy sense of humor belie the enormous impact of his work. SLEAP is used by 50,000 researchers in 93 countries, with Pereira’s lab offering support to everyone from sixth graders to university faculty. “Sixth graders are much faster at picking it up,” he laughs.

Pereira’s brilliance extends beyond computing. “He is probably one of the most motivating people I’ve ever worked with,” says Aldarondo, who worked with Pereira for three years on LEAP. “His passion is infectious, and he makes it easy to get excited about neuroscience and computational methods.”