New projectome captures ‘complex beast’ of serotonin system in mice

The mouse brain’s expansive serotonin system—which plays a role in everything from mood to movement—is composed of five distinct groups of neurons that target functionally related brain regions, according to a study published last month in Cell. 

The research provides the first “projectome,” or whole-brain map of connections, among serotonin neurons in a vertebrate, offering new insights into the system’s underlying organization. An independent team mapped the entire serotonin system in a roundworm in 2023.

The field has long known that serotonin neurons vary, but understanding how and where they connect within the brain was “pretty hazy,” says Jeremiah Cohen, professor of neuroscience at the University of Minnesota, who was not involved in the study. 

The new map reveals the nature and extent of these connections, he says, serving as “the anatomical scaffold that we can all use to try to understand this system more deeply.”

The team behind the work used viral-genetic tracing and whole-brain imaging to identify serotonin neuron projections from the dorsal and median raphe, building on 2018 findings in mice. For the new work, supported by the BRAIN Initiative, the researchers injected mice with a virus designed to make connected serotonin neurons light up throughout the brain, and they repeated the process until they had imaged all of the areas the neurotransmitter affects.

Their analysis of the images revealed five distinct projection regions among serotonin neurons: the hippocampal-entorhinal network, the basal ganglia, the cortical regions, the medial interbrain (composed of the medial thalamus and hypothalamus), and the brainstem and lateral thalamic nuclei.

These groupings suggest that the serotonin system is organized by functional relatedness rather than proximity to its targets, the researchers say.

Yet even among related brain regions, the projectome showed some unexpected disconnects. For example, the central amygdala and nearby basolateral amygdala, which both help regulate fear learning and emotion, belong to separate groups—a difference that tracks with their separate developmental origins and cellular makeup.

“It’s really giving us an overview of how serotonin views the brain,” says study investigator Liqun Luo, professor of neurobiology at Stanford University. The serotonin system is “a complex beast,” he says, and these connections help “define the rules of this complex world across multiple dimensions.”

M

Machine-learning models trained on either transcriptomic or spatial features alone identified the projectome groups with 41 percent and 58 percent accuracy, respectively. But combining these data yielded significantly improved predictions, suggesting that both expression and location offer important information about the projectome’s organizational structure. “It’s not a simple map,” Luo says. 

The five projectomic groups also influence behavior in overlapping but distinct ways, experiments in mice lacking serotonin in different projection targets revealed.

The basal ganglia group, for instance, seems to influence repetitive behavior—consistent with research connecting this region to obsessive-compulsive behavior—whereas the cortical regions, the medial interbrain, and the brainstem and lateral thalamic nuclei shape anxious and repetitive behaviors.

Several results differed between male and female mice, producing distinct or even opposite behaviors, suggesting that the function of some serotonin groups is sex-dependent.

These differences in behavioral outcomes give more credence to the idea that each projectome group does different things, says Jean-Claude Béïque, professor of cellular and molecular medicine at the University of Ottawa, who was not involved in the study. “The map has now been set,” he says. “The trick going forward is: What is that different thing?”

In addition to helping researchers disentangle serotonin’s role in behavior, this research also raises interesting questions about why the serotonin system follows these functional patterns and how early experiences might rewire its connections, Luo says. Further research might uncover additional subdivisions within the five projectomic groups, revealing other connections and organizing principles, he adds.

“This [projectome] will be a gold mine for hypothesis generation,” Cohen says.