To keep or not to keep: Neurophysiology’s data dilemma

As a neurophysiologist working with Neuropixels probes, I have encountered firsthand the data-management challenges discussed here. Our recent study, recording control and epileptic animals for up to three weeks almost continuously, generated a unique dataset of approximately 1 terabyte. This massive volume of data required us to design a specialized data transfer and storage system, highlighting the practical challenges of data management in modern neuroscience.

The dilemma of whether to retain raw data or keep only processed versions is particularly relevant to our work. My experience as a neurophysiologist has shown that revisiting raw data often leads to new insights and results, as novel analysis methods emerge over time. But sharing such large datasets—important for supporting open science—presents significant challenges. An optimistic estimate of the time required to download 1 terabyte of data over a typical high-speed internet connection (assuming 100 Mbps) would be approximately 22 hours. This time frame makes it impractical for most researchers to access and work with such datasets remotely, potentially limiting collaboration and reproducibility. Balancing the need for data preservation with accessibility and open-science principles remains a critical challenge in this case.

As a computational neuroscientist who aims to develop new analytic tools to uncover latent structures or representations of large-scale neural data, I always feel that it is crucial to be able to access both raw and processed data, along with well-documented metadata. First, the ability to examine all relevant signals and “noise” in the raw format may enable us to test new methods and hypotheses and to discover new structures or features from commonly conceived “noise.” Meanwhile, having preprocessed data at hand also gives us the opportunity to directly compare with the “norm.” In practice, I found that it is a good idea to have some representative datasets in both formats and run comparative analyses on both forms. Metadata becomes critical for mining data from a public repository, especially when there are no collaborating experimentalists to troubleshoot or direct questions regarding data collection and experimental details.

The answer to the question of what data to save may depend on the nature of data. For behavioral and calcium-imaging data, for instance, keeping the raw format at the highest temporal resolution is preferred. For standard EEG or local field potential recordings, it may be more convenient to keep data at a lower sampling rate.

Finally, the development of foundation models based on multimodal neural recordings would benefit from having access to both raw and processed data. Processed data can be viewed as the first-stage feature extraction of raw data. But it remains untested whether the processed data are the optimal information carrier for self-supervised learning paradigms commonly used in large language models and foundation models. The use of raw versus processed data in foundation models may also depend on the downstream tasks of interest. Overall, I can see the value and need for keeping raw data based on scientific questions.

Open-data initiatives are transformative, and we fully support advancing in this direction. But as experimentalists, we face some challenges worth discussing. In our lab, we put significant effort into sharing data and code in a way that is both meaningful and usable. Comprehensive documentation is crucial, as different types of experiments—whether recording single cells, using silicon probes or deploying Neuropixels—often require unique setups and custom-built solutions for syncing devices properly. This results in data files with varying formats, lengths and structures, all of which need careful curation to ensure they can stand alone for reuse and sharing. Given this, the effort invested in thorough data storage and metadata generation can be substantial, requiring us to carefully prioritize our resources.

Understanding the nuances of the data is also vital. The brain’s complexity introduces many sources of variability that are crucial for generalizing findings. Factors such as sleep, sex, hormonal status, genetic background and other environmental influences—including the relationship between animals and the experimentalist or how experiments are run—can significantly impact results, sometimes in subtle ways. These factors are not drawbacks but essential aspects of the scientific process. In some cases, the costs and effort of storing fully documented data outweigh the benefits of rerunning experiments, especially when the focus is on driving new discoveries.

One argument in favor of preprocessing can be built around the fact that raw experimental data often contain artifacts due to motion, instrumentation noise and instability—crosstalk between the measurement modalities and unwanted biophysical processes. An example of the latter from neuroscience is hemodynamic darkening in fluorescence images, caused by an increase in light absorption by hemoglobin during the hemodynamic response. For experimentalists, the ability to recognize and tackle these problems is acquired through years of training. In many cases, these artifacts can be removed by preprocessing. In the example of hemoglobin absorption, fluorescence intensity can be corrected by estimating the change in hemoglobin concentration and scaling the signal appropriately. Other examples are electrical interference and photoelectric effect in experiments that combine electrophysiological recordings with optical imaging or photostimulation. In these cases, linear decomposition methods such as principal component analysis and independent component analysis are often used to isolate and remove the artifact. Without preprocessing, such artifacts may dominate the variance of the data, making it practically useless for training AI models and other computational applications. In other words, this is a case in favor of preprocessing conducted by experts who understand the nature of data acquisition.

It is often difficult to foresee all possible applications of a dataset. For example, most researchers recording extracellular potentials with multi-contact electrodes, such as modern Neuropixels probes, are interested only in the spikes that contain the high-frequency part of the signal. They may thus, for practical reasons, be tempted to store only high-pass filtered signals or even just the time points of detected spikes extracted by application of spike-sorting algorithms. But our group uses this kind of data to constrain neural network models. For this alternative use, the low-frequency part of the potential, the local field potential, is key. Thus, access to raw data is essential for us.

Any kind of processing of raw data removes information, so access to raw data is always preferable. But if only processed data can be stored, it is essential that the processing procedure is described in detail so that the same procedure can be applied, for example, to virtual data obtained from biophysical simulations. Only then can a quantitative comparison across experiments be performed.

Data management is a growing concern for experimentalists, especially as datasets from technologies such as Neuropixels (about 50 gigabytes) and calcium imaging (about 30 gigabytes) continue to expand. When I started as a graduate student in 1992, we recorded 288 kilobyte data files using window discriminators connected to oscilloscopes, capturing spike counts and rat positions during 16-minute sessions. A decade later, we moved to tetrode-configured electrodes, generating 50 megabyte raw data files, from which we could isolate single units and create compact files of only 100 to 200 kilobytes per neuron. Today, we routinely extract 200 to 500 single units from Neuropixels data and store them in compact time series files (about 20 megabytes) for analysis, rarely revisiting the original raw data.

Despite this, storing raw data is essential. On multiple occasions, I’ve identified improvements or corrected errors years after data collection—such as improving single unit discrimination by 10 percent through resampling with cubic splines or detecting timing errors in data acquisition. These cases were possible only because we had preserved the raw data. The cost of storing these files is negligible compared with the human and financial resources required to redo experiments. For example, a 64-channel silicon probe costs about $1,000, the same price as 64 terabytes of storage, which can hold approximately 7,000 hours of raw recordings.

Although the costs of raw data storage are low, the burden of managing such data long term exceeds the capabilities of most labs. As a community, we need to adopt standards, such as Neurodata Without Borders, and formats, such as Multiscale Electrophysiology Data, to ensure data are both preserved and accessible. Transitioning from a culture of self-sufficiency to one of shared responsibility will require institutional infrastructure, funding and widespread adoption of these standards, ensuring that our datasets can drive future discoveries.

Being able to link the validity and reliability of findings to underlying raw data is a cornerstone for rigor and reproducibility in scientific research. In the realm of neuroscience, this is no different. But the complexity of neuroscientific raw data and metadata—and their provenance places significant resource challenges on data producers to make neuroscientific collections FAIR (findable, accessible, interoperable and reusable) for both people and machines. The diversity of instruments, species and experimental questions is far too great and provides a sparse smattering of bits of neuroscientific knowledge.

In this universe, given the increasing amount of data being collected and now disseminated, it will eventually become imperative to make informed decisions about which data to retain and which to discard. I am one of the maintainers of the DANDI neurophysiology data repository, which is housing close to 1 petabyte of data. Though we have a responsibility as a repository to archive these data, we do need to ask: Are all of these data equally useful?

There are several factors that make this a challenging question. Here, I discuss a critical few that we should consider when determining which neuroscientific raw data to keep.

First, the research question and study design are of paramount importance. The data that are relevant to the specific question being investigated should be prioritized. Exploratory analyses and pilot studies may require a more comprehensive collection of data and metadata, whereas confirmatory studies can often focus on specific aspects of the data that are directly related to the hypotheses being tested. Any data for which most of the knowledge has been extracted could be considered less important than data that offer more possibilities of reuse. Thus, an automatic sunsetting system could be placed on data. For example, instead of infinite archival, we could say you have five years. If in those five years we discover other untapped uses, we can extend the sunset period.

Second, the quality of data and metadata, including what is missing, is another crucial consideration. Neuroscientific raw data can be affected by noise, artifacts and technical issues. The data we store have to be of the highest quality in order to make reuse easier and findings more robust. Yet the community has not converged on methods for quality assessment and control, especially in the context of new instruments across the breadth and depth of neuroscience. Adopting common quality-control methods will be an important step toward systematic evaluation and eventual sunsetting of data.

Third, several kinds of biases can affect neuroscientific data. Given the diversity of neuroscience approaches, some datasets may come from a single researcher and others from thousands of researchers. Some may represent specific brain regions or circuits, and others may represent specific cohorts or socioeconomic groups. When preserving data, it is important to consider who or what aspects of neuroscience have not been represented. When the same genetic model, or cell lines, or the same brain sample is used, it may limit the generalizability of findings.

We need to keep in mind that not all data are created equal. Every dataset comes with a rationale that led to its generation. Some datasets in neuroscience are based on specific hypotheses, and others are related to more open processes of creating diverse and large biobanks. Deciding which neuroscientific raw data to keep involves careful consideration of the research question, data quality, diversity and ethical principles. It should also take into account the generation of future datasets. Such a universal decision-making approach, however, does not currently exist. Neither do we need to discard data today. Logistically, I believe, we can preserve the world’s neuroscientific data production. Thus, it is more important to keep everything and use that information to improve our preservation and selection processes and policies.

Keeping raw data is key for transparency, reproducibility and enabling open-access science, ensuring that datasets can be used for multiple studies. But the sheer size and volume of raw data is growing at a rate that far outpaces what is feasible from the perspectives of monetary cost and physical storage size, and consistent ways of documenting metadata—such as an animal’s age, behavioral features and prior experience—across projects remain limited. We have taken the approach of posting all processed data, such as spike times and the position of the animal, and saving all raw data on local servers that are backed up to cloud storage facilities. I think pushing toward improved infrastructures for sharing raw data, as well as the appropriate metadata, is important, however—not only for the reasons above, but because it ensures that experimental data, which can take an enormous amount of time and resources to collect, is maximized in terms of scientific discovery.

In my experience, it’s always best to work with the raw data. First, because we can check that our results are robust to a variety of hyperparameter choices in the processing methods. Second, because methods and software for data-processing change over time, sometimes even within the course of a single project. Therefore, I want the freedom to revisit the data with new methods. Third, data-processing is sometimes biased toward the questions being studied, limiting the potential for data reusability and future discoveries. That said, if raw data cannot be stored, the best compromise is probably to publish processed data that are as close as possible to raw data and to make sure that both the data and metadata are thoroughly documented.

Finally, with neuroscience becoming more interdisciplinary than ever before, the best way to move forward is by nurturing the dialogue between experimentalists and theorists. I have benefited tremendously from this dialogue. With this kind of exchange, experimentalists will have a better understanding of the best way to publish the data for them to be useful to a wider audience. And theorists will gain a deeper understanding of how the data were collected and processed.

Neurophysiology datasets gain immense value when multiple modalities, such as imaging, electrophysiology, anatomy and behavior, are integrated. For instance, the activity of specific neurons in a mouse cortex is meaningful only when aligned with the precise stimuli, environmental factors and time stamps across different instruments. But managing this complexity requires expertise in engineering, data science and biology, making data-processing and storage increasingly challenging. This underscores the need for an accessible data ecosystem that integrates collection, processing and storage tools for neuroscientists, as the current burden on individual trainees to manage their own datasets often compromises data quality and integrity.

Our field would benefit from focusing on fewer, more meaningful datasets, with standardized collection and sharing practices akin to those in particle physics. Efforts such as the International Brain Laboratory and the OpenScope program are early steps toward this model, aiming to promote specialization and collective resources. By concentrating on maintaining high-quality, broadly useful datasets, neuroscience could better support long-term studies while reducing the pressure on individual labs to handle data management and reuse independently.

Large open-source datasets have been transformative for theorists. In the past, gaining access to data often depended on building trust with experimentalists, which created inequalities favoring larger labs and institutions. I experienced this firsthand when I started in the 2010s—at that time, being a resident theorist in an experimental lab was often the only way to test models. Now, with initiatives such as the Allen Brain Observatory and International Brain Laboratory, theorists can test their models independently, turning ideas into publications without needing direct access to experimental labs.

These open datasets also address ethical concerns around animal research by reducing the need for new experiments. Although some hypotheses still require dedicated experiments, honing models on open-source data leads to more focused, efficient testing. The Allen Institute’s OpenScope project goes further by not only enabling scientists to propose their dream experiments, but also making all resulting data publicly available after the first publication, democratizing access to state-of-the-art neurotechnology.

Data-sharing is crucial for scientific progress. Unused data are unrealized scientific potential. But effective data-sharing is best achieved through collaboration. Experimenters possess valuable background knowledge that can help collaborators avoid misunderstandings and errors in analysis. Scientifically, such collaboration fosters cross-pollination of ideas. This is essential for advancing science. As disciplines mature, diverse perspectives weave together and integrate, leading to deeper understanding. Data-sharing provides an opportunity to build infrastructure that encourages this integration. Collaborative activities such as meetings, discussions and manuscript-writing expose researchers to new knowledge and viewpoints. By contrast, one-sided use of publicly available data misses these valuable opportunities for scientific exchange and growth.

Our new era of shared data affords theorists and computational neuroscientists opportunities in many different domains. One particularly rich direction is sharing the raw data from experimental stimuli, as well as neural recordings. In the past, auditory or visual experiments were often described in a few schematics—sometimes because the stimuli were simple and parameterizable, and sometimes because sharing large stimulation files was impossible without shipping hard drives. Even for simple stimuli, the actual stimulus files could be incredibly valuable—revealing whatever number of choices were made in shaping the actual display and enabling reanalysis or, at the very least, recapitulation of the identical stimuli in models or simulations (of course, as well as for new experiments). For naturalistic stimuli, the possibilities are even more vast. There are a multitude of ways to collapse “features” in a high-dimensional natural video or audio recording. By sharing original stimulus files, researchers can run new analyses to uncover the essence of a neural code, sometimes with surprising results. There is hope in taming the complexity of natural inputs if we have the opportunity to pilot new ideas on existing datasets. The power and speed of modern data storage and transmission makes this possible, if not imperative.

Data management is a major concern for experimentalists, especially for principal investigators who must balance data safety with the growing costs of managing large datasets. In many labs, hard drives are filling up, and cloud storage is becoming overwhelmed. Though data go through various phases—from collection to analysis and long-term storage—the preservation of raw data from successful experiments is crucial. We often don’t realize what we might have missed. For instance, though I rarely share or request raw data, my lab recently reanalyzed raw recordings and uncovered previously unknown events that were visible only at the original sampling frequency. Such discoveries, though unpredictable, could occur more often than we expect if we keep and share raw data.

Another reason to store raw data is the possibility of errors in the data-processing pipeline. If the processing is faulty, having access to the original data enables us to reprocess it as needed. And the cost of storing raw data is relatively low compared with the cost of conducting the experiment itself. For example, a 64-channel silicon probe costs roughly $1,000, which is about the same price as 64 terabytes of high-quality storage—enough to hold 7,000 hours of raw recordings. Although the cost of storing calcium-imaging data may be higher, it remains small compared with the overall investment in experimental resources.

In summary, although there are legitimate concerns about the resources required to store raw data, the benefits far outweigh the costs. Preserving raw data not only safeguards against missed opportunities for discovery but also enhances transparency and reproducibility in research. By making raw data from high-quality experiments available to the community, we can enable new analyses and accelerate the pace of scientific progress.

My lab, along with others at the Allen Institute and the Sainsbury Wellcome Centre, is pioneering large-scale experiments in which we measure high-bandwidth behavior—roughly 0.2 to 1 terabyte per hour—and brain activity—approximately 150 gigabytes per hour—continuously over weeks. These experiments produce massive datasets that are extremely challenging to manage. The sheer volume of data makes even basic analysis a task requiring significant specialized expertise. Existing methods, designed for handling just an hour of low-bandwidth data, don’t scale to these new demands, making it difficult to quantify post-processing quality and decide on efficient compression, storage and distribution strategies.

To make these large-scale experiments feasible, we need substantial investments in developing robust, scalable tools. These efforts must go beyond individual student or postdoctoral researcher projects, focusing instead on creating stable, replicable and long-term supported data pipelines. Only then can we effectively harness the full potential of these high-throughput experiments.