Inconsistent analysis protocols in the data lab

Starting point

A publicly funded research network runs three data labs with around 40 data scientists and analysts in total. Roughly 2,600 analyses are carried out each year, about 50 per week on average. Every analysis requires a protocol covering data sources, preprocessing steps, model variants and results. Until now, these protocols have been written in different templates and tools, from Markdown notebooks to Office documents. A typical analysis involves two to three iterations, each with its own notes and interim results, which later have to be painstakingly merged into a consistent report.

The challenge

The head of the data lab found that, despite high scientific quality, the documentation of analyses varied widely. Important context such as the data slices used, feature engineering decisions or abandoned modeling paths was often missing or recorded only in personal notes. For replication requests, data scientists spent one to two hours per case on average understanding old protocols and filling gaps. Peer reviews took longer because reviewers first had to get used to individual writing styles and structures. At the same time, pressure was growing to make results easier to compare across projects and sites.

Why this remained unsolved

The problem had long been known but was accepted as necessary overhead. Standardized templates were introduced but only partly used, because they were seen as too rigid and did not fit every project. A classic reporting tool would have enforced additional mandatory fields but hampered the creative analysis process. There was no capacity for a dedicated IT project, and the data labs did not want to maintain yet another rigid application. So it remained a matter of recommendations and training sessions, which were often ignored under day-to-day time pressure.

Together with the data lab, we developed an assistance system that automatically generates structured protocol drafts from existing artifacts such as notebooks, pipelines, tickets and short free-text notes. The system reads metadata from version control, experiment tracking and the data catalog, links it with the data scientists' comments and proposes a consistent structure. Researchers stay in their familiar workflow, for example in the notebook or the project management tool, and receive suggestions there for the introduction, methodology, experiments, results and limitations. They can adapt, extend or discard these drafts, so they keep control over content and tone.

Where generative AI comes in

Generative AI analyzes code comments, notebook cells, experiment logs and short free-text notes, condenses them into readable sections and suggests suitable headings. It recognizes typical analysis components such as data preparation, model training or evaluation and maps them to a uniform structure that enables consistent protocols.

Transferability

The pattern suits any organization where analytical or experimental work happens in many variants and documentation often lags behind. Wherever notebooks, scripts or lab journals are produced, an assistance system can generate structured protocol drafts from existing artifacts. To ensure quality and acceptance, it is important that experts keep ownership of the content and use drafts only as a starting point.