Research IT · Automated cluster operations

Coordinating analysis jobs and maintenance windows in the data cluster

A research institution with 3,200 employees runs a central data cluster whose daily tasks used to be coordinated manually.

Technology
IT Operations

Initial problem

Analysis jobs and maintenance collide in cluster operations

Solution type

Agentic workflow for IT operations

Complexity

Transformation

Key metric

Fewer collisions per month

User group

IT operations and data science team

Implementation time

3-6 months

Operating model

On premises with a connected cloud service

Starting point

The institution runs a central data cluster for statistical analysis and machine learning. Around 3,200 employees across various departments use it. On average, 90 analysis jobs are scheduled per working day, and the IT team also coordinates about 40 maintenance tasks such as updates, reboots and capacity adjustments. Planning is spread across the ticketing system, email and spreadsheets. Conflicts between time-critical analyses and maintenance windows often only become visible shortly before the start, leading to aborted runs, longer turnaround times and unplanned night shifts.

90: analysis jobs per day
40: maintenance tasks per day

The challenge

Every day, the IT department has to decide which analysis jobs may run when and which maintenance tasks have to be postponed. Departments submit their requirements in varying quality, and priorities are rarely described clearly. At the same time, there are fixed regulatory maintenance windows and internal availability requirements. Coordination happens across several systems, so the operations team spends a lot of time on follow-up questions and manual consolidation. Critical jobs sometimes start during maintenance, while others are pushed back too far as a precaution. The data science team loses time waiting, while IT management can only keep track of risks and utilization with great effort.

“We often only notice conflicts when jobs start in the middle of a maintenance window and fail.”

“We spend too many hours piecing together priorities from tickets and emails.”

“We can hardly explain to management why certain analyses have to wait.”

“We feel we postpone maintenance too often for fear of complaints.”

Why this remained unsolved

The problem had long been known but was considered typical coordination overhead in a research environment. A dedicated IT project seemed oversized, yet the effort was too great to ignore. Classic scheduling tools were evaluated but failed because priorities and risks are mostly described in free-text tickets, emails and project descriptions. Full process automation was not realistic, as human decisions on regulatory requirements and research deadlines are still needed.

Calendar solution

A shared maintenance calendar was meant to make conflicts visible but was hardly kept up to date and did not show the priorities of analysis jobs.

Ticket rules

Rule-based ticket filters were meant to flag critical jobs but could not reliably interpret free-text descriptions and differing formats.

An agentic workflow was introduced that bundles analysis jobs and maintenance tasks from the ticketing system, the cluster scheduler and email inboxes. The workflow reads descriptions, deadlines and technical parameters, classifies them and suggests a sequence with minimal conflicts. For each working day, a consolidated view of planned jobs, maintenance and risks is created. The operations team receives suggestions for rescheduling, bundling and alternative time windows, can review them and approve them in a few clicks. Conflicts with regulatory maintenance windows or important project deadlines become visible early, so night shifts and aborted runs become much rarer. The workflow documents decisions and makes it clear to management and departments why certain jobs were prioritized or postponed.

Where generative AI comes in

Generative AI analyzes free text in tickets, emails and project descriptions, identifies priorities, deadlines and technical requirements and condenses them into structured decision inputs. It writes suggestions for daily plans, rescheduling and justifications in plain language and delivers consistent, reviewable recommendations.

Building blocks

Integration of the ticketing system and cluster scheduler to automatically capture daily analysis jobs and maintenance tasks.
Semantic analysis of free text, project context and technical parameters to determine priorities, deadlines and risks in a structured way.
Agentic workflow that brings jobs and maintenance together in a shared plan, flags conflicts and suggests concrete rescheduling.
Assistant interface for IT operations and the data science team, where suggestions are reviewed, adjusted and approved in a few clicks.
Automatic documentation of decisions with justifications, so management and audit requirements remain traceable at all times.

Collect signals
Every day, the workflow reads analysis jobs and maintenance tasks from the ticketing system, scheduler and designated email inboxes and assigns them to projects.
Understand context
Free-text descriptions, deadlines and technical parameters are analyzed to determine priority, duration, risk and regulatory requirements for each item.
Create plan suggestions
Based on utilization, maintenance windows and project deadlines, the workflow generates a daily plan with concrete start times and rescheduling suggestions.
Approval by IT operations
The operations team reviews the suggestions, makes adjustments where needed and approves the plan, which is then implemented in the scheduler and ticketing system.
Documentation and learning
Decisions and deviations are documented, and the workflow learns from confirmations and corrections for future planning suggestions.

Metric	Value	Basis
Collisions per monthKey metric	Down by around 70 percent	System measurementmeasured
Planning effort in IT operations	Reduced by about 35 hours per week	Time tracking before/afterestimated
Correction rate of suggestionsQuality metric	Around 15 percent of plans are adjusted manually	Samplemeasured

Effort class

Standard (60-150 person-days)

Payback period

6-12 months

Main lever

Time savings

Ongoing costs

Running the workflow, maintaining the interfaces and limited use of a cloud service for text analysis.

Limitation

The solution depends on the quality of descriptions in tickets and emails. If priorities and deadlines remain unclear, the operations team has to follow up. Critical maintenance still requires deliberate decisions; the workflow only supports them.

Connected systems

Integration with the central ticketing system, the cluster scheduler, selected email inboxes and an internal reporting tool for analysis.

Operation

Core components run in the institution's data center; the generative text analysis uses a privacy-compliant cloud service with limited, pseudonymized content.

Compliance

Access to tickets and emails is role-based, and sensitive personal data is minimized before processing. The workflow documents decisions for internal audits and follows the IT security policies.

Labeling

The interface clearly shows which suggestions were generated automatically and which steps were confirmed or adjusted manually.

Accountability

The workflow's suggestions are not binding; they support decisions. Binding approvals are still given by the responsible people in IT operations.

Escalation path

In case of conflicting requirements or critical conflicts, the workflow flags cases for escalation to IT management or designated contacts in the departments.

Traceability

For every decision, the underlying tickets, parameters and justifications are stored, so later queries and audits can trace the process.

Operational roles

IT operations is responsible for day-to-day use and approval of the plans, the data science team provides feedback on priorities, and IT architecture looks after the interfaces and further development.

Transferability

The pattern suits any situation where technical maintenance competes with business tasks for limited capacity and priorities are mostly described in free text. An agentic workflow that bundles signals from several systems, understands context and suggests plans with minimal conflicts can also help outside research environments to reduce coordination effort and make risks visible early, without replacing human responsibility.