Get more research done with the GPUs you already own.
Many campuses already run a shared endpoint for chat, but large batch jobs still go to private allocations: an endpoint built to answer requests one at a time is not built to take 100,000 of them, and an allocation reserves a GPU without keeping it busy. Otium is the shared batch layer your team runs. Researchers send the work with a deadline and get back to their research until they are told it is done; the platform keeps models loaded, batches every group’s requests together and recovers from interruptions.
Allocated is not the same as utilized
-
Optimization spread across every researcher
Researchers are experts in biology, chemistry or linguistics, yet batching, memory, throughput and model loading get tuned by whoever needs a model, on top of their research. That is a structural problem, not a failing of the researchers.
-
A GPU held for the whole job
A model server keeps its allocation while it loads, waits on the script feeding it and runs one group’s requests. Busy or waiting, the allocation is the same.
-
Infrastructure before science
Requesting a GPU, tuning the server, writing the queue, retries and resume logic, and resubmitting after a walltime or preemption, before the research starts.
What changes when inference is shared
Higher cluster efficiency
One loaded model serves every group’s work, and the serving engine batches their requests together as they arrive (continuous batching), with settings tuned per model and per GPU. Even a well-tuned private server batches only its own group’s requests; sharing is what lets every group’s work fill the same GPUs.
Where inference cost comes from →Send it and forget it
A batch file through the OpenAI SDK they already use, or a whole workflow, with a deadline. A signed webhook says when it has finished, whether it was a batch or a workflow. No model to load, no GPU to request, no retry or resume logic to write. Researchers who want to run their own servers still can; no one has to.
What batch inference is →One deadline for the whole pipeline
A method of several steps (extract, classify, check, summarise) is one workflow with one deadline for all of it. Each step’s results are kept as they finish, so an interruption costs only the work in flight, and a changed step reruns from there.
How workflows run →Beside Slurm and your gateway
Workers pull work from the queue and drive an OpenAI-compatible server (vLLM, llama.cpp) on their node; when one is preempted its jobs return to the queue at once. Launching workers as Slurm jobs into spare and preemptible capacity is on-prem work, built with design partners.
Running on capacity that can be taken back →
From a GPU per researcher to shared inference
An allocation reserves a GPU; it does not keep it busy. Shared serving puts the same work on GPUs that stay loaded and busy, and takes the infrastructure off researchers’ desks.
Today
Each researcher runs their own
- Group A GPU allocated for the whole job
- Group B GPU allocated for the whole job
- Group C GPU allocated for the whole job
Busy only while that group’s requests run; reserved the rest of the time.
- Request a GPU and size it
- Load and tune a model server
- Write the queue, retries and resume logic
- Resubmit after a walltime or preemption
With Otium
Researchers submit work to a shared service
One durable queue: every batch and workflow carries its own deadline
- Shared GPU 1
- Shared GPU 2
Each model loaded once, every group’s requests batched together and run back to back, since each batch’s deadline lets it wait its turn. The rest of each GPU’s time is free for more work.
- Submit a batch or a workflow, with a deadline
- Collect the results
Built and running today
- OpenAI-compatible batch files, through the SDK researchers already use
- Workers that serve any group’s jobs from one loaded model, with continuous batching in vLLM or llama.cpp
- Serving settings tuned per model and per GPU
- Workflows: several steps, one deadline, results kept step by step, reruns from any step
- Jobs returned to the queue at once when a worker is preempted
- A signed webhook when a batch or a workflow finishes, so nobody has to watch it
- Workers on your own hardware, limited to the hours you set
The on-prem edition, built with design partners
- The whole platform installed in your environment, nothing leaving campus
- Workers launched as Slurm jobs into spare and preemptible capacity when work is waiting
- Fair share between research groups
- Usage reports per group for Slurm accounting and chargeback
- Sign-in through your campus identity provider
Questions people ask first
- How much GPU capacity would we get back?
- We do not know, and neither does anyone without your numbers. Our working view, from talking with research-computing admins, is that GPUs allocated for inference are often far from fully used, because tuning them is spread across people whose job is research. It is a hypothesis, not a measurement. How much depends on how many researchers run their own servers and how busy those servers are. A pilot on your own work is how to find out.
- Can Otium use the unused part of our existing allocations?
- No. Otium never reaches into a job’s allocation. It runs on capacity you give it: GPUs set aside for shared inference, or, once the Slurm launcher is built, spare and preemptible capacity your scheduler hands out. The gain is that researchers no longer need allocations of their own for inference.
- We run Slurm, not Kubernetes. Does that work?
- The worker is one program beside the inference server, so a Slurm job can start both, and the control plane can run on a single VM. Having Otium submit those jobs itself, when work is waiting, is part of the on-prem edition.
- We already run a gateway for interactive use. Is this another one?
- No. A gateway answers requests as they arrive, which suits chat. A job of 100,000 requests needs a queue, a deadline and recovery from interruptions, so today it goes to a private allocation instead. Otium holds the whole job, plans it against its deadline and runs it in the room your interactive users leave.
- Isn’t this a shell script around sbatch?
- That is where most groups start. What the script does not give researchers is a queue that survives preemption, a deadline per job, pipelines with checked answers and reruns from any step, and one place to see what ran, what failed and why.
- Does our data leave campus?
- With on-prem, the whole platform runs in your environment and nothing leaves. With self-hosted workers, prompts and results pass through Otium’s cloud, encrypted at rest, which suits work without that requirement.
Get more research out of the GPUs you have.
Tell us about your cluster and how your researchers use models today. We are building the on-prem edition with design partners, and a pilot on your own work is how to measure what shared inference changes.