Testing

A workflow has unit tests. Each case gives an input and what every step returns, and says what should happen: which steps run, which are skipped and why, what a prompt contains, how retries and fallback escalate. Otium runs the real engine over those canned answers, with no model, no cost and in a few milliseconds, so a condition that skips the wrong branch shows up on your laptop, not in tomorrow's batch.

Canned answers always parse, so tests prove the graph's logic, not a model's output. For that, run a small batch.

Here is a support-ticket workflow: classify the ticket, draft a reply when it is a question, and hand urgent ones to a person through an external step.

name: ticket-triage
description: Classify a support ticket, draft a reply to questions, and hand urgent ones to a person.

input:
  schema:
    type: object
    required: [subject, body]
    properties:
      subject: { type: string }
      body: { type: string }

schemas:
  triage:
    type: object
    required: [category, urgent]
    properties:
      category: { type: string, enum: [bug, billing, question] }
      urgent: { type: boolean }
  reply:
    type: object
    required: [text]
    properties:
      text: { type: string }

nodes:
  classify:
    target: { product: otium-small }
    estimate: { tokens_in: 400, tokens_out: 30 }
    body:
      messages:
        - role: system
          content: "Classify the ticket. Reply as JSON with category and urgent."
        - role: user
          content: "${input.subject}\n\n${input.body}"
    output:
      validators:
        - { type: json_schema, schema: { $ref: "#/schemas/triage" } }
      on_invalid: { retry: 1, then: fallback }
    fallback:
      - { target: { product: otium-medium } }

  draft_reply:
    when: { eq: { path: "${nodes.classify.output.category}", value: question } }
    target: { product: otium-small }
    estimate: { tokens_in: 500, tokens_out: 200 }
    body:
      messages:
        - role: system
          content: "Draft a short, friendly reply. Reply as JSON with text."
        - role: user
          content: "${input.body}"
    output:
      validators:
        - { type: json_schema, schema: { $ref: "#/schemas/reply" } }
      on_invalid: { retry: 2 }

  # Urgent tickets wait for a person: the run parks until your system resumes it.
  handoff:
    type: external
    when: { eq: { path: "${nodes.classify.output.urgent}", value: true } }
    max_park: 4h
    park_estimate: { p50: 30m, p95: 1h }
    input:
      subject: "${input.subject}"
      category: "${nodes.classify.output.category}"
    output:
      on_failure: continue

Its tests sit beside it, one case per situation you care about:

workflow: testing-triage.yaml

cases:
  - name: a question gets a drafted reply
    input:
      { subject: "Export to CSV?", body: "Can I export my invoices as CSV?" }
    nodes:
      classify: { output: { category: question, urgent: false } }
      draft_reply: { output: { text: "Yes: Billing → Invoices → Export." } }
    expect:
      status: succeeded
      succeeded: [classify, draft_reply]
      skipped: { handoff: guard_false }
      prompts:
        draft_reply:
          contains: ["Can I export my invoices as CSV?"]
          not_contains: ["Export to CSV?"]

  - name: a bad answer retries, then falls back to a bigger model
    input:
      {
        subject: "Charged twice",
        body: "My card was charged twice this month.",
      }
    nodes:
      classify: # one response per attempt, in order
        - output: "Sure! Here is the JSON:" # not JSON at all
        - output: { category: refund, urgent: false } # JSON, but not a category we allow
        - output: { category: billing, urgent: false }
    expect:
      status: succeeded
      succeeded: [classify]
      skipped: { draft_reply: guard_false, handoff: guard_false }
      attempts: { classify: 3 }
      products: { classify: [otium-small, otium-small, otium-medium] }

  - name: an urgent bug waits for a person
    input: { subject: "Site down", body: "Every page returns 500." }
    nodes:
      classify: { output: { category: bug, urgent: true } }
      handoff: { output: { assignee: "on-call" } } # the body your system resumes the park with
    expect:
      status: succeeded
      succeeded: [classify, handoff]
      skipped: { draft_reply: guard_false }

  - name: nobody answers the handoff
    input: { subject: "Site down", body: "Every page returns 500." }
    nodes:
      classify: { output: { category: bug, urgent: true } }
      handoff: { lapse: true } # never resumed: the park runs out at max_park
    expect:
      status: partial
      succeeded: [classify]
      skipped: { draft_reply: guard_false }
      failed: { handoff: park_expired }

  - name: the first step's infrastructure fails
    input: { subject: "Hello", body: "Just saying hi." }
    nodes:
      # A failure walks the same ladder as a bad answer: retry, fall back, retry. One response
      # per attempt, or the case fails with "the case gives it no response".
      classify: [{ fail: true }, { fail: true }, { fail: true }, { fail: true }]
    expect:
      status: failed
      attempts: { classify: 4 }
      products:
        { classify: [otium-small, otium-small, otium-medium, otium-medium] }
      failed:
        classify: execution_failed
        draft_reply: upstream_failed
        handoff: upstream_failed

A test file has these top-level fields:

• workflow: the document under test, relative to the test file. Leave it out and x.test.yaml tests x.yaml.
• tier: the tier the cases are submitted at (standard if omitted). It sets the execution budget, so a graph too slow for the tier is refused exactly as a real batch would be.
• assets: the reference data the cases run against, in the same shape you publish it: a string for a shared asset, a map from key to string for a keyed one.
• cases: up to 100, each with a name, an input (checked against the workflow's input schema), optionally a custom_id, the canned nodes, and what to expect.

Unknown fields are errors, so a typo in expect can't quietly check nothing.

Under nodes, give each step that runs what it returns. One response is used for one attempt; a list gives one response per attempt, in order, which is how you script a retry or a fallback. Each response is one of:

• output: what the step produced. A string is used exactly as the model's reply, so output: "not json" exercises a validator. Anything else is sent as JSON.
• fail: true: the attempt fails the way infrastructure does, after its own retries. A failure walks the same ladder as a rejected answer: retry, fall back, retry.
• lapse: true: for an external step, nobody resumes the park, so it runs out at max_park.

An external step's output is the body your system would resume the park with. A tool step is canned like a model step: its output is your endpoint's JSON answer, and fail: true is your endpoint failing, reported as tool_error.

A step that runs when its responses have run out fails the case. That is deliberate: the graph did something you didn't plan for, which is exactly what a test is for.

$ otiumctl workflow test --run infrastructure
testing-triage.test.yaml
  FAIL the first step's infrastructure fails
         - classify made attempt 2, but the case gives it no response
         - run status "running", want "failed"
         run running
           classify           ready  2 attempts: otium-small → otium-small
           draft_reply        blocked
           handoff            blocked

0 passed, 1 failed

Every field of expect is optional, and each one you give is checked exactly: succeeded: [a, b] fails if c also succeeded.

status

The run’s final status: succeeded, partial (some steps failed with on_failure: continue), failed, expired or cancelled.

succeeded

Every step that succeeded. A step that succeeds and is not listed fails the case.

skipped

Every skipped step and why: guard_false (its own condition was false) or upstream_skipped (a step it reads from was skipped). An empty reason accepts either.

failed

Every failed step and why, such as invalid_output, execution_failed, upstream_failed, park_expired, tool_error or tool_timeout. An empty reason accepts any.

attempts

How many attempts a step made, retries and fallbacks included.

products

The product each attempt targeted, in order. This is how a test tells a fallback that escalated from one that only retried.

prompts

Substrings the prompt of a step’s final attempt must contain (contains) or must not (not_contains): a guarded section that dropped out, an input that never arrived.

rejected

The error a submission is refused with before it becomes a run, such as missing_assets or window_too_short. When set, the case expects no run at all.

otiumctl workflow test runs locally, with no network, no credentials and no GPU. Give it files or directories (the current one by default); a directory is searched for *.test.yaml.

$ otiumctl workflow test
testing-triage.test.yaml
  ok   a question gets a drafted reply
  ok   a bad answer retries, then falls back to a bigger model
  ok   an urgent bug waits for a person
  ok   nobody answers the handoff
  ok   the first step's infrastructure fails

5 passed, 0 failed

• --run <text> runs only the cases whose name contains it.
• --verbose prints every case's trace (each step, its status, attempts and products), not only the failing ones.
• --json prints the reports as JSON.
• --snapshots and --update: see below.

It exits non-zero when any case fails, so it drops straight into a pre-commit hook or CI. The same runner is behind the Tests pane of the portal's workflow editor, and behind the API, which takes the document and the suite as text and returns the report:

jq -n --rawfile document testing-triage.yaml --rawfile tests testing-triage.test.yaml \
    '{document: $document, tests: $tests}' |
  curl -s https://api.getotium.ai/v1/workflows/test \
    -H "Authorization: Bearer $OTIUM_TOKEN" \
    -H "Content-Type: application/json" \
    -d @-

The API writes nothing and runs no inference. A suite that runs answers 200 whether or not its cases pass; the report says which. A document that doesn't compile, or a suite that doesn't parse, is 422.

Assertions check the parts of a prompt you thought of. Snapshots catch the rest. Run once with --update and every step's rendered prompt in every case is written beside the test file, under __snapshots__/<suite>/<case>/<step>.txt:

[system]
Draft a short, friendly reply. Reply as JSON with text.

[user]
Can I export my invoices as CSV?

Commit them. From then on, --snapshots compares every prompt with its file, and any change to what you send a model, from a reworded instruction to a section that stopped appearing, fails with the line that moved:

  FAIL a question gets a drafted reply
         - draft_reply's prompt differs from its snapshot
           (__snapshots__/testing-triage/a-question-gets-a-drafted-reply/draft_reply.txt):
           line 2 was "Draft a short, friendly reply. Reply as JSON with text.",
           now "Draft a short, formal reply. Reply as JSON with text."; run with --update to accept

If the change is what you meant, run --update again and the new prompt shows up as a diff in review, next to the change that caused it.

otiumctl workflow validate runs the same compiler publishing does. It prints the shape of the graph, the critical path, and the estimated duration and cost per item:

$ otiumctl workflow validate testing-triage.yaml
3 nodes (1 external, 0 tool), critical path 2 deep: classify → handoff
est. duration p50 1803s / p95 3613s, of which execution p95 39s (parks excluded: what the tier's budget is checked against)
est. cost $0.000171 per item, $1.71 per 10,000
  (tokens as declared on each node; rates illustrative, not a quote)
1 affinity groups, 1 barriers, 0 cache groups

It exits non-zero when the document would be rejected. --strict fails on warnings too, --quiet prints nothing and leaves the answer in the exit status, --plan prints the compiled plan as JSON, and --mermaid prints the graph as a Mermaid flowchart. A warning says what is wrong and what it costs. Here the handoff's park_estimate was widened to p95: 2h, so the worst case is four times the typical one:

$ otiumctl workflow validate testing-triage.yaml --strict
warn: fragile_critical_path: the critical path's p95 (7213s) is 4.0× its p50 (1803s) — one long-tail node makes this graph fragile against its completion window
3 nodes (1 external, 0 tool), critical path 2 deep: classify → handoff
…
otiumctl: document has warnings and --strict is set

Errors stop a document from publishing:

structural

The document doesn’t parse, or a field is missing or has the wrong type.

unknown_node

A binding or after: names a step that doesn’t exist.

unknown_binding_path

A binding reads a field the input schema or that step’s output schema doesn’t declare.

unknown_schema_ref

A $ref points at a schema the document doesn’t define.

unknown_asset

A binding reads an asset the document doesn’t declare.

asset_index_mismatch

A keyed asset is read without an index, or a shared one with one.

consumed_without_validator

A later step reads this step’s output, but nothing checks that output’s shape first.

unguarded_strict_predicate

A condition reads a field from a step that can be skipped, without allowing for its absence.

impossible_comparison

A condition compares a field with a value its schema says it can never hold.

external_node_without_input

An external step has no input to park with.

external_input_references_nothing

An external step’s input reads no earlier step, so it would park on nothing.

invalid_response_format

A step’s response_format is not one the model service accepts.

response_format_validator_drift

response_format asks the model for one schema while the validator checks another.

cycle

Steps depend on each other in a loop.

unreachable_node

A step can never run because it sits downstream of a cycle.

unsupported_scope

A step asks for a scope other than item, the only one available.

unknown_tool_endpoint

At publish: the document names a tool endpoint you haven’t registered.

Warnings publish, but each one is something that costs time or money:

fake_edge

after: makes a step wait for one it reads nothing from. Often the cheapest speed-up you have.

redundant_after

after: repeats an order a binding already creates.

validated_without_on_invalid

A step has a validator but no on_invalid, so a rejected output fails at once with no retry.

then_fallback_without_list

on_invalid says then: fallback, but the step has no fallback list.

noop_fallback

A fallback target is identical to the step’s own target: a retry in disguise.

guarded_section_before_asset

A conditional prompt section sits before an asset, so the shared start of the prompt varies by item.

asset_after_item_data

An asset comes after per-item text, so the shared part of the prompt can’t be reused.

fragile_critical_path

The slowest chain’s worst case is several times its typical case: one long-tail step can blow the window.

park_dominates

An external step’s max_park makes up most of the estimate and it has no park_estimate.

The codes are stable, so it is safe to search logs or CI output for one. The browser demo and the portal's editor show the same findings as you type.

Both commands run offline, so a CI step needs only otiumctl and your files. Check every document strictly, then run every suite against its snapshots:

#!/bin/sh
# Fail the build on a document that would be rejected, a warning, a broken
# test case, or a prompt that changed without its snapshot being updated.
set -e
for doc in workflows/*.yaml; do
  case "$doc" in *.test.yaml) continue ;; esac
  otiumctl workflow validate "$doc" --strict --quiet
done
otiumctl workflow test workflows --snapshots