Validation and fallback

output.validators is the gate between a step and everything that reads it. A failure retries in place, seconds later on the same warm worker, instead of costing another batch round trip. When retries run out, fallback moves that one item to a different target, so the bulk is handled cheaply and you spend more only on the hard few percent.

The validators are an ordered list, checked in turn until one rejects. Today there is one type, json_schema. Write the schema inline, or once under the document's schemas and reference it with $ref: "#/schemas/<name>", as below.

The validator enforces type (one type, or a list such as [string, "null"] for an optional value), properties, required, items, enum and additionalProperties. Length, count and range keywords (minLength, maxItems, minimum and the rest) and description are passed to the model as guidance but never fail an output. Any other keyword is a compile error, so a typo is caught rather than ignored.

A step whose output another step reads must have a validator: without one, a malformed answer is found downstream instead of where it happened, and the compiler refuses the document.

classify gets one retry on the small model, then moves to the medium model, then tries the medium model again at temperature 0. If all of that fails, the run carries on without it.

name: triage-ticket
description: Classify a support ticket and draft a reply; summarize it either way.

input:
  schema:
    type: object
    required: [ticket]
    properties:
      ticket: { type: string }

schemas:
  triage:
    type: object
    required: [category, urgent]
    properties:
      category: { type: string, enum: [billing, bug, account, other] }
      urgent: { type: boolean }

nodes:
  classify:
    target: { product: otium-small }
    estimate: { tokens_in: 400, tokens_out: 30 }
    body:
      response_format:
        { type: json_schema, json_schema: { $ref: "#/schemas/triage" } }
      messages:
        - role: system
          content: "Classify the support ticket. Reply as JSON."
        - role: user
          content: "${input.ticket}"
    output:
      validators:
        - { type: json_schema, schema: { $ref: "#/schemas/triage" } }
      on_invalid: { retry: 1, then: fallback }
      on_failure: continue
    fallback:
      - { target: { product: otium-medium } }
      - { target: { product: otium-medium }, overrides: { temperature: 0 } }

  reply:
    target: { product: otium-medium }
    estimate: { tokens_in: 500, tokens_out: 250 }
    body:
      messages:
        - role: system
          content: "Draft a short, polite reply to a ${nodes.classify.output.category} ticket."
        - role: user
          content: "${input.ticket}"

  summarize:
    target: { product: otium-small }
    estimate: { tokens_in: 400, tokens_out: 40 }
    body:
      messages:
        - role: user
          content: "Summarize this ticket in one line: ${input.ticket}"

Its tests show each path. The responses are given in attempt order, and products checks which target each attempt went to:

cases:
  - name: a good answer on the first try
    input: { ticket: "I was charged twice this month." }
    nodes:
      classify: { output: { category: billing, urgent: false } }
      reply: { output: "Sorry about the double charge. We have refunded it." }
      summarize: { output: "Customer was charged twice." }
    expect:
      status: succeeded
      succeeded: [classify, reply, summarize]
      products:
        classify: [otium-small]
      prompts:
        reply: { contains: ["a billing ticket"] }

  - name: a bad answer retries, then moves to the bigger model
    input: { ticket: "The export button does nothing." }
    nodes:
      classify:
        - output: "Sure! It's a bug." # not JSON
        - output: { category: defect, urgent: true } # not one of the categories
        - output: { category: bug, urgent: true } # fallback[0] gets it right
      reply: { output: "Thanks for the report. We are looking at it." }
      summarize: { output: "Export button broken." }
    expect:
      status: succeeded
      products:
        classify: [otium-small, otium-small, otium-medium]

  - name: every attempt rejected; the summary still arrives
    input: { ticket: "???" }
    nodes:
      summarize: { output: "Unclear ticket." }
      classify:
        - output: "no idea"
        - output: "still no idea"
        - output: { category: other }
        - output: { category: other }
        - output: "{"
        - output: "{"
    expect:
      status: partial # on_failure: continue keeps what did succeed
      succeeded: [summarize]
      failed: { classify: invalid_output }
      skipped: { reply: upstream_failed }
      products:
        classify:
          [
            otium-small,
            otium-small,
            otium-medium,
            otium-medium,
            otium-medium,
            otium-medium,
          ]

A validator judges an answer after the fact. To make a good answer more likely in the first place, put a response_format in the step's body that references the same schema, as classify does above. The model is then constrained to that shape as it writes. If a step's response_format and its validator name different schemas, the document does not compile: the model would be asked for one shape and graded on another.

output.on_invalid decides what a rejected answer costs you:

• retry: N runs the step again on the same target, up to N more times.
• then: fallback moves on to the next entry in the step's fallback list once retries run out. Each entry gets its own retry allowance.

fallback is a list of attempt policies, tried in order. Each one is a target (a product) and, optionally, overrides: request parameters, such as temperature or max_tokens, to change for that attempt. It does not have to climb in size. The same product with a different setting is a legitimate entry, which is why the list holds policies rather than a next size up.

Every attempt is billed at its own target's rate, including the ones a validator rejects: the model did the work.

The compiler warns about the patterns that are usually mistakes:

• A validator with no on_invalid: a rejected answer fails the step with no retry at all, which gives up the cheapest fix a workflow has.
• then: fallback with no fallback list. That falls through to on_failure, which is legal for a step already on the largest sensible target, but drop then if that is what you mean.
• A fallback entry identical to the step's own target with no overrides. That is a retry, so write it as one.

Once every policy is used up, the step has failed with invalid_output, and output.on_failure decides what happens to the run:

• fail_run, the default, stops the run. Nothing new starts.
• continue leaves the step failed and lets the rest of the run go on. Steps that read it are skipped with upstream_failed; steps that do not still run.

Either way, a run's status comes from what it produced. A run with some outputs is partial, a first-class result with those outputs in it, and is only failed when nothing useful came out. In the example, summarize does not read classify, so the third test case ends partial with the summary delivered.

Every finished run has a status (succeeded, partial, failed, expired or cancelled) and an outcome saying why it stopped:

• completed: every step that could run did.
• completed_with_skips: a success in which conditions skipped some steps.
• node_failed: a step failed.
• budget_exhausted or deadline_exceeded: the run's time ran out (see Running workflows).
• park_expired: an external step was never resumed.
• cancelled: you cancelled it.

Each step has a terminal_reason, and each reason a fixed disposition: pruned means the workflow correctly chose not to run the step, and degraded means something prevented it. A step that a condition skipped and a step skipped because its input failed are both skipped; only the disposition tells them apart. Count failures on disposition, never on skipped, or a workflow that prunes heavily by design will look broken.

Every attempt is recorded: the target and model that served it, its output whether accepted or rejected, the validator's verdict, tokens and cost. When a validator rejects something you expected to pass, this is where to look:

otiumctl run attempts <run-id> classify

# or over HTTP
GET https://api.getotium.ai/v1/runs/{run_id}/nodes/classify/attempts

A verdict says where an answer broke and which rule it broke, never the content itself. The codes are not_json, missing_field, wrong_type, not_in_enum and schema_violation:

{"ok": false, "code": "not_in_enum", "pointer": "/category",
 "expected": "one of [billing bug account other]"}

A run that stops while an attempt has already finished, because a clock ran out or another step failed, still checks that attempt and keeps its output if it passes. You are never billed for an answer and then denied it.