Models and evaluation

Use step.model() for one application-owned structured model operation. Use workflow.judge() for one assessment followed by a deterministic score gate, and workflow.judgeUntil() when a candidate must be revised until it passes. Use step.delegate() when an external harness owns inference and tools.

Configure an OpenAI-compatible worker​

The built-in worker is exported from the product subpath:

import { createOpenAiCompatibleModelWorker } from "@dromio/workflow/product";

export const responseModel = createOpenAiCompatibleModelWorker({
  baseUrl: process.env.OPENAI_BASE_URL,
  model: process.env.OPENAI_MODEL,
  apiKey: process.env.OPENAI_API_KEY,
  maxAttempts: 2,
  temperature: 0.2,
});

baseUrl and model are required after configuration is resolved. Pass them explicitly from your host configuration; this example uses the conventional OPENAI_BASE_URL, OPENAI_MODEL, and OPENAI_API_KEY names. The worker does not require a particular environment-variable naming scheme. Keep credentials in the host's secret manager, never in workflow documents or committed source.

The worker speaks an OpenAI-compatible chat-completions protocol. A custom provider can implement ModelWorkerPort and be injected through the same model option. The authoring package does not choose a provider for you.

Define a model step​

import { promptFile, step } from "@dromio/workflow";

export const generateResponse = step.model({
  id: "responses.generate",
  input: {prompt: promptSchema},
  output: {generatedResponse: generatedResponseSchema},
  model: responseModel,
  prompt: promptFile(new URL("./generate.md", import.meta.url)),
  buildPrompt: ({prompt}) => ({request: prompt}),
  sideEffects: ["model.call"],
});

The SDK makes exactly one worker complete() call, supplies a JSON Schema derived from the output contracts, parses the response as an object, and validates every output key. It does not run a hidden evaluator.

A model may be provided at three levels. Runtime placement wins over the workflow-level model, which wins over the step definition default. This makes unit tests straightforward: inject a deterministic worker without changing the step's prompt or contracts.

Model lifecycle events​

Provider-backed calls emit model.request.started, zero or more model.response.delta events, model.response.completed, or model.request.failed. They include trace context joined to the step attempt. Delta events represent public response bytes, not private model reasoning.

Define a score policy​

import { defineScorePolicy } from "@dromio/workflow/core";

export const responsePolicy = defineScorePolicy({
  id: "score.responses.generated",
  satisfies: [
    {id: "answers-prompt", description: "Directly answers the prompt."},
  ],
  gaps: [
    {id: "not-responsive", description: "Does not answer the prompt.", severity: "high"},
  ],
  risks: [],
  gates: [
    {id: "pass", minScore: 0.8, nextAction: "complete", status: "pass"},
    {id: "revise", minScore: 0, nextAction: "revise", status: "revise"},
  ],
});

defineScorePolicy() returns the CandidateScorePolicy accepted by both workflow.judge() and step.gate(). The policy, not prompt prose, owns the threshold and next action.

Judge once​

import { promptFile, step } from "@dromio/workflow";
import { promptedOperationEvaluationSchema } from "@dromio/workflow/core";

export const evaluateResponse = step.evaluate({
  id: "responses.evaluate",
  input: {
    generatedResponse: generatedResponseSchema,
    prompt: promptSchema,
  },
  evaluator: {
    model: responseModel,
    prompt: promptFile(new URL("./evaluate.md", import.meta.url)),
    output: {evaluation: promptedOperationEvaluationSchema},
  },
  policy: responsePolicy,
});

import { step, workflow } from "@dromio/workflow";

const assessor = step.model({
  id: "responses.assess",
  input: {candidate: responseSchema},
  output: {evaluation: promptedOperationEvaluationSchema},
  prompt: evaluateResponsePrompt,
});

export const judgeResponse = workflow.judge({
  id: "responses.judge",
  input: {candidate: responseSchema},
  assessor,
  policy: responsePolicy,
});

The assessor may instead be step.delegate(), a deterministic step, or a multi-step child workflow. workflow.judge() does not choose a provider. It adds the deterministic gate and returns evaluation plus decision.

The assessor writes evaluation.score. The gate reads that value from its input.evaluation.score. After the run, the same canonical score is in session.state.evaluation.score; step definitions do not retain a .result.

For an already-produced evaluation, use step.gate() directly:

const gateResponse = step.gate({
  id: "responses.gate",
  input: {evaluation: promptedOperationEvaluationSchema},
  policy: responsePolicy,
});

The gate calls no model. It outputs decision plus the original evaluation and emits score, evaluation, and operation-decision events.

Judge until the candidate passes​

export const improveResponse = workflow.judgeUntil({
  id: "responses.improve",
  input: {prompt: z.string()},
  produce: generateResponse,
  judge: judgeResponse,
  revise: reviseResponse,
  maxAttempts: 3,
});

The visible nodes are produce, judge, gate routing, revise, and complete. Failed judgments send the evaluation feedback to the revision step. Passing routes to completion. Exhaustion fails clearly unless the bundle is given an explicit human-review step. Completed attempts are checkpointed and are not replayed after a durable suspension.

Harness-owned and multi-agent judging​

const panel = step.delegate({
  id: "responses.panel",
  input: {candidate: responseSchema},
  output: {judgments: z.array(judgmentSchema).min(3)},
  instructions: "Collect independent judgments with score, rationale, and gaps.",
  capabilities: {preferred: ["subagents"]},
});

The harness may spawn subagents, but step.gate() never does. An ordinary deterministic aggregation step calculates the declared mean, median, minimum, or weighted score from judgments[] and emits the canonical evaluation. The gate consumes only that canonical evaluation. Individual scores remain evidence; an opaque aggregate is not proof that multiple judges ran.

Eject the generated workflow​

const {assessor, gate, workflow} = evaluateResponse.eject();

Use eject() when the convenience topology needs custom authored steps. The file-writing overload can emit explicit source and paired workflow documents; after ejection, your code owns their evolution. .workflow exposes the generated child without changing it; .eject() exposes its assessor and gate so you can customize that topology.

Compatibility wrappers​

step.evaluate() remains a deprecated, type-safe placement wrapper around the same model-assessor and gate topology. Its .workflow and .eject() contracts remain available during migration. New code authors workflow.judge() and places it with step.workflow() only when a parent boundary is needed.

step.promptedContract() is also deprecated but supported. New structured requirement loops use workflow.clarifyUntil(), whose resolve, judge, gate, ask, merge, and revise phases are real executable nodes. See Human input and waits.

The previous compatibility shape remains valid:

import { step } from "@dromio/workflow";
import { defineEvaluationBar } from "@dromio/workflow/core";
import { z } from "zod";

const briefSchema = z.object({
  kind: z.literal("brief"),
  requirements: z.array(z.object({
    id: z.string(),
    label: z.string(),
    required: z.boolean().optional(),
    status: z.enum(["satisfied", "missing", "ambiguous", "unsupported"]),
    value: z.string().nullable(),
  })),
});

const resolveBrief = step.promptedContract({
  id: "brief.resolve",
  input: {prompt: z.string()},
  output: {audience: z.string().min(1)},
  contract: briefSchema,
  definition: {
    id: "brief.contract-loop",
    questionPolicy: {id: "brief.questions", maxAnswerAttempts: 2},
    revisionPolicy: {id: "brief.revisions", maxContractLoops: 2},
    resolveContract: ({answers}) => {
      const audience = typeof answers.audience === "string" ? answers.audience : null;
      return {
        kind: "brief" as const,
        requirements: [{
          id: "audience",
          label: "Audience",
          required: true,
          status: audience ? "satisfied" as const : "missing" as const,
          value: audience,
        }],
      };
    },
    evaluateContract: ({contract}) => {
      const passed = contract.requirements[0]?.status === "satisfied";
      return defineEvaluationBar({
        label: "Brief clarity",
        subjectId: "brief",
        threshold: 0.8,
        score: passed ? 1 : 0.4,
        status: passed ? "pass" : "needs_input",
        nextAction: passed ? "complete" : "ask",
        satisfies: [{id: "audience", passed, reason: passed ? "Known." : "Missing."}],
        gaps: passed ? [] : [{id: "audience", message: "Audience missing.", severity: "high"}],
        risks: [],
        questions: [],
      });
    },
    createQuestions: ({contract}) =>
      contract.requirements[0]?.status === "satisfied"
        ? []
        : [{id: "audience", type: "text", prompt: "Who is this for?"}],
  },
  mapCompleted: ({contract}) => ({
    audience: contract.requirements[0]?.value ?? "",
  }),
});

const {catalog, document} = resolveBrief.inspect();

inspect() exposes its legacy inspection topology; it does not make those legacy phase records separately executable.

Test without spending tokens​

Unit-test parsing, graph shape, policy decisions, and events with a fake ModelWorkerPort. Add an opt-in integration test for transport and credentials, then a separate semantic evaluation suite for prompt and model quality. A passing deterministic test does not prove live model quality.

Evaluate harness-owned work​

An external harness does not require a model router inside the Workflow App. The handoff boundary is provider-neutral:

Use an application-owned step.model() after the delegated step when the app must own one schema-constrained judgment. Put the numeric threshold and routing in deterministic workflow code or step.gate() so the graph shows why it advanced.

When the judgment itself should be harness-owned—for example, the harness may spawn multiple reviewer subagents—use another step.delegate() whose output contract includes the score, threshold evidence, and feedback. Dromio still sees one durable handoff and validates its structured result; it does not implement the harness's subagent scheduler.