Prompt evaluation

Routes are relative to <PUBLIC_BASE_URL>; request and response bodies are JSON unless stated otherwise. Authentication, scopes, pagination and the error format are described in the REST API guides.

POST `/api/v1/workflows/{id}/nodes/{nodeId}/eval`

Start an evaluation run on a categorizer node

Runs the SAVED draft node (an ai.categorize) on a sample of messages and scores it against the reference set. 202: cases arrive over the WebSocket, the detail is read with GET /eval-runs/{id}. A node not yet saved is eval.draft_not_saved.

Access — Member session or API key with scope workflows:write.

Parameters

Name	In	Type	Required
`id`	path	`string`	yes
`nodeId`	path	`string`	yes

Request body (application/json)

Field	Type	Required
`sample`	`object`	yes

JSON Schema

json

{
  "type": "object",
  "properties": {
    "sample": {
      "oneOf": [
        {
          "type": "object",
          "properties": {
            "kind": { "type": "string", "const": "recent" },
            "limit": { "default": 50, "type": "integer", "minimum": 10, "maximum": 200 }
          },
          "required": [ "kind" ]
        },
        {
          "type": "object",
          "properties": { "kind": { "type": "string", "const": "reference" } },
          "required": [ "kind" ]
        }
      ]
    }
  },
  "required": [ "sample" ]
}

Responses

202 — The run, queued.

Field	Type	Required
`run`	`object`	yes

JSON Schema

json

{
  "type": "object",
  "properties": {
    "run": {
      "type": "object",
      "properties": {
        "id": { "type": "string" },
        "workflowId": { "type": "string" },
        "nodeId": { "type": "string" },
        "status": {
          "type": "string",
          "enum": [ "queued", "running", "succeeded", "failed" ]
        },
        "prompt": { "type": "string" },
        "categories": { "type": "array", "items": { "type": "string" } },
        "multiLabel": { "type": "boolean" },
        "model": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
        "totals": {
          "type": "object",
          "properties": {
            "total": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
            "done": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
            "failed": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 }
          },
          "required": [ "total", "done", "failed" ],
          "additionalProperties": false
        },
        "createdAt": { "type": "string" },
        "finishedAt": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }
      },
      "required": [
        "id",
        "workflowId",
        "nodeId",
        "status",
        "prompt",
        "categories",
        "multiLabel",
        "model",
        "totals",
        "createdAt",
        "finishedAt"
      ],
      "additionalProperties": false
    }
  },
  "required": [ "run" ],
  "additionalProperties": false
}

400 — The request does not match its schema.

401 — No valid session or API key (auth.unauthenticated).

403 — Refused: insufficient role (auth.forbidden), missing scope (api_key.scope_missing, details.required names it) or a route closed to API keys (api_key.session_required).

429 — The API key exceeded its rate limit (api_key.rate_limited); Retry-After says when to retry.

This operation accepts an Idempotency-Key header: replaying the same request with the same key returns the original response instead of acting twice. See Idempotency.

Error codes — request.bad_request, workflow.not_found, eval.draft_not_saved, eval.node_not_categorizer, eval.node_params_invalid, eval.sample_empty, eval.llm_unavailable. The common codes (request.bad_request, auth.unauthenticated, api_key.scope_missing, api_key.rate_limited…) apply to every route; see Errors.

GET `/api/v1/workflows/{id}/nodes/{nodeId}/eval-runs`

List the evaluation runs of a node

Each run carries the prompt and categories it was run with, so two runs can be compared side by side.

Access — Member session or API key with scope workflows:read.

Parameters

Name	In	Type	Required
`id`	path	`string`	yes
`nodeId`	path	`string`	yes

Responses

200 — The runs.

Field	Type	Required
`runs`	`object[]`	yes

JSON Schema

json

{
  "type": "object",
  "properties": {
    "runs": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "workflowId": { "type": "string" },
          "nodeId": { "type": "string" },
          "status": {
            "type": "string",
            "enum": [ "queued", "running", "succeeded", "failed" ]
          },
          "prompt": { "type": "string" },
          "categories": { "type": "array", "items": { "type": "string" } },
          "multiLabel": { "type": "boolean" },
          "model": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
          "totals": {
            "type": "object",
            "properties": {
              "total": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
              "done": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
              "failed": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 }
            },
            "required": [ "total", "done", "failed" ],
            "additionalProperties": false
          },
          "createdAt": { "type": "string" },
          "finishedAt": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }
        },
        "required": [
          "id",
          "workflowId",
          "nodeId",
          "status",
          "prompt",
          "categories",
          "multiLabel",
          "model",
          "totals",
          "createdAt",
          "finishedAt"
        ],
        "additionalProperties": false
      }
    }
  },
  "required": [ "runs" ],
  "additionalProperties": false
}

401 — No valid session or API key (auth.unauthenticated).

403 — Refused: insufficient role (auth.forbidden), missing scope (api_key.scope_missing, details.required names it) or a route closed to API keys (api_key.session_required).

429 — The API key exceeded its rate limit (api_key.rate_limited); Retry-After says when to retry.

Error codes — workflow.not_found. The common codes (request.bad_request, auth.unauthenticated, api_key.scope_missing, api_key.rate_limited…) apply to every route; see Errors.

GET `/api/v1/workflows/{id}/nodes/{nodeId}/few-shot`

Get few-shot examples from my corrections

Balanced examples drawn from the reference set, ready to paste in a prompt. Always 200: an empty reference set yields an empty block.

Access — Member session or API key with scope workflows:read.

Parameters

Name	In	Type	Required
`id`	path	`string`	yes
`nodeId`	path	`string`	yes
`limit`	query	`integer`	no

Responses

200 — The examples.

Field	Type	Required
`block`	`string`	yes
`examples`	`object[]`	yes
`messageIds`	`string[]`	yes
`referenceCount`	`integer`	yes

JSON Schema

json

{
  "type": "object",
  "properties": {
    "block": { "type": "string" },
    "examples": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "messageId": { "type": "string" },
          "subject": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
          "from": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
          "excerpt": { "type": "string" },
          "categories": { "type": "array", "items": { "type": "string" } }
        },
        "required": [ "messageId", "subject", "from", "excerpt", "categories" ],
        "additionalProperties": false
      }
    },
    "messageIds": { "type": "array", "items": { "type": "string" } },
    "referenceCount": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 }
  },
  "required": [ "block", "examples", "messageIds", "referenceCount" ],
  "additionalProperties": false
}

400 — The request does not match its schema.

401 — No valid session or API key (auth.unauthenticated).

403 — Refused: insufficient role (auth.forbidden), missing scope (api_key.scope_missing, details.required names it) or a route closed to API keys (api_key.session_required).

429 — The API key exceeded its rate limit (api_key.rate_limited); Retry-After says when to retry.

Error codes — request.bad_request, workflow.not_found. The common codes (request.bad_request, auth.unauthenticated, api_key.scope_missing, api_key.rate_limited…) apply to every route; see Errors.

GET `/api/v1/eval-runs/{id}`

Read an evaluation run

The cases, paginated by cursor, and the score, computed on the WHOLE run. A run of another member is a 404.

Access — Member session or API key with scope workflows:read.

Parameters

Name	In	Type	Required
`id`	path	`string`	yes
`cursor`	query	`string`	no
`limit`	query	`integer`	no

Responses

200 — The run, its score and a page of cases.

Field	Type	Required
`run`	`object`	yes
`score`	`object`	yes
`cases`	`object[]`	yes
`nextCursor`	`string \| null`	yes

JSON Schema

json

{
  "type": "object",
  "properties": {
    "run": {
      "type": "object",
      "properties": {
        "id": { "type": "string" },
        "workflowId": { "type": "string" },
        "nodeId": { "type": "string" },
        "status": {
          "type": "string",
          "enum": [ "queued", "running", "succeeded", "failed" ]
        },
        "prompt": { "type": "string" },
        "categories": { "type": "array", "items": { "type": "string" } },
        "multiLabel": { "type": "boolean" },
        "model": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
        "totals": {
          "type": "object",
          "properties": {
            "total": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
            "done": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
            "failed": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 }
          },
          "required": [ "total", "done", "failed" ],
          "additionalProperties": false
        },
        "createdAt": { "type": "string" },
        "finishedAt": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }
      },
      "required": [
        "id",
        "workflowId",
        "nodeId",
        "status",
        "prompt",
        "categories",
        "multiLabel",
        "model",
        "totals",
        "createdAt",
        "finishedAt"
      ],
      "additionalProperties": false
    },
    "score": {
      "type": "object",
      "properties": {
        "scored": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
        "exactMatches": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
        "accuracy": {
          "anyOf": [
            { "type": "number", "minimum": 0, "maximum": 1 },
            { "type": "null" }
          ]
        },
        "multiLabel": { "type": "boolean" },
        "perCategory": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "category": { "type": "string" },
              "expectedCount": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
              "predictedCount": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
              "truePositives": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
              "falsePositives": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
              "falseNegatives": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
              "precision": {
                "anyOf": [
                  { "type": "number", "minimum": 0, "maximum": 1 },
                  { "type": "null" }
                ]
              },
              "recall": {
                "anyOf": [
                  { "type": "number", "minimum": 0, "maximum": 1 },
                  { "type": "null" }
                ]
              },
              "f1": {
                "anyOf": [
                  { "type": "number", "minimum": 0, "maximum": 1 },
                  { "type": "null" }
                ]
              }
            },
            "required": [
              "category",
              "expectedCount",
              "predictedCount",
              "truePositives",
              "falsePositives",
              "falseNegatives",
              "precision",
              "recall",
              "f1"
            ],
            "additionalProperties": false
          }
        },
        "confusion": {
          "oneOf": [
            {
              "type": "object",
              "properties": {
                "kind": { "type": "string", "const": "single" },
                "labels": { "type": "array", "items": { "type": "string" } },
                "counts": {
                  "type": "array",
                  "items": {
                    "type": "array",
                    "items": {
                      "type": "integer",
                      "minimum": 0,
                      "maximum": 9007199254740991
                    }
                  }
                }
              },
              "required": [ "kind", "labels", "counts" ],
              "additionalProperties": false
            },
            {
              "type": "object",
              "properties": {
                "kind": { "type": "string", "const": "multi" },
                "perCategory": {
                  "type": "array",
                  "items": {
                    "type": "object",
                    "properties": {
                      "category": { "type": "string" },
                      "truePositives": {
                        "type": "integer",
                        "minimum": 0,
                        "maximum": 9007199254740991
                      },
                      "falsePositives": {
                        "type": "integer",
                        "minimum": 0,
                        "maximum": 9007199254740991
                      },
                      "falseNegatives": {
                        "type": "integer",
                        "minimum": 0,
                        "maximum": 9007199254740991
                      },
                      "trueNegatives": {
                        "type": "integer",
                        "minimum": 0,
                        "maximum": 9007199254740991
                      }
                    },
                    "required": [
                      "category",
                      "truePositives",
                      "falsePositives",
                      "falseNegatives",
                      "trueNegatives"
                    ],
                    "additionalProperties": false
                  }
                }
              },
              "required": [ "kind", "perCategory" ],
              "additionalProperties": false
            }
          ]
        }
      },
      "required": [
        "scored",
        "exactMatches",
        "accuracy",
        "multiLabel",
        "perCategory",
        "confusion"
      ],
      "additionalProperties": false
    },
    "cases": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "mailboxId": { "type": "string" },
          "from": {
            "anyOf": [
              {
                "type": "object",
                "properties": { "name": { "type": "string" }, "email": { "type": "string" } },
                "required": [ "email" ],
                "additionalProperties": false
              },
              { "type": "null" }
            ]
          },
          "subject": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
          "snippet": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
          "receivedAt": { "type": "string" },
          "folderLabels": { "type": "array", "items": { "type": "string" } },
          "flags": {
            "type": "object",
            "properties": {
              "seen": { "type": "boolean" },
              "flagged": { "type": "boolean" },
              "draft": { "type": "boolean" },
              "sent": { "type": "boolean" }
            },
            "required": [ "seen", "flagged", "draft", "sent" ],
            "additionalProperties": false
          },
          "signals": {
            "type": "object",
            "properties": {
              "isAutoReply": { "type": "boolean" },
              "isNoReply": { "type": "boolean" },
              "isMailingList": { "type": "boolean" },
              "isFromSelf": { "type": "boolean" }
            },
            "required": [ "isAutoReply", "isNoReply", "isMailingList", "isFromSelf" ],
            "additionalProperties": false
          },
          "hasAttachments": { "type": "boolean" },
          "status": { "type": "string", "enum": [ "pending", "succeeded", "failed" ] },
          "predicted": {
            "anyOf": [
              {
                "type": "object",
                "propertyNames": { "type": "string", "minLength": 1 },
                "additionalProperties": { "type": "boolean" }
              },
              { "type": "null" }
            ]
          },
          "expected": {
            "anyOf": [
              {
                "type": "object",
                "propertyNames": { "type": "string", "minLength": 1 },
                "additionalProperties": { "type": "boolean" }
              },
              { "type": "null" }
            ]
          },
          "error": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }
        },
        "required": [
          "id",
          "mailboxId",
          "from",
          "subject",
          "snippet",
          "receivedAt",
          "folderLabels",
          "flags",
          "signals",
          "hasAttachments",
          "status",
          "predicted",
          "expected",
          "error"
        ],
        "additionalProperties": false
      }
    },
    "nextCursor": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }
  },
  "required": [ "run", "score", "cases", "nextCursor" ],
  "additionalProperties": false
}

400 — The request does not match its schema.

401 — No valid session or API key (auth.unauthenticated).

403 — Refused: insufficient role (auth.forbidden), missing scope (api_key.scope_missing, details.required names it) or a route closed to API keys (api_key.session_required).

429 — The API key exceeded its rate limit (api_key.rate_limited); Retry-After says when to retry.

Error codes — request.bad_request, eval.run_not_found. The common codes (request.bad_request, auth.unauthenticated, api_key.scope_missing, api_key.rate_limited…) apply to every route; see Errors.

PUT `/api/v1/eval-runs/{id}/cases/{messageId}/expected`

Correct the expected labels of a case

A replacement of the expected labels, written to the case AND to the reference set in one transaction. A label unknown to the run is eval.unknown_category.

Access — Member session or API key with scope workflows:write.

Parameters

Name	In	Type	Required
`id`	path	`string`	yes
`messageId`	path	`string`	yes

Request body (application/json)

Field	Type	Required
`expected`	`object`	yes

JSON Schema

json

{
  "type": "object",
  "properties": {
    "expected": {
      "type": "object",
      "propertyNames": { "type": "string", "minLength": 1 },
      "additionalProperties": { "type": "boolean" }
    }
  },
  "required": [ "expected" ]
}

Responses

200 — The corrected case and the new reference count.

Field	Type	Required
`case`	`object`	yes
`referenceCount`	`integer`	yes

JSON Schema

json

{
  "type": "object",
  "properties": {
    "case": {
      "type": "object",
      "properties": {
        "id": { "type": "string" },
        "mailboxId": { "type": "string" },
        "from": {
          "anyOf": [
            {
              "type": "object",
              "properties": { "name": { "type": "string" }, "email": { "type": "string" } },
              "required": [ "email" ],
              "additionalProperties": false
            },
            { "type": "null" }
          ]
        },
        "subject": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
        "snippet": { "anyOf": [ { "type": "string" }, { "type": "null" } ] },
        "receivedAt": { "type": "string" },
        "folderLabels": { "type": "array", "items": { "type": "string" } },
        "flags": {
          "type": "object",
          "properties": {
            "seen": { "type": "boolean" },
            "flagged": { "type": "boolean" },
            "draft": { "type": "boolean" },
            "sent": { "type": "boolean" }
          },
          "required": [ "seen", "flagged", "draft", "sent" ],
          "additionalProperties": false
        },
        "signals": {
          "type": "object",
          "properties": {
            "isAutoReply": { "type": "boolean" },
            "isNoReply": { "type": "boolean" },
            "isMailingList": { "type": "boolean" },
            "isFromSelf": { "type": "boolean" }
          },
          "required": [ "isAutoReply", "isNoReply", "isMailingList", "isFromSelf" ],
          "additionalProperties": false
        },
        "hasAttachments": { "type": "boolean" },
        "status": { "type": "string", "enum": [ "pending", "succeeded", "failed" ] },
        "predicted": {
          "anyOf": [
            {
              "type": "object",
              "propertyNames": { "type": "string", "minLength": 1 },
              "additionalProperties": { "type": "boolean" }
            },
            { "type": "null" }
          ]
        },
        "expected": {
          "anyOf": [
            {
              "type": "object",
              "propertyNames": { "type": "string", "minLength": 1 },
              "additionalProperties": { "type": "boolean" }
            },
            { "type": "null" }
          ]
        },
        "error": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }
      },
      "required": [
        "id",
        "mailboxId",
        "from",
        "subject",
        "snippet",
        "receivedAt",
        "folderLabels",
        "flags",
        "signals",
        "hasAttachments",
        "status",
        "predicted",
        "expected",
        "error"
      ],
      "additionalProperties": false
    },
    "referenceCount": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 }
  },
  "required": [ "case", "referenceCount" ],
  "additionalProperties": false
}

400 — The request does not match its schema.

401 — No valid session or API key (auth.unauthenticated).

403 — Refused: insufficient role (auth.forbidden), missing scope (api_key.scope_missing, details.required names it) or a route closed to API keys (api_key.session_required).

429 — The API key exceeded its rate limit (api_key.rate_limited); Retry-After says when to retry.

Error codes — request.bad_request, eval.run_not_found, eval.case_not_found, eval.unknown_category. The common codes (request.bad_request, auth.unauthenticated, api_key.scope_missing, api_key.rate_limited…) apply to every route; see Errors.

Triggers

AI

Logic

Data

Actions

Account

AI provider

Service

Reference

Prompt evaluation

POST `/api/v1/workflows/{id}/nodes/{nodeId}/eval`

GET `/api/v1/workflows/{id}/nodes/{nodeId}/eval-runs`

GET `/api/v1/workflows/{id}/nodes/{nodeId}/few-shot`

GET `/api/v1/eval-runs/{id}`

PUT `/api/v1/eval-runs/{id}/cases/{messageId}/expected`

Prompt evaluation ​

POST /api/v1/workflows/{id}/nodes/{nodeId}/eval ​

GET /api/v1/workflows/{id}/nodes/{nodeId}/eval-runs ​

GET /api/v1/workflows/{id}/nodes/{nodeId}/few-shot ​

GET /api/v1/eval-runs/{id} ​

PUT /api/v1/eval-runs/{id}/cases/{messageId}/expected ​

Prompt evaluation

POST `/api/v1/workflows/{id}/nodes/{nodeId}/eval`

GET `/api/v1/workflows/{id}/nodes/{nodeId}/eval-runs`

GET `/api/v1/workflows/{id}/nodes/{nodeId}/few-shot`

GET `/api/v1/eval-runs/{id}`

PUT `/api/v1/eval-runs/{id}/cases/{messageId}/expected`