AI Judge
Ask an AI model a fixed question about evidence a flow already holds, branch on its labeled verdict, and control what it costs with token caps, caching, and a daily budget.
Use a Judge step to let an AI model classify evidence before the flow decides what to do. You write a fixed question and a set of labels, and choose which values from the trigger and earlier steps the model sees. The model answers with one of your labels, a confidence, and a short reason.
Add a Judge step
Add the block
Drag Judge from the block palette onto the flow.
Write the question
Write one fixed question, for example "Is this sign-in consistent with the user's normal behavior?". The question can't contain values from the run.
Set the labels
Add the labels the model must choose from, for example benign, suspicious, and malicious.
Choose the evidence
Pick the values the model should see, such as alert.entities.user or the output of an earlier search step. Optionally set an entity, a single value the verdict is about.
Branch on the verdict
Add a Condition after the Judge step that reads results.<step id>.verdict, for example results.judge.verdict = "malicious".
What the model gets
The model gets only:
- your question;
- your labels;
- the evidence values and the entity you chose, with the path each one came from.
nano passes the evidence as data, inside a fenced block, and tells the model to ignore any instructions in it. The model gets no tools, can't run searches, and can't see anything you didn't select.
What the step returns
| Output | Contents |
|---|---|
results.<id>.verdict | One of your labels, or unknown |
results.<id>.confidence | 0 to 1. Always 0 for unknown. |
results.<id>.why | The model's reason, up to 1,024 characters |
results.<id>.tokens | Tokens counted for this call. 0 for a cached verdict. |
results.<id>.cached | true when the verdict came from the cache |
If the model's reply isn't a JSON verdict, or names a label you didn't define, the verdict is unknown with confidence 0. nano never guesses a label.
Limits
| Setting | Limit |
|---|---|
| Question | Up to 2,048 characters |
| Labels | 1 to 32. Letters, digits, _, and -, up to 64 characters each, unique ignoring case. unknown is reserved. |
| Evidence | 1 to 32 values, at most 32 KiB together |
| Entity | One value (text, number, or true/false), at most 1,024 bytes |
| Token cap | 2,000 by default, up to 8,000 for each call |
| Timeout | 60 seconds |
| Cache | 6 hours by default, from 1 second up to 24 hours |
Cost controls
The token cap
Before each call, nano reserves the step's token cap from the daily budget. After the call, it returns the part it didn't use. Your AI provider doesn't report token counts to nano here, so nano estimates the tokens used from the length of the question, evidence, and reply, and counts at most the cap.
The cache
nano caches each verdict for the step's cache time. The cache key covers the question, labels, entity, evidence, and token cap, so the same question about the same evidence returns the cached verdict and uses no tokens.
The daily token budget
All Judge steps on your deployment share one daily budget. Set it at Settings → SOAR → AI Judge daily token budget, from 0 to 1,000,000,000 tokens. The default is 250,000. Changing it needs settings:system.
When a call's token cap doesn't fit in what is left of the day's budget, nano doesn't call the model. The step fails with judge_budget_exhausted and the run fails. A budget of 0 stops all Judge calls. Cached verdicts still work.
How Judge results are recorded
Each Judge call is recorded as a step of its run: the verdict, confidence, reason, tokens, and whether it was cached. Open the run on Automation → Runs to see it. Judge calls in tests use the budget too.
Troubleshoot
The step fails with judge_budget_exhausted
The daily budget is used up. Raise AI Judge daily token budget, lower the step's token cap, or wait for the next day. Check which flows run Judge steps most often on Automation → Runs.
The step fails with judge_timeout or judge_provider_error
The AI provider didn't answer within 60 seconds, or returned an error. Check the provider in AI & pivt. nano returns the reserved tokens to the budget.
The verdict is always unknown
The evidence doesn't decide the question, or the model isn't answering in the expected format. Make the question narrower, check that the evidence paths hold values for this trigger, and give labels that cover the possible answers.