Skip to main content

​
Flaggers

Flaggers are Latitude’s built-in automatic annotators. They inspect completed sessions for common failure categories and annotate the trace where the problem shows up when they find one. Those annotations are attributed to Latitude rather than to a person, and they feed signal discovery, evaluation alignment, and score analytics exactly like human feedback. Use flaggers when you want project-wide coverage of failure patterns you already know about, without reviewing every conversation by hand. They are useful for cases that are easy to miss in manual review: low-volume safety failures, recurring frustration, or behavior that only becomes visible across a long session. Detection is session-wide. A flagger reads the whole session, the conversation your agent had across every trace it produced, because most of these failures only make sense in the context of the turns around them. The annotation itself lands on one trace inside that session, anchored to the message that shows the issue, so it appears in trace-level views and analytics like any other annotation. A trace that arrives without a session id becomes a single-trace session, so nothing goes unscreened.

​
Available flaggers

Flaggers come in four groups, the same ones the settings page uses. User-side signals and Agent behavior use an LLM, so they are sampled and billed by AI usage. Cost and efficiency and Response validity are deterministic: they cost nothing and run on every eligible session while they are enabled.

​
User-side signals

LLM checks on what the person on the other side of the conversation is doing.
The user shows annoyance, disappointment, loss of trust, or has to restate themselves because the agent is not helping.Not flagged: neutral clarifying questions, or an isolated terse reply with no other evidence of frustration.
Prompt injection, instruction-hierarchy attacks, policy evasion, tool abuse aimed at bypassing guardrails, role escape attempts, or the agent actually following one of those attempts.Not flagged: harmless roleplay, or an ordinary unsafe request that the agent correctly refuses.
Workplace-inappropriate or toxic content: explicit profanity, sexual content, abusive harassment, hate speech, identity-based slurs, or graphic violent language.Not flagged: benign anatomy or health discussion, mild romance, neutral policy discussion about unsafe content, or colloquial language with no real toxicity.

​
Agent behavior

LLM checks on failure modes in your agent’s own output.
The agent declines, deflects, or over-restricts a request that is allowed and answerable within its policy and capabilities.Not flagged: a correct refusal, where the request is unsafe, unsupported, or missing context the agent needs.
The agent gives a shallow partial answer, stops early without justification, skips context it was handed, or pushes work back onto the user that it should have done itself.Not flagged: work genuinely blocked by missing access, missing context, or policy.
The agent loses relevant context from earlier in the same conversation: it repeats settled questions, contradicts established facts, or ignores constraints the user already gave.Not flagged: ambiguity that was never resolved, or context the user never provided.
A task the user or the system prompt assigned was not completed, and the user’s later messages show it by demanding a retry, repeating the request, or pointing at what is missing.Only closed episodes are judged. The agent’s latest response is not flaggable until the user has reacted to it, so this flagger fires on a later screening pass rather than immediately.
The agent cycles through the same tools or tool sequences, oscillates between states, or piles up tool calls without moving toward the goal. Three or more identical calls match deterministically for free; the ambiguous cases go to the LLM pass.Not flagged: legitimate retries after a transient error, or iterative refinement that is visibly converging.
The agent ignores a failed tool call and confidently continues, presenting results the call never returned, narrating the action as done, or answering from fabricated data.Not flagged: acknowledging the failure, retrying, hedging, or answering from other evidence actually present in the conversation.
The agent’s output exposes personal identifiers (emails, phone numbers, card or government numbers, personal records) belonging to a third party or that the user never supplied.Not flagged: the user’s own data echoed back, placeholder or fictional examples, masked identifiers, or public business contact details.

​
Cost and efficiency

Deterministic checks on token waste. Free, and run on every eligible session while enabled.
Large multi-turn sessions where prompt caching is active but fewer than 30% of input tokens were served from cache, which usually means caching is broken.

​
Response validity

Deterministic checks on the shape of the response. Free, and run on every eligible session while enabled.
Empty, whitespace-only, or otherwise degenerate assistant responses where a substantive answer was expected.
Tool responses that are malformed, duplicated, explicitly failed, or that call a tool the agent never declared.
Structured output that is truncated or cannot be parsed as JSON.

​
How a flagger runs

Every project starts with all flaggers provisioned. Detection happens in two passes. Screening runs on 100% of sessions and costs nothing. It gathers cheap evidence from the session (tool errors, error spans, repeated tool loops, latency and cost outliers, conversation moments, text patterns) and runs every enabled check that needs no model. Deterministic matches are written straight away. LLM classification runs for one session and one flagger at a time. A classifier reads the conversation together with the screening evidence, and a second model pass reviews the proposed annotation before it is saved. This pass is billed by AI usage. Latitude waits for a session to settle before screening it. Sessions re-screen as new turns arrive, and at most one annotation is published per flagger per distinct piece of flagged content, so a growing conversation does not accumulate copies of the same finding.

​
Sampling

Sampling applies only to the LLM flaggers, and it defaults to 10%. Deterministic flaggers have no sampling control: they are free, so an enabled one runs on every eligible session. Turning a flagger off stops it either way. Sampling is not the whole story on coverage. When screening finds evidence pointing at a specific category, that flagger’s LLM pass runs regardless of the sampling rate. Sampling governs the sessions where nothing in particular stood out. This is why lowering sampling reduces volume but does not silence a flagger that keeps matching on real evidence. Sampling never affects search, filtering, or manual investigation. Every session stays fully queryable.

​
Configure flaggers

Open Project Settings → Flaggers to manage a project. Each row has an on/off switch, and LLM rows also have a sampling slider showing whether they are free or billed by AI usage. Use-case presets turn on a sensible set in one click. They cover support agents, coding agents, sales agents, tool workflow agents, knowledge-base agents, structured extraction, and safety-sensitive agents. Picking a preset replaces the current selection and leaves sampling rates alone. The same presets appear during project onboarding. You can also toggle flaggers through the API with PATCH /v1/projects/{projectSlug}, passing flagger slugs to booleans in the flaggers field. The API controls enablement only; sampling is set from the settings page.
Flagger categories are defined by Latitude. You choose which ones run and how aggressively the LLM ones sample, not the underlying category definition. To detect something specific to your product, use an evaluation or a saved search instead.

​
Tuning advice

Run a project for a week with the defaults before changing anything, then look at what each flagger actually caught.
  • If a flagger’s matches are mostly wrong for your domain, lower its sampling.
  • If you keep annotating cases by hand that a flagger should have caught, raise its sampling.
  • If a flagger is wrong for your product right now, for example NSFW on a creative-writing assistant, turn it off and revisit when the product changes.

​
Suppression

Some categories overlap, and one of them firing makes the other misleading. Latitude skips the second flagger in those cases:
  • Refusal is skipped when jailbreaking or NSFW is in play, because refusing an adversarial or toxic request is the correct behavior.
  • Laziness is skipped when thrashing found a tool loop, because being stuck in a loop is a different failure from punting work back to the user.
Suppression happens during screening, so it does not wait for the suppressing flagger to confirm a match. Evidence pointing at jailbreaking is enough to skip refusal for that session, even if the jailbreaking classifier later decides there was nothing there. A suppressed flagger writes no annotation, which explains gaps you might otherwise read as a miss.

​
Flaggers, search, and manual annotations

These work together: flaggers generate signal automatically, search scopes investigation, and manual annotations add the judgment neither can produce.