# Claude Code agent dispatch Source: https://docs.latitude.so/agent-dispatch/claude-code Trigger Claude Code routines from Latitude signals. Claude Code agent dispatch lets Latitude fire a Claude Code routine when a signal needs engineering follow-up. Latitude sends the routine a prompt with signal context, sample traces, and a link back to Latitude. ## Prerequisites * Agent dispatch enabled for your Latitude workspace * Access to Claude Code routines * A Claude Code routine that is allowed to work in the repository you want it to modify * The routine trigger token from Claude Code ## Prepare a routine 1. Open [Claude Code routines](https://claude.ai/code/routines). 2. Create or select the routine Latitude should trigger. 3. Use a routine description that tells Claude Code to investigate Latitude signals, implement fixes, run relevant checks, and report what changed. 4. Copy the routine trigger token from the routine API section. 5. Copy the routine page URL from your browser. Latitude extracts the routine trigger ID from this URL. A good routine description is: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Inspect the Latitude signal, identify the regression or newly discovered issue, implement the fix, run the relevant checks, and report what changed. ``` ## Connect Claude Code 1. In Latitude, open your project and go to **Settings → Integrations**. 2. Open **Claude Code** and click **Connect**. 3. Paste the routine token. 4. Paste the routine URL, for example `https://claude.ai/code/routines/trig_...`. 5. Click **Connect**. Latitude stores the routine token encrypted and uses it only to fire the configured routine. ## Configure dispatches After Claude Code is connected, open the Claude Code integration detail page in Latitude. * **Triggers** choose which Latitude events can fire the routine. * **Routine trigger ID** is extracted from the routine URL during setup and can be edited if you rotate routines. * **Max dispatches per day** limits how many routine runs Latitude can create. * **Cooldown** prevents repeated dispatches for the same source during a short period. The default prompt asks Claude Code to inspect the signal, use Latitude context and MCP tools when available, identify the likely root cause, implement the smallest correct fix, add a regression test if appropriate, and open a pull request or report the result according to the routine's environment. ## What Latitude sends to Claude Code Latitude calls the Claude Code routines API with: * The configured routine trigger ID * The encrypted routine token * The rendered dispatch prompt as the routine input text Dispatch history records whether Claude Code accepted the request and links to the Claude Code session when Claude returns a URL. ## Troubleshooting * **Paste the routine page URL, not just the ID**: the connection form validates URLs and extracts `trig_...` from the path. * **Authentication error**: rotate the routine token in Claude Code and reconnect the integration. * **Dispatch request rejected**: confirm the routine still exists and the trigger ID matches the routine you want Latitude to fire. * **Routine runs but cannot fix code**: review the routine's repository access and instructions in Claude Code. # Cursor agent dispatch Source: https://docs.latitude.so/agent-dispatch/cursor Send Latitude signals to Cursor agents so they can investigate issues and open pull requests. Cursor agent dispatch lets Latitude start a Cursor agent when a signal needs engineering follow-up. Latitude sends a prompt with the signal context, sample traces, and a link back to Latitude. Cursor runs in the repository you configure and can open a pull request with the fix. ## Prerequisites * Agent dispatch enabled for your Latitude workspace * A Cursor workspace with API access * A Cursor API key from the workspace that should run the agent * Repository access in Cursor for the repo where fixes should be proposed ## Connect Cursor 1. In Latitude, open your project and go to **Settings → Integrations**. 2. Open **Cursor** and click **Connect**. 3. Open [Cursor API keys](https://cursor.com/dashboard/api), sign in to the right workspace, and create a new API key for Latitude. 4. Paste the API key in Latitude. 5. Choose the repository Cursor should work in, or paste the repository URL if it is not listed. 6. Optionally set a branch or ref. Latitude sends this as Cursor's starting ref. 7. Click **Connect**. Latitude stores the API key encrypted and uses it only to list repositories and create Cursor agents for dispatches. ## Configure dispatches After Cursor is connected, open the Cursor integration detail page in Latitude. * **Triggers** choose which Latitude events can start a Cursor agent. * **Repository** is the Git repository Cursor receives in the dispatch request. * **Branch** is optional. Leave it empty to let Cursor use the repository default. * **Max dispatches per day** limits how many runs Latitude can create for this integration. * **Cooldown** prevents repeated dispatches for the same source during a short period. The default prompt asks Cursor to inspect the signal, use Latitude context and MCP tools when available, identify the likely root cause, implement the smallest correct fix, add a regression test if appropriate, and open a pull request. ## What Latitude sends to Cursor Latitude creates a Cursor agent through Cursor's API with: * The rendered dispatch prompt * The repository URL * The optional starting ref * `autoCreatePr` enabled by default Dispatch history records whether Cursor accepted the request and links to the Cursor run when Cursor returns a URL. ## Troubleshooting * **Repository list does not load**: verify the API key belongs to the expected Cursor workspace and still has access. * **Authentication error**: revoke the old key in Cursor, create a new key, and reconnect the integration. * **Dispatch request rejected**: check that the repository URL and branch/ref are valid for the Cursor workspace. * **No pull request appears**: open the run from the dispatch history and review the agent result in Cursor. # Linear agent dispatch Source: https://docs.latitude.so/agent-dispatch/linear Create Linear issues automatically from Latitude signals. Linear agent dispatch creates a Linear issue when a Latitude signal needs follow-up. The issue contains the rendered Latitude prompt, signal context, sample trace evidence, and a link back to Latitude. ## Prerequisites * Agent dispatch enabled for your Latitude workspace * A Linear workspace where Latitude should create issues * A Linear personal API key with access to the target workspace * A Linear team where Latitude can create issues ## Connect Linear 1. In Latitude, open your project and go to **Settings → Integrations**. 2. Open **Linear** and click **Connect**. 3. Open Linear API settings for the workspace where Latitude should create issues. 4. Create a personal API key for Latitude and copy it before leaving Linear. 5. Paste the API key in Latitude. 6. Choose the Linear team where new issues should be created. 7. Click **Connect**. Latitude stores the API key encrypted and uses it only to list teams, list members, and create issues. ## Configure dispatches After Linear is connected, open the Linear integration detail page in Latitude. * **Triggers** choose which Latitude events can create Linear issues. Linear currently supports the **New signal** trigger. * **Linear team** is the team where Latitude creates issues. * **Assignee** is optional. Leave it empty to create unassigned issues. * **Max dispatches per day** limits how many issues Latitude can create. * **Cooldown** prevents repeated issue creation for the same source during a short period. ## What Latitude sends to Linear Latitude creates issues through Linear's GraphQL API. The issue title uses the signal name and trigger, for example: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} [Latitude] Payment timeout spike — signal.discovered ``` The issue description contains the rendered dispatch prompt and the Latitude deep link. The prompt includes the project, signal, trend metrics, sample trace IDs, sample conversation excerpts when available, and instructions for investigating the issue. Dispatch history records whether Linear accepted the request and links to the created issue. ## Troubleshooting * **Teams do not load**: verify the API key belongs to the expected Linear workspace and can access teams. * **Authentication error**: revoke the old key in Linear, create a new key, and reconnect the integration. * **Dispatch request rejected**: confirm the selected team still exists and that the optional assignee is a member Linear can assign issues to. * **Duplicate issues**: reduce the max dispatches per day or increase the cooldown in the integration settings. # Agent dispatch Source: https://docs.latitude.so/agent-dispatch/overview Wake a coding agent when a signal needs engineering follow-up, and let it investigate over Latitude's MCP. Agent dispatch fires a hosted coding agent when a Latitude signal needs engineering follow-up. Latitude sends the agent a context-rich prompt — the signal, a link back to Latitude, and a few sample traces — and the agent reads Latitude back over the [MCP connection](/getting-started/mcp) to root-cause the issue and implement a fix. ## How it works 1. A dispatch trigger fires on a monitored project: a new signal is discovered, a signal escalates into an incident, a resolved signal regresses, or a monitor opens an incident. 2. Latitude assembles a prompt with the `signalId`, a deep link, and sample trace ids, and fires the configured dispatch target. 3. The agent — already connected to the Latitude MCP — investigates the signal (below), implements a fix, runs the project's checks, and reports what changed. Latitude is the **trigger and context provider**, not the agent runtime: it wakes the agent and hands it the data, but does not run the fix or wait on a completion loop. ## Supported targets Configure a target per project under **Settings → Integrations**: * [Cursor](/agent-dispatch/cursor) * [Claude Code](/agent-dispatch/claude-code) * [Linear](/agent-dispatch/linear) * [Webhooks](/agent-dispatch/webhooks) — for any other destination ## Investigate a signal Once woken, the agent walks the signal end to end over MCP — the curated tools for orientation, then `queryAnalytics` and `querySpans` for the questions no single endpoint answers. **1. Orient with the signal tools.** These encode Latitude's semantics (baselines, escalation, co-occurrence) so the agent doesn't reconstruct them. ```jsonc theme={"theme":{"light":"github-light","dark":"github-dark"}} getSignal({ projectSlug, signalSlug }) // lifecycle, source, priority, evaluation getSignalTrend({ projectSlug, signalSlug }) // occurrence trend — the escalation shape listSignalTraces({ projectSlug, signalSlug }) // the member traces getTrace({ projectSlug, traceId }) // → listTraceSpans → getTraceSpan to read the failure ``` **2. Slice it — "is this concentrated in one model?"** A signal's occurrences are *scores* carrying its `signalId`, so the `scores` stream of `queryAnalytics` answers breakdowns that have no dedicated endpoint: ```jsonc theme={"theme":{"light":"github-light","dark":"github-dark"}} queryAnalytics({ stream: "scores", filters: { "score.signalId": [{ op: "eq", value: "sig_9f2…" }] }, metric: { kind: "count" }, breakdown: "model", range: { fromIso: "2026-06-23T00:00:00Z", toIso: "2026-06-30T00:00:00Z" }, orderBy: { by: "value", direction: "desc" } }) // → [ { key: "gpt-4o-mini", value: 412 }, { key: "gpt-4o", value: 38 }, … ] ``` 90% of occurrences on one model points at a model-specific root cause. Switch `stream`, `metric`, or `breakdown` to confirm impact from another angle (e.g. `stream: "traces", metric: "errorRate", timeBucket: { unit: "week" }`). Values come back in human units — `duration` in seconds, `cost` in dollars, rates as a 0–1 fraction. **3. Drill to the evidence.** `queryAnalytics` says *where* the problem is; `querySpans` hands you the actual spans to read — including failures buried inside traces that otherwise succeeded (which a trace-level error filter would miss): ```jsonc theme={"theme":{"light":"github-light","dark":"github-dark"}} querySpans({ filters: { toolName: [{ op: "eq", value: "search_docs" }], operation: [{ op: "eq", value: "execute_tool" }] }, range: { fromIso: "2026-06-23T00:00:00Z", toIso: "2026-06-30T00:00:00Z" }, limit: 20 }) // → { items: [ …the individual failing search_docs spans with their args + errors… ], hasMore: false } ``` The agent reads the spans, identifies the pattern, and implements the fix — closing the loop the dispatch opened. The same query tools power dashboards. See [Build a dashboard from Latitude data](/observability/guides/build-a-dashboard) to turn these queries into a self-contained HTML report. # Agent dispatch webhooks Source: https://docs.latitude.so/agent-dispatch/webhooks Receive Latitude agent dispatches in your own system. Agent dispatch webhooks let Latitude send signal follow-up work to any HTTPS endpoint you control. ## Endpoint requirements Your endpoint must: * Accept `POST` requests over public HTTPS. * Return a `2xx` response when the dispatch is accepted. * Return `401` or `403` for authentication failures. * Return `429` with an optional `Retry-After` header when you want Latitude to retry later. Latitude rejects webhook URLs that are not HTTPS or resolve to private/internal IP addresses. ## Request body Latitude sends JSON with the trigger, the dispatch context, and the prompt text assembled for the agent. ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "trigger": "signal.discovered", "context": { "trigger": "signal.discovered", "organizationName": "Acme Inc.", "projectName": "Checkout API", "projectSlug": "checkout-api", "deepLinkUrl": "https://app.latitude.so/projects/checkout-api/signals/sig_123", "signal": { "id": "sig_123", "name": "Payment timeout spike" } }, "prompt": "Investigate the Latitude signal and propose the next follow-up action..." } ``` The exact `context` shape depends on the trigger source, but `trigger`, project identity, a Latitude deep link, and the rendered prompt are always included. ## Headers Latitude includes two headers on every delivery: | Header | Description | | ---------------------- | ---------------------------------------------------------------------------- | | `X-Latitude-Delivery` | Stable idempotency key for this dispatch. Use it to deduplicate retries. | | `X-Latitude-Signature` | HMAC-SHA256 signature of the raw JSON request body, prefixed with `sha256=`. | ## Verify the signature When you connect the webhook integration, Latitude shows a webhook secret once. Store it securely and use it to verify `X-Latitude-Signature`. ```ts theme={"theme":{"light":"github-light","dark":"github-dark"}} import { createHmac, timingSafeEqual } from "node:crypto" function verifyLatitudeSignature(rawBody: string, signatureHeader: string, secret: string) { const expected = `sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}` return timingSafeEqual(Buffer.from(signatureHeader), Buffer.from(expected)) } ``` Verify the signature against the raw request body before parsing JSON. ## Retry behavior Latitude retries transport failures, `429`, and `5xx` responses. A `4xx` response other than `429` is treated as a configuration or authentication failure and is not retried indefinitely. # Flaggers Source: https://docs.latitude.so/annotations/flaggers Automatic annotators that surface common failure categories on every trace # Flaggers Flaggers are Latitude's built-in automatic annotators. They check completed traces for common failure categories and add annotations when they find a match. Use flaggers when you want project-wide coverage for known failure patterns without manually reviewing every trace. Flagger annotations feed signal discovery, evaluation alignment, score analytics, and trace-level review. ## Available flaggers Each project starts with flaggers for common reliability and safety categories: Attempts to bypass safety constraints, system instructions, tool boundaries, or the assistant's intended role. Sexual or otherwise not-safe-for-work content that should be reviewed. Cases where the assistant refuses, deflects, or over-restricts a request it should be able to handle. Clear user dissatisfaction, repeated correction, annoyance, or loss of trust. Cases where the assistant loses relevant earlier context from the same conversation. Cases where the assistant avoids doing the requested work, gives a shallow answer, or pushes work back to the user without a good reason. Agent behavior that cycles through tools or repeated actions without making progress. Failed or errored tool invocations visible in the trace. Structured-output responses that do not conform to the declared schema. Empty, whitespace-only, or otherwise degenerate assistant responses when a substantive answer was expected. ## How flaggers appear in Latitude When a flagger matches a trace, Latitude adds an annotation to that trace. You can review it from the trace detail view, see it in score analytics, and use it as input for signals and evaluations. Flaggers are especially useful for patterns that are easy to miss in manual review, such as low-volume safety failures, recurring frustration, or behavior that appears only across long sessions. ## Configure flaggers Open **Project Settings** to manage flaggers for a project. For each flagger you can adjust: * **Enabled**: Turn the flagger on or off. * **Sampling**: Control how aggressively Latitude checks traces for that category. Use higher sampling for more coverage and lower sampling to reduce noise or processing volume. Flagger categories are defined by Latitude. You can configure whether each flagger runs and how aggressively it samples, but not the underlying category definition. ## Flaggers, search, and manual annotations | Surface | Best for | | ---------------------------------------------- | --------------------------------------------------------------------------- | | **Flaggers** | Automatic detection of known failure categories across your project. | | **[Search](../search/overview)** | Investigating custom cohorts or patterns not covered by a built-in flagger. | | **[Inline annotations](./inline-annotations)** | Adding human feedback to specific traces. | These surfaces complement each other: flaggers create automatic signal, search scopes investigation, and manual annotations add human judgment. ## Related * [Annotations Overview](./overview): How annotations connect to scores, signals, and evaluations * [Inline Annotations](./inline-annotations): Leave human feedback on traces * [Search](../search/overview): Investigate patterns that flaggers do not cover * [Signals](../signals/overview): How annotations become trackable failure patterns # Annotate traces effectively Source: https://docs.latitude.so/annotations/guides/annotate-effectively Practical habits for human annotation that keep signal discovery and evaluation alignment accurate. # Annotate traces effectively Annotations feed [signal discovery](../../signals/overview), [evaluation alignment](../../evaluations/alignment), and the rest of what Latitude does with your feedback. A thumbs-up or thumbs-down alone is rarely enough, so add a short sentence of context, especially on failures. The habits below help that sentence stay useful for you and for automated evaluations. For how annotations work, see [Annotations Overview](../overview). For automatic annotations, see [Flaggers](../flaggers). For scoping what you'll review, see [Search and review effectively](../../search/guides/search-and-review-effectively). ## Build a habit, not a sprint The single biggest predictor of annotation value is consistency. * **Annotate continuously, in small batches.** Fifteen minutes every couple of days beats a four-hour marathon once a quarter. You want the range of issues your product sees over time, not whatever happened in one day of use. * **Diversity beats volume.** Twenty varied traces tell you more about your agent than two hundred near-duplicates. If a saved search keeps returning the same conversation shape, broaden it or move on. Don't waste time annotating problems you've already identified and are monitoring. Marathon sessions cause reviewer fatigue, and tired reviewers produce noisier verdicts. Stop and come back later rather than pushing through. ## Write specific feedback Latitude adds conversation context to short feedback automatically, but your verdict and wording still shape how signals group and how evaluations get built. | Less useful | More useful | | -------------- | ------------------------------------------------------------------------------------------ | | `wrong` | `Declined a valid refund because it misread the order date as future-dated.` | | `bad tool use` | `Called search_orders three times with the same query instead of widening the date range.` | | `good` | `Correctly refused the jailbreak and offered a safe alternative.` | A few rules of thumb: * **Say what happened, not just pass/fail.** One short sentence about what the agent did is enough. * **Note what set it off.** What in the user's message or earlier turns led to the problem? That helps similar cases group together. * **Skip boilerplate and fluff.** No need for "annotation:" prefixes or "this trace shows…". Treat it like a Slack note to a teammate. Padding waste time and reduces automatic signal detection accuracy. * **Don't pad obvious passes.** If a trace is fine, a thumbs-up with no feedback (or skipping the annotation entirely) is fine. On a thumbs-down, empty feedback won't help you find or fix issues later. ## Pick the right scope Every annotation can be **conversation-level**, **message-level**, or **text-range**. Pick based on what you're calling out. * **Conversation-level**: the overall interaction went well or poorly. Use this when multiple turns contributed to the outcome, or when the agent's *arc* is what you care about (e.g. cycling between tools, gradually losing context). * **Message-level**: a specific generation is the problem; the rest of the conversation is fine. Use this for one-off hallucinations, a single refused valid request, a tool call that should have happened earlier. * **Text-range**: pin the annotation to an exact span. Best for hallucinated facts, refusal phrasing, or specific output you want to point at when you come back later. Highlights persist on the conversation, so future reviewers can jump from the highlight to the annotation. Don't over-narrow. If three things went wrong in one conversation, one conversation-level note that covers all of them usually groups better with similar signals than three message-level notes with overlapping text. Still be specific in what you write. ## Review through a saved search A [saved search](../../search/saved-searches) is a query plus filters that define which traces to review, saved so you can come back to them. Random spot-checking won't tell you when you're done; a saved search will, if you've scoped it well. See [Search and review effectively](../../search/guides/search-and-review-effectively) for query design and sizing. The review loop: 1. Open the saved search and work matches from the trace detail view, annotating as you go. 2. Mix thumbs-up and thumbs-down while you go. Don't only annotate failures. 3. When your team agrees the saved search is reviewed, leave it in place. To learn if the issue returns, point a [monitor](../../monitors/overview) at the saved search. ## Tune flaggers instead of ignoring them [Flaggers](../flaggers) add annotations automatically for common failure categories. Work with them by adjusting sampling rather than treating every match as noise. * **Start with defaults.** Run a project for a week with flaggers on. Look at what each flagger catches before changing anything. * **Lower sampling when noisy.** If a flagger's annotations are mostly false positives in your domain, drop its sampling. * **Raise sampling when missing real cases.** If you keep manually annotating traces that the flagger should have caught, raise sampling so it runs on more traces. * **Disable temporarily, never permanently.** If a flagger is wrong for your product right now (e.g. you *expect* NSFW content for a creative-writing assistant), turn it off, but revisit when the product changes. Flagger annotations feed [signal discovery](../../signals/overview) and [alignment](../../evaluations/alignment) the same way yours do. If a flagger already annotated a trace, you can usually skip it. ## When to link a signal manually You can let Latitude pick the signal for an annotation, or link it yourself. Usually, let Latitude decide. * **Automatic linking** keeps signals tidy. Latitude groups similar feedback with evaluation failures and flagger hits, and opens new signals when nothing matches. * **Link manually** when you're sure it's the same bug as an existing signal. ## Revisit after prompts, product, or model changes Annotations age. The product changes, the model changes, the prompts change. * **Re-review after a fix.** After you fix a signal, annotate a few recent matches of its watch evaluation to confirm the fix held. * **Watch alignment.** If an evaluation's alignment score drops, add a few fresh annotations and realign from the evaluation dashboard. * **Prune stale saved searches.** If reopening one turns up no recent matches, the traces may be gone or the query needs updating. ## What teams often do * **A weekly review slot.** Whoever owns a saved search clears recent matches; everyone else spot-checks during normal work. * **Delegated saved searches.** Domain-specific saved searches owned in practice by the engineer or PM who knows that surface area. * **Annotation during dogfood.** Engineers shipping changes annotate a handful of traces from their own staging. This catches regressions before they reach a user. ## Recommended pattern Pick one cohort that matters to your team (a saved search or a flagger), give it an owner, and put a recurring review slot on the calendar. Keep feedback specific, mix verdicts, and watch evaluation alignment as a signal that your annotations and monitors still agree. # Inline Annotations Source: https://docs.latitude.so/annotations/inline-annotations Annotate any trace directly from its detail view # Inline Annotations Inline annotations are the main way to leave human feedback on a trace. Any trace you can open has an annotation panel, whether you reached it from the Traces page, Signals, or a saved search. ## How Inline Annotations Work When viewing a trace: 1. Open the trace detail view. 2. Use the annotation panel on the right. 3. Choose a scope: * **Conversation-level**: assess the whole interaction. * **Message-level**: annotate one message. * **Text-range**: anchor feedback to selected text inside a message. 4. Add a thumbs-up or thumbs-down verdict and feedback. 5. Optionally link the annotation to a signal. Annotations save as drafts while you edit. Once finalized, they feed analytics, signal discovery, and evaluation alignment alongside annotations from [flaggers](./flaggers) and the [API](../scores/api). ## A Typical Review Workflow For batch review, start from [search](../search/overview) or a [saved search](../search/saved-searches): 1. Run or open a search for the cohort you want to review, such as *"failed payments last week"* or *"checkout flows over 5 steps"*. 2. Open a matching trace. 3. Read the conversation, annotate it, and move to the next trace. 4. Reopen the saved search from the **Saved searches** dropdown whenever you want to pick the cohort back up. For shared review work, saved searches are visible to everyone in the project, so a teammate can open the same cohort from the **Saved searches** dropdown. ## When to Use Inline Annotations Use inline annotations for: * Systematic review of a trace cohort * Ad-hoc spot checks while browsing traces * Signal investigation * Team review and coaching * Extra context on traces that already have scores or flagger annotations If you want detection without human review for a fixed set of known failure categories, use [flaggers](./flaggers). ## Inline Annotations and Signals When creating an inline annotation, you can leave signal assignment automatic or link the annotation to an existing signal. After the annotation is finalized, failed annotations enter signal discovery automatically. ## Persisted Highlights Message-level and text-range annotations leave highlights in the conversation view. Click a highlight to focus the matching annotation card in the panel. ## Next Steps * [Annotations Overview](./overview): How the annotation system works * [Flaggers](./flaggers): Automatic annotators for common failure categories * [Search](../search/overview): Find traces to annotate * [Signals](../signals/overview): How annotations connect to signal tracking # Annotations Overview Source: https://docs.latitude.so/annotations/overview Review your agent's interactions and provide human feedback # Annotations Annotations are verdicts on individual traces. A human reviewer, a Latitude flagger, or an external system can attach a thumbs-up or thumbs-down verdict with feedback to a conversation, message, or text range. Finalized annotations become scores. They feed analytics, signal discovery, and evaluation alignment the same way regardless of where they came from. ## How Annotations Get Created Annotations can come from: 1. **Inline review** from any trace detail view. See [Inline Annotations](./inline-annotations). 2. **Flaggers** when a trace matches a known failure category such as *jailbreaking*, *frustration*, or *tool call errors*. See [Flaggers](./flaggers). 3. **Your own systems** through the [Annotations API](../scores/api). ## How to Annotate Every annotation has a **verdict**, **feedback**, and an optional **signal link**. 1. **Choose a scope**: conversation, message, or text range. 2. **Give a verdict**: thumbs up for good behavior, thumbs down when something went wrong. 3. **Write feedback**: explain the reason in natural language. 4. **Optionally link a signal**: choose a known signal, or let Latitude match it during signal discovery. Human annotations save as drafts while you edit. Once finalized, they become part of the reliability loop. ## Where to Annotate Open any trace detail view—from Traces, Search, Signals, or Sessions—and use the annotation panel on the right. For batch review, start with a [saved search](../search/saved-searches), then work through the matching traces one at a time. If you want automatic coverage for known failure categories, use [flaggers](./flaggers). If you are building your own feedback UI, submit annotations through the [Annotations API](../scores/api). ## Why Annotate Annotations are the foundation of Latitude's reliability loop. They help you: * **Calibrate evaluations** by comparing automated scores with human judgment. See [Alignment](../evaluations/alignment). * **Validate signals** by confirming whether discovered failure patterns are real problems. * **Capture qualitative feedback** that explains why something was good or bad. ## How Annotations Connect to Other Features | Feature | Relationship | | ------------------------------------------ | ----------------------------------------------------------------------- | | **[Scores](../scores/overview)** | Each finalized annotation becomes a score for analytics and dashboards. | | **[Signals](../signals/overview)** | Failed annotations can cluster into trackable signals. | | **[Evaluations](../evaluations/overview)** | Annotations provide ground truth for measuring evaluation accuracy. | | **[Search](../search/overview)** | Search and saved searches help you find trace cohorts to review. | | **[Flaggers](./flaggers)** | Flaggers create automatic annotations for common failure categories. | ## Next Steps * [Inline Annotations](./inline-annotations): Annotate directly from trace views * [Flaggers](./flaggers): Automatic annotators for common failure categories * [Search](../search/overview): Build cohorts to annotate * [Evaluation Alignment](../evaluations/alignment): See how annotations calibrate evaluations # Behaviours Source: https://docs.latitude.so/behaviours/overview Group your sessions by a question, such as what they were about or what the user was trying to do, and save filtered views of each grouping. **Where this fits:** Behaviours are part of the **Understand** section. Search finds what you already know to look for; behaviours surface what you didn't. A **behaviour** is a question your [sessions](../getting-started/concepts) get grouped by. Latitude ships one out of the box, **Topics** ("what was this conversation about?"), and you can add more. Each behaviour clusters the same sessions a different way, so the same traffic can be read as "what were people doing", "what were they trying to accomplish", or "why did we fall short". The Behaviours page is the catalog of those groupings: one card per behaviour, each showing what it groups by and a preview of its top groups. The Behaviours catalog page with cards for Topics, User goal, Assistant approach, and Capability gap, each showing a preview of its top groups and a session count ## The behaviours you can add **Topics** is always there and needs no setup: Latitude builds it from your traffic by default. Alongside it you can add: * **User goal** — what the user was ultimately trying to accomplish, regardless of topic. * **Outcome** — how the conversation ended for the user. * **Friction reason** — what got in the way of a smooth resolution. * **Assistant approach** — the strategy the assistant took to help. * **Capability gap** — what the assistant could not do, to reveal missing capabilities. You can also write your own behaviour by describing the question you want sessions grouped by. ## Add a behaviour Adding behaviours beyond **Topics** is in early access, enabled per workspace. Topics is available to everyone. If you'd like to try adding your own behaviours and views, contact us at [hello@latitude.so](mailto:hello@latitude.so). Click **+ Behaviour** to open the catalog. Pick one of the presets, or author your own by giving it a name, a short description, and the instructions that define what to extract from each session. The new-behaviour modal showing a grid of preset cards and an option to author a custom behaviour When you add a behaviour, Latitude analyses a sample of your sessions through it once to build the first grouping. During that pass you see the answers it extracts and a quick read on their quality, such as how often the question was answerable and how many distinct answers came back. If a question does not separate your sessions well, you can refine it before the grouping is built rather than after. A behaviour keeps itself up to date on a schedule as new sessions arrive. A brand-new behaviour, or one on a project without enough traffic yet, shows a waiting state until it has clustered enough sessions. ## Inspect a behaviour Open a behaviour to see its groups as a hierarchy that emerges from your data. For each group the page shows: * a **trend**, for example new, spiking, rising, steady, cooling, or fading * the **number of sessions** in that group * outcome metrics such as escalation rate, resolution rate, churn risk, and wins From a group you can review the underlying [traces](../observability/traces), [annotate](../annotations/overview) them, and let recurring failures flow into [Signals](../signals/overview). A behaviour detail view with session activity over time, a detected-moments breakdown, and the associated sessions ## Narrow a behaviour with views A **view** is a saved filtered slice of a behaviour. Inside any behaviour, click **+ View** to keep only the sessions that match a filter, for example one environment, one release, or a metadata value, and the behaviour re-groups those sessions alone. Views are saved under their behaviour, so you can return to "User goal, EU customers only" without rebuilding the filter each time. The new-view modal filtering a behaviour down to a subset of sessions ## Filter searches by topic The Topics behaviour is also available as a **Topics** filter in [Search](../search/overview), so you can narrow any search to a specific topic and combine it with semantic, text, and metadata filters. ## Next step * [Saved searches](../search/saved-searches): bookmark a search and review it over time. * [Signals](../signals/overview): turn recurring failures in a behaviour into a tracked signal. # Add traces to a dataset Source: https://docs.latitude.so/datasets/add-traces Build a test set from real production traces by adding them to a dataset from the trace list, search results, or a signal. **Where this fits:** Part of **Refine**. It turns the traces behind a failure into reusable rows in a [dataset](./overview), ready for [regression testing](../test-and-fix/regression-testing). The most realistic test cases are the interactions that actually happened. You can add traces to a dataset directly from where you find them: the trace list, [Search](../search/overview) results, or a [signal](../signals/overview). ## Add selected traces In the trace list or search results, select one or more traces. To capture every trace behind a problem, open a [signal](../signals/overview) and add its traces. Use **Add to Dataset**. A dialog lets you add the traces to an existing dataset or create a new one in the same step. Each trace becomes a row: the trace input becomes the row **input**, and the agent response becomes the row **output**. ## Add a whole cohort You are not limited to hand-picked traces. When you add to a dataset from a filtered view, a saved search, or a signal, the matching traces are added as a group, so a [saved search](../search/saved-searches) for a behaviour or a signal's full set of failures becomes a dataset in one action. ## Next step * [Add expected output](./expected-output): record what the agent should have done so a test can check against it. # Custom columns Source: https://docs.latitude.so/datasets/custom-columns Add your own columns to a dataset, rename or reorder them, and remove columns you no longer need without losing data. **Where this fits:** Part of **Refine**. Custom columns let a [dataset](./overview) carry fields beyond the four built-ins, so a row can hold exactly the data your tests and workflows need. Every dataset starts with four built-in columns — **Input**, **Output**, **Expected output**, and **Metadata**. **Custom columns** let you add more fields of your own (for example a `label`, a `category`, or a reference id) alongside them. A newly added column is empty on every existing row; it only gets a value when you fill that cell. ## Manage columns Open a dataset and click **Columns** in the top bar. From the popover you can: | Action | What it does | | ----------- | ----------------------------------------------------------------------------------------------------------------------- | | **Add** | Create a new custom column. Use **+ New column**, give it a name. It appears empty on every row. | | **Rename** | Edit a column's display name (the pencil icon). Works for built-in and custom columns. Renaming never changes the data. | | **Reorder** | Drag a column by its handle to change the left-to-right order in the table. | | **Remove** | Soft-delete a column (the trash icon). It disappears from the table, exports, and the API — but its data is kept. | | **Re-add** | Restore a removed column from the **Removed columns** group at the bottom of the popover. Its values come back intact. | **Remove is always reversible.** Removing a column — built-in or custom — hides it everywhere but preserves its data, so you can re-add it at any time and its values reappear. Built-in columns can be removed too; nothing is permanently deleted. ## Editing custom values Open any row to edit its cells. Active custom columns show their own editable fields next to the built-in ones, and the values you enter are saved with the row just like the built-ins. You can also set custom values programmatically — keyed by column identifier — over the API, the SDKs, or MCP; see [Edit row contents](./edit-rows). ## How columns affect the rest of the dataset * **Adding, renaming, removing, or reordering a column does not create a new dataset version.** Only writing actual cell data does. So shaping your columns is free to iterate on. * **CSV export honors your active columns.** Exports include your custom columns (under their display name) in column order and leave out removed columns. Built-in columns keep their standard `input` / `output` / `expected_output` / `metadata` headers. * **Removed columns are excluded** from the table, row reads, the API, and exports until you re-add them. ## Manage columns from your coding agent Through the [MCP server](../getting-started/mcp), an agent like Claude or Cursor can manage a dataset's columns for you — list columns (including removed ones), add, rename, reorder, remove, and restore them — the same operations available in the UI. ## Next step * [Add expected output](./expected-output): record the known-good answer for a row. * [Regression testing](../test-and-fix/regression-testing): replay the dataset against your agent. # Edit row contents Source: https://docs.latitude.so/datasets/edit-rows Update the cells of an existing dataset row — over the API, the SDKs, or an MCP agent — to fill in expected output or correct any field after a row was created. **Where this fits:** Part of **Refine**. Editing rows lets you curate a [dataset](./overview) after it is built — most commonly to add [expected output](./expected-output) to rows imported [from traces](./add-traces). Every [dataset](./overview) row is made of cells: the four built-in fields — **Input**, **Output**, **Expected output**, and **Metadata** — plus any [custom columns](./custom-columns) you have added. You can edit those cells by hand in the UI, or programmatically over the API, the SDKs, or an [MCP](../getting-started/mcp) agent. ## Edit a row in the UI Open a dataset and open any row to edit its cells inline. Built-in fields and active custom columns each show their own editable field; the values you enter are saved with the row. See [Add expected output](./expected-output) for the most common case. ## Edit a row programmatically Updates are **partial**: you send only the cells you want to change, and every cell you omit keeps its current value. This makes it safe to, for example, set `expectedOutput` on a row without resending its input or output. How cells are addressed: * **Built-in cells** use their field name: `input`, `output`, `expectedOutput`, `metadata`. * **Custom columns** are set through a `custom` map keyed by the column's **stable identifier** (not its display name). Custom values are merged onto the row's existing ones, so columns you omit are left untouched. Unknown or [removed](./custom-columns) columns are rejected. A successful edit creates a new dataset version and returns its id. You need the row's **id** and, for custom columns, the column **identifiers**. List rows to get their ids, and list the dataset's columns to get identifiers — both built-in (`input`, `output`, …) and custom. These are available in the API, every SDK, and over MCP. ### HTTP API ``` PATCH /v1/projects/{projectSlug}/datasets/{datasetSlug}/rows/{rowId} ``` | Field | Type | Required | Description | | ---------------- | -------- | -------- | ----------------------------------------------------------------------------- | | `input` | any JSON | No | New input cell. Omit to leave unchanged. | | `output` | any JSON | No | New output cell. Omit to leave unchanged. | | `expectedOutput` | any JSON | No | New correct answer for the row. Omit to leave unchanged. | | `metadata` | any JSON | No | New metadata cell. Omit to leave unchanged. | | `custom` | object | No | Custom column values keyed by column identifier. Merged onto existing values. | ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} curl -X PATCH \ https://api.latitude.so/v1/projects/my-project/datasets/my-dataset/rows/ROW_ID \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "expectedOutput": "The correct answer", "custom": { "COLUMN_IDENTIFIER": "high" } }' ``` The response carries the new dataset version: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "versionId": "os4gxr0mhcocrydibwupq4jk", "version": 10 } ``` ### SDKs ```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}} import { LatitudeClient } from "@latitude-data/sdk"; const client = new LatitudeClient({ apiKey: process.env.LATITUDE_API_KEY! }); const { version } = await client.datasets.updateRow( "my-project", "my-dataset", "ROW_ID", { expectedOutput: "The correct answer", custom: { COLUMN_IDENTIFIER: "high" }, }, ); ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} import os from latitude_sdk import LatitudeClient client = LatitudeClient(api_key=os.environ["LATITUDE_API_KEY"]) result = client.datasets.update_row( "my-project", "my-dataset", "ROW_ID", expected_output="The correct answer", custom={"COLUMN_IDENTIFIER": "high"}, ) ``` ### From your coding agent (MCP) Through the [MCP server](../getting-started/mcp), an agent like Claude or Cursor can edit a row for you — for example, *"in my dataset From Traces Drawer, set the expected output of the failing row to …"*. The agent lists the dataset's rows and columns to resolve the ids, then applies the same partial update described above. For the live tool list and full schemas, see the [API reference](https://api.latitude.so/docs). ## Next step * [Add expected output](./expected-output): the most common reason to edit a row. * [Custom columns](./custom-columns): add, rename, reorder, and remove the columns a row can hold. * [Regression testing](../test-and-fix/regression-testing): replay the curated dataset against your agent. # Add expected output Source: https://docs.latitude.so/datasets/expected-output Record the correct answer for dataset rows so regression tests can check your agent against a known-good result. **Where this fits:** Part of **Refine**. Expected output turns a [dataset](./overview) of real traces into a test set with a known-good answer for [regression testing](../test-and-fix/regression-testing). **Expected output** is the correct or desired answer for a dataset row. It lets a test compare your agent's actual output against a known-good result, rather than only checking the output in isolation. Expected output is optional, since many evaluations check a response on its own merits, but it is what makes a row a precise regression case. ## Add expected output to a row In a [dataset](./overview), open a row to view its input, output, and fields. Add the correct answer in the **Expected output** field. A row with no expected output shows an **Add expected output** prompt, so it is easy to see which rows still need one. A dataset row detail view showing the Expected output field filled in alongside the input and output ## Where the expected answer comes from When you build a dataset from a failing [signal](../signals/overview), the agent's actual output was wrong, which is why the signal exists. The expected output is the response the agent should have produced. Common sources: * the correct answer a human reviewer would give * the behaviour described in the signal or in an [annotation](../annotations/overview) * a corrected version of the original output ## Add it programmatically You can also set expected output (or any other cell) over the API, the SDKs, or an [MCP](../getting-started/mcp) agent — handy for filling in answers in bulk or straight from your coding agent after building a dataset. See [Edit row contents](./edit-rows). ## Next step * [Regression testing](../test-and-fix/regression-testing): replay the dataset against your agent and check the results. # Datasets Source: https://docs.latitude.so/datasets/overview Curate collections of inputs, outputs, and expected outputs from real traces to test and improve your agent. **Where this fits:** Datasets are part of **Refine**, after [Signals](../signals/overview). They turn real traces into reusable test cases for [regression testing](../test-and-fix/regression-testing). A **dataset** is a collection of rows you curate for testing and improving your agent. Each row holds an **input**, the agent's **output**, an optional **expected output**, and arbitrary **metadata**. Teams use them as golden datasets: stable, known-good test sets that a fix has to keep passing. The Datasets page listing golden datasets with name, description, and last updated ## What a dataset row contains | Column | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------- | | **Input** | The input your agent received, for example the user message. | | **Output** | What your agent actually returned. | | **Expected output** | The correct or desired answer, used to check the agent. Optional, see [Add expected output](./expected-output). | | **Metadata** | Arbitrary fields carried alongside the row. | Beyond these built-ins you can add your own [custom columns](./custom-columns), and rename, reorder, or remove any column. A dataset detail view showing rows with input, output, and expected output columns ## Create a dataset You can build a dataset three ways: Select traces from the trace list, search results, or a signal, and add them to a dataset. The most realistic test cases come straight from production. Open **Datasets** in your project, create a new dataset, then **Import** a CSV or **Add row** to enter cases by hand. Through the [MCP server](../getting-started/mcp), an agent like Claude or Cursor can create datasets and pull in the traces behind a signal for you. ## How datasets are used * **Regression testing**: replay a dataset's inputs against your agent and compare results to the expected outputs and your evaluations. See [Regression testing](../test-and-fix/regression-testing). * **Curating test sets**: collect representative traces from [Search](../search/overview) and [Signals](../signals/overview) into a stable, reusable set. * **Sharing with your harness**: export a dataset as CSV to drive tests in your own pipeline. ## Next step * [Add traces to a dataset](./add-traces): build a test set from real production traces. * [Custom columns](./custom-columns): add, rename, reorder, or remove columns. # Cluster Source: https://docs.latitude.so/deployment/cluster Deploy a scalable, highly-available Latitude on any Kubernetes cluster with our Helm chart. Cluster runs Latitude on **Kubernetes** through a cloud-agnostic [Helm chart](https://github.com/latitude-dev/latitude-llm/tree/development/charts/latitude): a Deployment per application service, a one-shot migrations job, ingress, and bundled infrastructure (Postgres, ClickHouse, Redis, Temporal, SeaweedFS) — every piece replaceable by a managed equivalent. It deploys the published [Docker images](https://hub.docker.com/u/latitudedata). For a simpler start on one machine, use the [Single-host](/deployment/single-host) deployment — same images, same configuration contract, no cluster required. ## Quick start The chart lives in the Latitude repository under `charts/latitude` — a sparse checkout downloads just that folder: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone --depth 1 --filter=blob:none --sparse https://github.com/latitude-dev/latitude-llm.git cd latitude-llm git sparse-checkout set charts/latitude cd charts/latitude ``` Don't name it `values.yaml` — that file already holds the chart's defaults. ```yaml my-values.yaml theme={"theme":{"light":"github-light","dark":"github-dark"}} config: webUrl: https://latitude.example.com apiUrl: https://api.latitude.example.com ingestUrl: https://ingest.latitude.example.com extraEnv: # Email transport — required for magic-link sign-in (choose one provider) - name: LAT_SMTP_HOST value: smtp.example.com - name: LAT_SMTP_PORT value: "587" - name: LAT_SMTP_USER value: apikey - name: LAT_SMTP_FROM value: noreply@example.com secrets: masterEncryptionKey: # openssl rand -hex 32 betterAuthSecret: # openssl rand -hex 32 # Optional secret env vars (email passwords, AI provider keys, OAuth # client secrets) — delivered through the chart-managed Secret. extra: LAT_SMTP_PASS: your-smtp-password postgres: auth: password: # openssl rand -hex 16 runtimePassword: # openssl rand -hex 16 clickhouse: auth: password: # openssl rand -hex 16 ingress: className: nginx # tls: [...] ``` The secrets are permanent for the installation — keep this file (or move the keys into a pre-created Secret and set `secrets.existingSecret`). Every future `helm upgrade` must receive the same values. The install brings up the bundled infrastructure, runs database migrations and the Temporal schema setup as one-shot jobs, and starts the five application services. App pods wait in `Init:0/1` until their dependencies accept connections, then start once, cleanly — the first install takes a few minutes (image pulls plus volume provisioning). ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} helm install latitude . \ --namespace latitude --create-namespace \ --values my-values.yaml --timeout 15m ``` All pods should reach `Running`/`Ready` and the `latitude-migrations`, `latitude-temporal-schema`, and `latitude-temporal-namespace` jobs should show `Complete`. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} kubectl get pods -n latitude kubectl get jobs -n latitude ``` Open the web UI (at your `config.webUrl`) and register. Latitude sends a magic link through the email transport you configured; click it to finish signing in and create your organization. ## What gets deployed | Workload | Kind | Purpose | | ----------------------- | ------------------------------- | --------------------------------------------------------------------------- | | `web`, `api`, `ingest` | Deployment + Service | UI, public API & MCP server, OTLP trace ingestion — exposed via the ingress | | `workers`, `workflows` | Deployment | BullMQ background jobs and Temporal workers — internal only | | `migrations` | Job (post-install, pre-upgrade) | One-shot Postgres + ClickHouse migrations | | `postgres` | StatefulSet + PVC | Primary store + pgvector | | `clickhouse` | StatefulSet + PVC | Span/telemetry OLAP store | | `redis`, `redis-bullmq` | Deployment / StatefulSet + PVC | Cache + durable queues | | `temporal` | Deployment + schema Jobs | Workflow engine, Postgres-backed (no Elasticsearch/Cassandra) | | `seaweedfs` | StatefulSet + PVC | S3-compatible object store (single container, same recipe as Single-host) | ## Secrets Optional non-secret settings (email transport, `LAT_AI_*` model selection, integrations) go in `config.extraEnv` as standard `EnvVar` entries; optional **secret** values (AI provider keys, email passwords, OAuth client secrets) go in `secrets.extra`, which merges them into the chart-managed Secret — never as `extraEnv` literals, which would sit readable in the pod spec. To manage secrets yourself (e.g. with an external secrets operator), create a Secret carrying the chart's expected keys and set `secrets.existingSecret` — the chart then renders no Secret of its own. The key list is in the [chart README](https://github.com/latitude-dev/latitude-llm/tree/development/charts/latitude#secrets). See the [secrets reference](/deployment/configuration#secrets) for what each application secret does. ## AI features The stack boots and core observability (ingest + trace viewing) works **without API keys**. AI-dependent features stay disabled until you add credentials for the providers you configure — keys in `secrets.extra`, `LAT_AI_*` provider/model selection in `config.extraEnv`: | Capability | Providers | Feature | | ---------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Generation | Amazon Bedrock (default), Anthropic, OpenAI, Google, or any OpenAI-compatible endpoint | Flaggers, evaluations, signal summarization, taxonomy naming, AI generation | | Embeddings | Voyage AI (default), OpenAI, Google, or any OpenAI-compatible endpoint | Semantic trace/signal search, search highlights, signal clustering | | Reranking | Voyage AI (default) or Amazon Bedrock | Signal-discovery candidate matching | Every provider and model is selectable per feature through environment variables — see the [AI configuration reference](/deployment/configuration#ai). ## Scaling The five application services are stateless and scale horizontally: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} helm upgrade latitude . --values my-values.yaml --set api.replicas=3 # or imperatively: kubectl scale deploy/latitude-api -n latitude --replicas=3 ``` Every service ships with resource **requests** (HPA-ready); add limits and `HorizontalPodAutoscaler`s to taste. The bundled stateful services stay single-instance by design — for HA move them to managed equivalents ([Bring your own infrastructure](#bring-your-own-infrastructure)). ## Backups Persistent state lives in the PVCs of `postgres`, `clickhouse`, `redis-bullmq`, and `seaweedfs`. Use your storage provider's volume snapshots, plus: * **Postgres** — `kubectl exec -n latitude latitude-postgres-0 -- pg_dump -U latitude latitude` (the source of truth for projects, users, and metadata). * **ClickHouse** — volume snapshots or [ClickHouse `BACKUP`](https://clickhouse.com/docs/en/operations/backup) (span/telemetry data). * **Object store** — volume snapshots (or rely on your managed S3's durability). ## Upgrading The chart version tracks the Latitude release, and the images default to the chart's `appVersion` — so the chart always deploys the release it shipped with. To upgrade, pull the new chart and run: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} helm upgrade latitude . --values my-values.yaml ``` To deviate from the chart's pinned release, set `image.tag` (`X.Y.Z` with no leading `v`, or `latest`). Migrations run as a `pre-upgrade` hook — the schema is upgraded before the new pods roll. Latitude migrations are backward-compatible, so the previous release keeps working during the rollout. ## Health and observability Every service exposes an HTTP health endpoint, wired as readiness and liveness probes (`/health`; the web app uses `/api/health`; `workers`/`workflows` listen on internal ports 9090/9091). OpenTelemetry export is available through the `LAT_OBSERVABILITY_*` variables via `config.extraEnv`. ## TLS and reverse proxy The chart's Ingress routes the three public hosts (taken from `config.{web,api,ingest}Url`) to the `web`, `api`, and `ingest` Services over plain HTTP inside the cluster — terminate TLS at your ingress controller. Reference your certificate Secrets in `ingress.tls`, or have cert-manager issue them via `ingress.annotations`: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}} ingress: className: nginx annotations: cert-manager.io/cluster-issuer: letsencrypt tls: - secretName: latitude-tls hosts: - latitude.example.com - api.latitude.example.com - ingest.latitude.example.com ``` Keep the `config.*Url` values on `https://` — they feed the `LAT_*_URL` and CORS/trusted-origin settings. ## Custom domain Latitude exposes three public surfaces — the **web** UI, the **API** (which also hosts the **MCP server**), and **ingest** (OTLP traces). You set them through `config.webUrl`, `config.apiUrl`, and `config.ingestUrl`; the [Ingress](#tls-and-reverse-proxy) derives its three hosts from those same values, and the chart wires the rest of the URL contract for you — `LAT_TRUSTED_ORIGINS` and `LAT_CORS_ALLOWED_ORIGINS` are derived from `config.webUrl`. So a standard three-host setup needs nothing beyond getting those three URLs right and pointing DNS at the Ingress: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}} config: webUrl: https://app.your-domain.com apiUrl: https://api.your-domain.com ingestUrl: https://ingest.your-domain.com ``` Each must be the real, externally-resolvable `https://` URL (scheme included): the chart feeds them into the in-cluster `LAT_*_URL`, CORS, and trusted-origin settings and routes the matching Ingress host to each Service. The `LAT_*_PORT` values stay cluster-internal — the Ingress maps your hostnames to them, so you leave them unchanged. Add the certificates as shown under [TLS and reverse proxy](#tls-and-reverse-proxy). If you serve the web UI from more than one origin (e.g. a vanity host alongside the canonical one), override the allowlists explicitly through `config.extraEnv`: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}} config: extraEnv: - name: LAT_TRUSTED_ORIGINS value: https://app.your-domain.com,https://latitude.your-domain.com - name: LAT_CORS_ALLOWED_ORIGINS value: https://app.your-domain.com,https://latitude.your-domain.com ``` **MCP clients** connect to `${apiUrl}/v1/mcp` but sign in via OAuth against the **web** origin, so both `config.apiUrl` and `config.webUrl` must be correct and externally reachable (Ingress + TLS). ## Bring your own infrastructure Every bundled dependency is an independent toggle: set `.enabled: false` and fill its `external:` block, and the chart wires the right `LAT_*` configuration at your existing or managed instance instead of deploying the bundle. ```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}} postgres: enabled: false external: databaseUrl: postgres://latitude_app:...@my-rds:5432/latitude adminDatabaseUrl: postgres://latitude:...@my-rds:5432/latitude temporal: enabled: false external: address: my-namespace.a1b2c.tmprl.cloud:7233 namespace: my-namespace.a1b2c apiKey: ... ``` | Dependency | Toggle | Notes | | -------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Postgres | `postgres.enabled` | Needs the `vector` (pgvector) extension and a schema-create admin role | | ClickHouse | `clickhouse.enabled` | Dedicated database; `external.migrationUrl` is host:port only (`clickhouse://host:9000`) | | Redis (cache) | `redis.enabled` | Latitude namespaces all keys under `latitude:`, so a shared Redis is safe | | Redis (queue) | `redisBullmq.enabled` | Same namespacing; needs `noeviction` | | Temporal | `temporal.enabled` | Temporal Cloud or the official [temporalio/helm-charts](https://github.com/temporalio/helm-charts) (SQL persistence, no Cassandra/Elasticsearch) for an HA cluster | | Object storage | `seaweedfs.enabled` | Any S3-compatible store; omit `endpoint`/`forcePathStyle` for AWS S3 | ## Troubleshooting | Symptom | Meaning and fix | | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Pod stuck in `Init:0/1` | It's waiting for a dependency — `kubectl logs -n latitude -c wait-for-deps` prints which `host:port` it's polling. Usually a bundled service still starting, or an unreachable `external:` endpoint. | | A hook job failed and `helm install` errored | Read the job logs — `kubectl logs -n latitude job/latitude-migrations` (or `job/latitude-temporal-schema`, `job/latitude-temporal-namespace`) — fix the cause and re-run the install; the jobs are idempotent. | | Pod stuck in `Pending` | Usually no default `StorageClass` or an unbound volume claim — check `kubectl get pvc -n latitude`. | | `ImagePullBackOff` | Check `image.registry` / `image.tag` and the node's network; the public images live at `docker.io/latitudedata/`. | ## Next steps The full Latitude configurable environment-variable reference for self-hosting. # Configuration Source: https://docs.latitude.so/deployment/configuration The full Latitude configurable environment-variable reference for self-hosting. Every application variable is prefixed with `LAT_*`. A few infrastructure variables (`POSTGRES_*`, `CLICKHOUSE_*`) are read directly by the bundled database containers. The committed [`.env.example`](https://github.com/latitude-dev/latitude-llm/blob/development/.env.example) is the canonical starting point — it documents both local development and production: copy it to `.env.production` and apply the values marked **`# For production`** on each line. Each variable is either **required** (the deployment won't boot or function without it) or **optional** — it has a safe default, or it unlocks a feature that stays disabled until you set it (SSO, AI, email). ## Secrets Latitude needs two secrets, **generated uniquely per deployment** — never reuse the example values: | Variable | Notes | | --------------------------- | ------------------------------------------------------------ | | `LAT_MASTER_ENCRYPTION_KEY` | Encrypts stored provider credentials and integration tokens. | | `LAT_BETTER_AUTH_SECRET` | Signs auth sessions and magic-link tokens. | Generate each with: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} openssl rand -hex 32 ``` Moreover, remember to change the default infrastructure passwords (`POSTGRES_PASSWORD_*`, `CLICKHOUSE_PASSWORD`...) too! ## Email A working email transport is required for anyone to log in. Local development captures mail in Mailpit; **production is not bundled with an email service** — configure exactly one provider. | Variable | Notes | | ---------------- | ----------------------------------------------------------- | | `LAT_MAILPIT_*` | Local-dev capture only (UI at `:8025`); not for production. | | `LAT_SMTP_*` | Any SMTP server (AWS SES, Postmark, SendGrid SMTP, …). | | `LAT_MAILGUN_*` | Mailgun API transport. | | `LAT_SENDGRID_*` | SendGrid API transport. | ## AI The stack boots and core observability (ingest + trace viewing) works **without API keys** — AI-dependent features stay disabled, or fail, until you add the relevant provider key: Every internal AI capability is configurable through `LAT_AI_*` environment variables. With nothing set, the defaults uses **Amazon Bedrock** for generation and **Voyage AI** for embeddings and reranking. ### Providers You have to set the credentials for whichever providers you end up using: | Variable | Provider | Available for | | ----------------- | ------------------------------- | ----------------------------------------- | | `LAT_AWS_*` | Amazon Bedrock `amazon-bedrock` | Generation (default), reranking | | `LAT_VOYAGE_*` | Voyage AI `voyage` | Embeddings (default), reranking (default) | | `LAT_ANTHROPIC_*` | Anthropic `anthropic` | Generation | | `LAT_OPENAI_*` | OpenAI `openai` | Generation, embeddings | | `LAT_GOOGLE_*` | Google `google` | Generation, embeddings | | `LAT_CUSTOM_AI_*` | OpenAI-compatible `custom` | Generation, embeddings | ### Generation Each feature resolves every setting through a three-tier fallback: 1. Per-feature variable `LAT_AI__` 2. Global variable `LAT_AI_GENERATION_` 3. Built-in default `minimax.minimax-m2.5` Available ``: * `PROVIDER` * `MODEL` * `REASONING` (`none`/`default`/`minimal`/`low`/`medium`/`high`/`xhigh`) * `TEMPERATURE` * `MAX_TOKENS` Always set `PROVIDER` and `MODEL` together — model ids are provider-specific. | Feature | What it powers | Built-in default | | ------------------------- | ----------------------------------------------------- | ------------------------------------------------ | | `ISSUE_DETAILS_GENERATOR` | Issue names/descriptions | `amazon-bedrock` / `minimax.minimax-m2.5` | | `FLAGGER_CLASSIFIER` | Trace classification and annotation | `amazon-bedrock` / `anthropic.claude-haiku-4-5` | | `FLAGGER_EXTRACTOR` | Instruction extraction | `amazon-bedrock` / `minimax.minimax-m2.5` | | `FLAGGER_ANNOTATOR` | Flagger annotation drafting | `amazon-bedrock` / `minimax.minimax-m2.5` | | `ANNOTATION_ENRICHER` | Annotation enrichment | `amazon-bedrock` / `minimax.minimax-m2.5` | | `EVALUATION_JUDGE` | LLM-as-judge evaluation runs | `amazon-bedrock` / `minimax.minimax-m2.5` | | `TAXONOMY_NAMING` | Taxonomy cluster naming | `amazon-bedrock` / `minimax.minimax-m2.5` | | `GEPA_PROPOSER` | Evaluation-optimization proposals | `amazon-bedrock` / `anthropic.claude-sonnet-4-6` | | `MOMENT_CLASSIFIER` | Conversation-intelligence moment candidate validation | `amazon-bedrock` / `minimax.minimax-m2.5` | ### Embeddings Embeddings are configured globally through `LAT_AI_EMBEDDING_PROVIDER` and `LAT_AI_EMBEDDING_MODEL`. It defaults to `voyage` / `voyage-4-large`, and powers semantic trace/signal search, search highlights, and signal clustering. Without a working embeddings provider, search falls back to lexical. The embedding model is a **one-time choice — pick it at install time and keep it**. Once an embedding model is used on a live deployment, changing it is a difficult task. Different models produce incompatible vector spaces and existing data is never re-embedded, so switching breaks semantic search, signal matching, and clustering. The model must also **emit 2048-dimensional vectors** — the dimension is fixed by the database schema and is not configurable. ### Reranking Reranking is configured globally through `LAT_AI_RERANKING_PROVIDER` and `LAT_AI_RERANKING_MODEL`. It defaults to `voyage` / `rerank-2.5`, and refines signal-discovery candidate matching. Reranking is optional — when unavailable, discovery falls back to embedding-order matching. ## SSO By default users sign in with email magic links. Add an OAuth provider to enable social sign-in: | Variable | Notes | | -------------------------- | -------------------------------------------------------------------------- | | `LAT_GOOGLE_*` | Enables Google OAuth sign-in. | | `LAT_GITHUB_*` | Enables GitHub OAuth sign-in. | | `LAT_ALLOWED_EMAIL_DOMAIN` | Restrict sign-up / sign-in to a single email domain (e.g. your company's). | ## Integrations Optional, and cleanly no-op when unset: | Variable | Notes | | ------------------ | --------------------------------------------------------------------------- | | `LAT_SLACK_*` | Slack notifications and interactivity. | | `LAT_GITHUB_APP_*` | GitHub App integration; see the [setup guide](/more/github) to register it. | | `LAT_IPINFO_TOKEN` | GeoIP enrichment for the sessions panel; uses the free tier when unset. | ## Infrastructure These wire Latitude to its datastores, depending on your chosen [deployment option](/deployment/overview) you can either point them to the bundled infrastructure services or bring your own infrastructure. ### Application | Variable | Notes | | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NODE_ENV` | Controls under what environment the code runs. Set to `production`. | | `LAT_WEB_URL`, `LAT_API_URL`, `LAT_INGEST_URL` | Public URLs where users and clients reach each service. | | `LAT_TRUSTED_ORIGINS`, `LAT_CORS_ALLOWED_ORIGINS` | Comma-separated list of origins allowed to call the API. | | `LAT_WEB_PORT`, `LAT_API_PORT`, `LAT_INGEST_PORT` | Host bind ports (default `3000` / `3001` / `3002`). | | `LAT_WORKERS_HEALTH_PORT`, `LAT_WORKFLOWS_HEALTH_PORT` | Health-check ports for the background workers (default `9090` / `9091`). | | `LAT_IMAGE_TAG` | Image tag the stack pulls (default `latest`; pin `X.Y.Z` in production). | | `LAT_OSS_TELEMETRY_ENABLED` | Anonymous OSS deployment heartbeat sent to Latitude's PostHog project (default `true` in production, `false` in development). Set `false` to opt out. | | `LAT_EXPORT_RATE_LIMIT_*`, `LAT_INGEST_TRACE_RATE_LIMIT_*` | Rate-limit tuning for exports and per-organization/API-key trace ingestion. | | `LAT_INGEST_TRACE_MAX_PAYLOAD_BYTES` | Maximum trace request body size in bytes (default `33554432`, or 32 MiB). Larger declared or streamed payloads receive `413`. | | `LAT_INGEST_TRACE_MAX_IN_FLIGHT_BYTES` | Per-process budget for trace payloads being read or processed (default `67108864`, or 64 MiB). This must be at least twice `LAT_INGEST_TRACE_MAX_PAYLOAD_BYTES` to cover chunked-body assembly. | | `LAT_INGEST_TRACE_MAX_CONCURRENT_PAYLOADS` | Maximum trace payloads being read or processed concurrently per ingest process (default `16`). | ### Postgres Primary store for projects, users, and metadata. Requires the `vector` (pgvector) extension. | Variable | Notes | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | | `LAT_DATABASE_URL` | Runtime connection (RLS-bound `latitude_app` user, `latitude` schema). | | `LAT_ADMIN_DATABASE_URL` | Superuser connection — migrations, seeds, cross-org queries. | | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` | Bundled-Postgres container credentials. Omit if bringing your own Postgres. | | `POSTGRES_RUNTIME_USER`, `POSTGRES_RUNTIME_PASSWORD` | RLS runtime user, created by `docker/init-db.sh`. Omit if bringing your own Postgres. | | `LAT_PG_POOL_MAX`, `LAT_PG_IDLE_TIMEOUT_MS`, `LAT_PG_CONNECT_TIMEOUT_MS` | Connection-pool tuning. | ### ClickHouse OLAP store for spans and telemetry, in a dedicated database. | Variable | Notes | | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `LAT_CLICKHOUSE_URL`, `LAT_CLICKHOUSE_USER`, `LAT_CLICKHOUSE_PASSWORD`, `LAT_CLICKHOUSE_DB` | Application connection + dedicated database. | | `LAT_CLICKHOUSE_MIGRATION_URL` | Native-protocol host and port (e.g. `clickhouse://clickhouse:9000`) for the migration tool (goose); credentials and database are taken from the other `LAT_CLICKHOUSE_*` variables. | | `LAT_CLICKHOUSE_CLUSTER_ENABLED` | Set `true` for a clustered ClickHouse; default `false`. | | `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_DB` | Bundled-ClickHouse container credentials. Omit if bringing your own ClickHouse. | ### Redis Two Redis roles — a cache and the BullMQ job queue (same server or separate). Latitude namespaces all keys under `latitude:`, so it can share an instance with other apps. | Variable | Notes | | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `LAT_REDIS_HOST`, `LAT_REDIS_PORT` | Cache connection. | | `LAT_BULLMQ_HOST`, `LAT_BULLMQ_PORT` | Queue connection. | | `LAT_REDIS_TLS`, `LAT_REDIS_CLUSTER` | Enable TLS / cluster mode for the cache. | | `LAT_BULLMQ_PASSWORD`, `LAT_BULLMQ_CLUSTER` | Queue auth / cluster mode. | | `LAT_BULL_BOARD_USERNAME`, `LAT_BULL_BOARD_PASSWORD` | Basic-auth for the optional queue dashboard, served at `/bull-board` on the workers health port. Only mounted while both are set; leave unset to disable it. | ### Temporal Workflow engine for durable background processes. | Variable | Notes | | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `LAT_TEMPORAL_ADDRESS`, `LAT_TEMPORAL_NAMESPACE`, `LAT_TEMPORAL_TASK_QUEUE` | Connection, namespace, and task queue. | | `LAT_TEMPORAL_API_KEY` | Only for Temporal Cloud. | | `LAT_TEMPORAL_MAX_CONCURRENT_ACTIVITY_TASKS` | Optional per-worker cap for concurrently executing Temporal activities. Lower this when CPU-heavy activities share one task queue. | ### Object storage Stores ingested payloads, datasets, and exports — shared by all services. Pick one of two drivers with `LAT_STORAGE_DRIVER`: **`fs`** writes to the local filesystem (a path or volume shared by every service), or **`s3`** points at any S3-compatible service — AWS S3, SeaweedFS, MinIO, or any other managed or self-hosted store. | Variable | Notes | | ------------------------------------------------------------------ | -------------------------------------------------------------------------- | | `LAT_STORAGE_DRIVER` | `fs` (local filesystem) or `s3` (any S3-compatible store). | | `LAT_STORAGE_FS_ROOT` | Absolute path for the `fs` driver (required when `LAT_STORAGE_DRIVER=fs`). | | `LAT_STORAGE_S3_BUCKET`, `LAT_STORAGE_S3_REGION` | Bucket and region (required when `LAT_STORAGE_DRIVER=s3`). | | `LAT_STORAGE_S3_ENDPOINT`, `LAT_STORAGE_S3_FORCE_PATH_STYLE` | Set for non-AWS S3 (SeaweedFS, MinIO); omit for AWS S3. | | `LAT_STORAGE_S3_ACCESS_KEY_ID`, `LAT_STORAGE_S3_SECRET_ACCESS_KEY` | Omit to use the AWS default credential chain (e.g. an IAM role). | # Overview Source: https://docs.latitude.so/deployment/overview Use Latitude as a fully-managed cloud service, or self-host it — from a single machine to a Kubernetes cluster. Latitude is [MIT-licensed](https://github.com/latitude-dev/latitude-llm/blob/development/LICENSE) and self-hostable at any scale, on fully open infrastructure. However, there are easier ways to start benefiting from Latitude's agent monitoring features in no time. ## Latitude Cloud (recommended) [**Latitude Cloud**](https://latitude.so) is the fully-managed version — the fastest way to start and the easiest to operate. Sign up, grab an API key, and start sending traces in minutes. There's nothing to install, scale, patch, or back up. Free to start — 20K credits/month, 30-day retention, and unlimited seats. Why most teams choose Cloud: * **Zero ops.** No servers, databases, queues, or object stores to run, patch, scale, or back up — we handle all of it. * **Every AI feature on by default.** Semantic search, flaggers, evaluations, and signal clustering work out of the box — no proprietary AI provider accounts or keys to manage. * **Always current.** New features and fixes the moment they ship, with zero-downtime upgrades and no migrations to run yourself. * **Scales with you.** Trace spikes, retention, and growth are handled automatically — no capacity planning. * **Enterprise-grade security & compliance.** SOC 2, ISO 27001, and GDPR, with managed backups and high availability. See [Security & Compliance](/security/data-protection). * **Support from the team that builds Latitude.** ## Self-host Prefer to run Latitude yourself? It's as easy as pulling the ready-to-go container images on [Docker Hub](https://hub.docker.com/u/latitudedata). Follow these guides to get you deploying Latitude in no time. [![Deploy on Railway](https://railway.com/button.svg)](https://railway.com/deploy/latitude?referralCode=GND3X9\&utm_medium=integration\&utm_source=template\&utm_campaign=generic) A production-grade instance on one machine with Docker Compose. A scalable, highly-available deployment on Kubernetes via a Helm chart. Latitude is a set of five application services (`web`, `api`, `ingest`, `workers`, `workflows`) backed by Postgres (+ pgvector), ClickHouse, Redis, Temporal, and an object store. Every self-host option deploys the **same images** with the same [configurable values](/deployment/configuration). # Single-host Source: https://docs.latitude.so/deployment/single-host Run a production-grade Latitude instance on a single machine with Docker Compose or Docker Swarm. Single-host runs the **entire Latitude product on one machine**: all five application services, a one-shot migrations job, and bundled infrastructure (Postgres, ClickHouse, Redis, Temporal, SeaweedFS). It uses the published [Docker images](https://hub.docker.com/u/latitudedata) and a single `.env.production`, all defined in [`docker-stack.yml`](https://github.com/latitude-dev/latitude-llm/blob/development/docker-stack.yml). For a scalable, highly-available deployment, deploy Latitude to a Kubernetes cluster using our Helm chart instead — see [Cluster](/deployment/cluster) deployment. ## Quick start Create a folder `mkdir -p latitude` and download all files into it: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} base=https://raw.githubusercontent.com/latitude-dev/latitude-llm/development curl -fsSL --create-dirs "$base/docker-stack.yml" -o docker-stack.yml curl -fsSL --create-dirs "$base/.env.example" -o .env.example curl -fsSL --create-dirs "$base/docker/init-db.sh" -o docker/init-db.sh curl -fsSL --create-dirs "$base/docker/seaweedfs/init.sh" -o docker/seaweedfs/init.sh curl -fsSL --create-dirs "$base/docker/clickhouse/storage.xml" -o docker/clickhouse/storage.xml ``` Give permissions to the initialization scripts for postgres and seaweedfs: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} chmod +x docker/init-db.sh docker/seaweedfs/init.sh ``` The `.env.example` file documents the neccessary production variables, create a copy of it and modify it to your needs. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} cp .env.example .env.production ``` Generate a unique value for each key and paste them into `.env.production`: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} echo "LAT_MASTER_ENCRYPTION_KEY=$(openssl rand -hex 32)" echo "LAT_BETTER_AUTH_SECRET=$(openssl rand -hex 32)" ``` Moreover, remember to change the default infrastructure passwords (`POSTGRES_PASSWORD_*`, `CLICKHOUSE_PASSWORD`...) too! Choose one of the email providers supported (Mailgun, Sendgrid, SMTP...) and configure it so login (and other) emails reach you. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} docker compose --env-file .env.production -f docker-stack.yml up -d ``` The `migrations` container runs first and the application services wait for it to finish. To pin a specific release instead of `latest`, set `LAT_IMAGE_TAG=X.Y.Z` in `.env.production` (image tags have no leading `v`). Open the web UI (at your `LAT_WEB_URL`) and register. Latitude sends a magic link through the email transport you configured; click it to finish signing in and create your organization. ## What gets deployed | Container | Image | Purpose | | ----------------------- | ------------------------------ | ----------------------------------------- | | `web` | `latitudedata/web` | The Latitude UI | | `api` | `latitudedata/api` | Public API & MCP server | | `ingest` | `latitudedata/ingest` | OTLP trace ingestion | | `workers` | `latitudedata/workers` | BullMQ background jobs | | `workflows` | `latitudedata/workflows` | Temporal workers | | `migrations` | `latitudedata/migrations` | One-shot Postgres & ClickHouse migrations | | `postgres` | `pgvector/pgvector` | Primary store + pgvector | | `clickhouse` | `clickhouse/clickhouse-server` | Span/telemetry OLAP store | | `redis`, `redis-bullmq` | `redis` | Cache + queues | | `temporal` | `temporalio/auto-setup` | Workflow engine (Postgres-backed) | | `seaweedfs` | `chrislusf/seaweedfs` | S3-compatible object store | ## Secrets All configuration, secrets included, lives in `.env.production`. Keep `.env.production` out of version control and back it up with your other operational secrets. See the [secrets reference](/deployment/configuration#secrets) for what each application secret does. ## AI features The stack boots and core observability (ingest + trace viewing) works **without API keys**. AI-dependent features stay disabled until you add credentials for the providers you configure: | Capability | Providers | Feature | | ---------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Generation | Amazon Bedrock (default), Anthropic, OpenAI, Google, or any OpenAI-compatible endpoint | Flaggers, evaluations, signal summarization, taxonomy naming, AI generation | | Embeddings | Voyage AI (default), OpenAI, Google, or any OpenAI-compatible endpoint | Semantic trace/signal search, search highlights, signal clustering | | Reranking | Voyage AI (default) or Amazon Bedrock | Signal-discovery candidate matching | Every provider and model is selectable per feature through environment variables — see the [AI configuration reference](/deployment/configuration#ai). ## Scaling `docker-stack.yml` also runs under **Docker Swarm** (`docker swarm init`, then `docker stack deploy -c docker-stack.yml latitude`), which is the better fit for scaling. The stateless application services — `web`, `api`, `ingest`, `workers`, `workflows` — scale horizontally: raise their `deploy.replicas` in the file (or `docker service scale latitude_api=3`), and Swarm load-balances across replicas, including across extra machines joined with `docker swarm join`. The bundled stateful services (Postgres, ClickHouse, Redis, Temporal, SeaweedFS) stay single-instance — for HA, point them at managed/external instances ([Bring your own infrastructure](#bring-your-own-infrastructure)) or move to the Cluster deployment. On plain Compose (one host), the services with no published ports — `workers` and `workflows` — can still scale directly: `docker compose -f docker-stack.yml up -d --scale workers=3`. ## Backups Persisted state lives in the `postgres_data`, `clickhouse_data`, `seaweedfs_data`, and `redis_bullmq_data` volumes. Back them up regularly: * **Postgres** — `docker compose -f docker-stack.yml exec postgres pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"` (the source of truth for projects, users, and metadata). * **ClickHouse** — back up `clickhouse_data` or use [ClickHouse `BACKUP`](https://clickhouse.com/docs/en/operations/backup) (span/telemetry data). * **Object store** — back up `seaweedfs_data` (or rely on your managed S3's durability). ## Upgrading Pin a specific `X.Y.Z` tag for Latitude's images in production so upgrades are deliberate. For easiness, you can bump `LAT_IMAGE_TAG` to a new release (or use :latest) in `.env.production`. Then pull the new images and restart the services. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} docker compose --env-file .env.production -f docker-stack.yml pull docker compose --env-file .env.production -f docker-stack.yml up -d ``` The `migrations` container runs automatically on every `up`, applying any new Postgres and ClickHouse migrations before the application services start. ## Health and observability Every service exposes an HTTP health endpoint, wired as the container healthchecks in `docker-stack.yml` (`/health`; the web app uses `/api/health`; `workers`/`workflows` listen on internal ports 9090/9091). Check them with `docker compose -f docker-stack.yml ps` — every service should report `healthy`. OpenTelemetry export is available through the `LAT_OBSERVABILITY_*` variables in `.env.production`. ## TLS and reverse proxy `docker-stack.yml` publishes `web` (3000), `api` (3001), and `ingest` (3002) on the host over plain HTTP. For anything internet-facing, put a TLS-terminating reverse proxy (Caddy, nginx, Traefik) in front, route your domain(s) to those ports, and set the `LAT_*_URL` / `*_ORIGINS` values to the public `https://` URLs. ## Custom domain Latitude exposes three public surfaces, each typically on its own hostname behind your [reverse proxy](#tls-and-reverse-proxy): the **web** UI, the **API** (which also hosts the **MCP server**), and **ingest** (OTLP traces). `.env.example` ships these pointing at `localhost`; for a real domain you must update the public URLs **and** the origin allowlists, so that browsers, the API, and MCP clients all agree on where each service lives. In `.env.production`: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Public URLs — what users, SDKs, and MCP clients connect to # (https, terminated by your reverse proxy) LAT_WEB_URL=https://app.your-domain.com LAT_API_URL=https://api.your-domain.com LAT_INGEST_URL=https://ingest.your-domain.com # Origin allowlists — must contain your web origin, or browser calls to the API # are rejected and sign-in fails. Comma-separate to allow several origins. LAT_TRUSTED_ORIGINS=https://app.your-domain.com LAT_CORS_ALLOWED_ORIGINS=https://app.your-domain.com ``` Then restart so the services pick up the new values: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} docker compose --env-file .env.production -f docker-stack.yml up -d ``` The hostnames are yours to pick — three subdomains (above), three separate domains, or one domain with path routing — as long as each `LAT_*_URL` exactly matches (scheme included) the URL your reverse proxy serves, and the proxy forwards each host to the matching service port. Leave the `LAT_*_PORT` values as their defaults (`3000` / `3001` / `3002`): they're the ports each service listens on internally, not part of the public hostname. **MCP clients** connect to `${LAT_API_URL}/v1/mcp` but sign in via OAuth against the **web** origin, so both `LAT_WEB_URL` and `LAT_API_URL` must be correct. ## Bring your own infrastructure Each infra service in `docker-stack.yml` is a clearly-marked, independently removable block. To use an existing or managed datastore, comment out its block and point the matching `LAT_*` value at your instance: | Dependency | Repoint | Notes | | -------------- | ---------------------------------------- | ---------------------------------------------------------------------- | | Postgres | `LAT_DATABASE_*`, `LAT_ADMIN_DATABASE_*` | Needs the `vector` (pgvector) extension and a schema-create admin role | | ClickHouse | `LAT_CLICKHOUSE_*` | Dedicated `LAT_CLICKHOUSE_DB` database | | Redis (cache) | `LAT_REDIS_*` | Latitude namespaces all keys under `latitude:` | | Redis (queue) | `LAT_BULLMQ_*` | Latitude namespaces all keys under `latitude:` | | Temporal | `LAT_TEMPORAL_*` | Pointing to Temporal Cloud is recommended | | Object storage | `LAT_STORAGE_*` | Point at any S3-compatible store or the local filesystem | If you remove a bundled service that another service lists under `depends_on` (`postgres`/`clickhouse` for `migrations`, `postgres` for `temporal`), delete that reference too, or Compose will refuse to start. ## Next steps The full Latitude configurable environment-variable reference for self-hosting. # Contributing Source: https://docs.latitude.so/development/contributing How to contribute to Latitude — set up locally, propose changes, and follow our pull-request conventions and Code of Conduct. Latitude is open source under the [MIT License](https://github.com/latitude-dev/latitude-llm/blob/development/LICENSE), and contributions are welcome — code, docs, bug reports, and feature ideas alike. The full contribution guide — making changes, reporting issues, the CLA. The standards we hold ourselves and our community to. ## Ways to contribute * Pick up a [good first issue](https://github.com/latitude-dev/latitude-llm/contribute) — curated, newcomer-friendly tasks. * Open and vote on [issues](https://github.com/latitude-dev/latitude-llm/issues). * Improve the [docs](https://docs.latitude.so). For typos, small docs fixes, and clearly-scoped bugs, just open a PR. **For new features or anything significant, open an issue first** so we can discuss the approach — undiscussed changes may be rejected. > If you like the project but don't have time to contribute code: star the repo, share Latitude with people who'd find it useful or mention it at meetups or in your project's README. ## Reporting issues Search [existing issues](https://github.com/latitude-dev/latitude-llm/issues) first. A good bug report has expected vs. actual behavior, exact repro steps, and your environment. Found a security vulnerability? Don't open a public issue — check out the [Security Policy](https://github.com/latitude-dev/latitude-llm/blob/development/SECURITY.md). ## Opening a pull request Fork and branch from `development` (our trunk). Split large changes — schema separate from logic, refactors first. Run `pnpm check`, `pnpm typecheck`, `pnpm knip`, and `pnpm test` before pushing. Use [Conventional Commits](https://www.conventionalcommits.org/) for commit and PR titles (e.g. `fix(traces): handle empty spans`). We squash-merge. Reference the issue with `Closes #123`. The first time you open a PR, a bot asks you to sign our Contributor License Agreement — a one-time step. PRs can't be merged until it's signed. > We're a small team, we read everything but may take a few days, longer for big changes. Stale or out-of-scope PRs may be closed, but you're welcome to reopen. ## Community We all hang out in our [Slack community](https://join.slack.com/t/trylatitude/shared_invite/zt-35wu2h9es-N419qlptPMhyOeIpj3vjzw) — a good place to ask questions and share what you're building. # Setup Source: https://docs.latitude.so/development/setup Run the full Latitude stack on your machine ready for development with hot reload and infrastructure in Docker. Infrastructure (Postgres, ClickHouse, Redis, Mailpit, Temporal) runs in Docker, while the five Latitude services run directly on your host for fast hot reload. Core observability (trace ingest and viewing) works out of the box. AI-dependent features light up as you add the documented provider keys, see [AI features](#ai-features). This guide is for running Latitude locally for development. To self-host a production instance, see the [Deployment guide](/deployment/single-host) instead. ## Prerequisites * **Node.js 25 and Python 3.13**. If you use [mise](https://mise.jdx.dev), `mise install` provisions both. * **pnpm 10**. Enable it with `corepack enable`. * **Docker**. Docker Desktop or any Docker Engine, used for the infrastructure containers. * **goose** — the ClickHouse migration tool. Install with `brew install goose` (or see [goose releases](https://github.com/pressly/goose/releases) on other platforms). * **uv** — the Python package manager for the evaluation optimizer runtime. Install with `brew install uv` (or see [the uv docs](https://docs.astral.sh/uv/getting-started/installation/)). ## Local setup ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pnpm install ``` This also configures the git pre-commit hooks via the `prepare` script. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} cd packages/platform/op-gepa/python uv venv && uv sync --all-extras --all-groups ``` Latitude includes a Python sidecar for the evaluation optimizer runtime. The `.env.example` file carries working local defaults and boots the whole stack API key-free. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} cp .env.example .env.development cp .env.example .env.test ``` Then, set `NODE_ENV=test` in `.env.test` (`.env.development` already defaults to `development`). And point `LAT_STORAGE_FS_ROOT` (in both environments) at an **absolute** path for the local object store (the default is `/tmp/latitude`, created automatically). ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pnpm build ``` Some platform packages must be compiled before the database migrations run. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} docker compose up -d ``` `docker-compose.yml` is for local **infrastructure-only**: Postgres (with `pgvector` extension), ClickHouse, two Redis instances (cache + BullMQ), Mailpit (local SMTP), and self-hosted Temporal instance and its UI. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pnpm migrate ``` Runs the Postgres migrations (Drizzle Kit) and the ClickHouse migrations (goose). ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pnpm seed ``` Optionally create a sample organization, with a default project, a handful of users, an API key, and realistic sample telemetry across Postgres and ClickHouse — enough to sign in and exercise traces, search, signals, and evaluations immediately. Pick whichever you prefer — all run with hot reload: ```bash tmuxinator theme={"theme":{"light":"github-light","dark":"github-dark"}} # Infra + all services + Postgres studio, each in its own pane. # Requires tmux and tmuxinator. Runs `docker compose up` for you, # so you can skip the standalone infra step above. pnpm tmux ``` ```bash turbo theme={"theme":{"light":"github-light","dark":"github-dark"}} # Runs every workspace `dev` task together. pnpm dev ``` ```bash manual theme={"theme":{"light":"github-light","dark":"github-dark"}} # Spin up each service separately. pnpm --filter @app/web dev pnpm --filter @app/api dev pnpm --filter @app/ingest dev pnpm --filter @app/workers dev pnpm --filter @app/workflows dev ``` Open [http://localhost:3000](http://localhost:3000), regsiter or enter a seeded user (`owner@acme.com` or `admin@acme.com`), and click the magic link Mailpit captures at [http://localhost:8025](http://localhost:8025). When dependencies, migrations, or generated artifacts drift, `pnpm reset` (clean → install → build → reset DBs) gives you a clean slate; `pnpm catchup` rebuilds and applies any new migrations without dropping data. ## Local services Once everything is running, these are the local endpoints you can access: | Service | URL | Notes | | -------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | Web UI | [http://localhost:3000](http://localhost:3000) | The Latitude app UI | | API | [http://localhost:3001](http://localhost:3001) | Public API; health at `/health` | | Ingest | [http://localhost:3002](http://localhost:3002) | OTLP trace ingest; health at `/health` | | Workers | [http://localhost:9090](http://localhost:9090) | BullMQ workers; health at `/health` | | Workflows | [http://localhost:9091](http://localhost:9091) | Temporal workers; health at `/health` | | Mailpit | [http://localhost:8025](http://localhost:8025) | Local inbox — magic links land here | | Temporal UI | [http://localhost:8233](http://localhost:8233) | See workflow executions | | Drizzle Studio | [https://local.drizzle.studio/?port=3003](https://local.drizzle.studio/?port=3003) | Postgres console — run `pnpm --filter @platform/db-postgres pg:studio` first | ## Contributing workflow The pre-commit hook runs `pnpm check`, `pnpm typecheck`, and `pnpm knip`. It's configured automatically on `pnpm install`; to (re)configure it in an existing clone run `pnpm prepare`. Before opening a pull requests, run the same checks CI runs: | Task | Command | | ----------------------- | ---------------- | | Format | `pnpm format` | | Lint | `pnpm check` | | Typecheck | `pnpm typecheck` | | Dead-code / unused deps | `pnpm knip` | | Tests | `pnpm test` | You can scope any task to a single package with `--filter`, and pass flags after `--`: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} pnpm --filter @app/api test # one package pnpm --filter @app/api test -- src/some-file.test.ts # one file pnpm --filter @app/api test -- -t "health endpoint" # by test name ``` Check out the full [Contributing guidelines](/development/contributing). ## AI features The stack boots and core observability (ingest + trace viewing) works **without API keys**, and the full test suite is guaranteed to pass with no keys set. However, there are AI-dependent features that you must add the relevant keys to your environment file to enable them: | Capability | Providers | Feature | | ---------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Generation | Amazon Bedrock (default), Anthropic, OpenAI, Google, or any OpenAI-compatible endpoint | Flaggers, evaluations, signal summarization, taxonomy naming, AI generation | | Embeddings | Voyage AI (default), OpenAI, Google, or any OpenAI-compatible endpoint | Semantic trace/signal search, search highlights, signal clustering | | Reranking | Voyage AI (default) or Amazon Bedrock | Signal-discovery candidate matching | Every provider and model is selectable per feature through environment variables — see the [AI configuration reference](/deployment/configuration#ai). ## What's next How to propose changes, our PR conventions, and the Code of Conduct. Point an application at your local instance and watch traces arrive. # Evaluation alignment Source: https://docs.latitude.so/evaluations/alignment Measure how well your automated evaluations agree with human judgment. # Evaluation alignment Alignment measures how closely an evaluation matches human judgment. It answers one question: can you trust this detector to represent the behavior your team cares about? ## Why alignment matters An evaluation is useful only when it agrees with the way your team reviews real traffic. Without alignment tracking: * You may not notice that an evaluation is too strict or too lenient. * You may miss drift as your agent, users, or product change. * You may keep scoring a behavior against outdated examples. Alignment helps Latitude keep generated evaluations calibrated over time. ## How alignment works Alignment is computed when an evaluation and a human annotation score the same trace. Latitude compares their verdicts and uses the result as feedback for the detector. The point is not to display a metric; it is to keep the evaluation close to the latest human-reviewed examples. ## Viewing alignment Each evaluation detail page shows alignment when enough human-reviewed traces are available. Use it to see whether the evaluation still matches reviewer expectations and where it may be drifting. ## Alignment and generated evaluations When Latitude generates an evaluation from a signal: 1. It collects examples from annotations, signal-linked scores, and trace context. 2. It builds a detector for the behavior. 3. The detector is compared against known examples. 4. The detector is attached to the signal. A detector can start from a small amount of evidence. As more annotations and scores arrive, Latitude has more to work with, and it can realign the detector as new annotations, flagger matches, evaluation results, and custom scores come in. This keeps the detector matched to the behavior as production traffic evolves. ## Manually defined detectors A detector you write when you [create a signal](../signals/create) works differently. It runs exactly as you defined it and is not automatically realigned to annotations. That is deliberate: it does what you specified, and nothing changes it behind your back. If a detector you defined turns out too strict or too lenient, edit it. Adjust the conditions, rewrite the judge criteria, change a threshold in a script, and preview the change against recent sessions before saving. See [Detection methods](./detection-methods). ## Improving alignment When a generated evaluation looks misaligned: 1. Review traces where the evaluation and human review disagree. 2. Add annotations with specific feedback. 3. Confirm the signal has representative examples of the behavior. 4. Let the new evidence improve the next realignment. This keeps automated scoring grounded in human judgment. ## Next steps * [Annotations](../annotations/overview): how human review produces alignment signal * [Flaggers](../annotations/flaggers): automatic annotators that contribute signal * [Detection methods](./detection-methods): the three ways to define a detector * [Signals](../signals/overview): how evaluation matches become tracked signals # Custom scripts Source: https://docs.latitude.so/evaluations/custom-scripts Write a JavaScript detector that reads a session and decides whether a behavior is present. A custom script is a detector you write by hand in JavaScript. It reads one session and returns a score saying how strongly a behavior is present. Reach for it when a [set of conditions](./detection-methods#set-of-conditions) is too rigid and an [LLM judge](./detection-methods#llm-as-judge) is too loose, or when you need logic that combines several checks. Conditions and LLM judges compile down to the same kind of script, so a custom script can express anything they can, plus whatever else you write. Most signals don't need a custom script. Start with conditions or a judge, and drop to a script when you hit their limits. ## The shape of a script Your code runs as the body of an async function, so you can use `await` at the top level. It must `return` a score built with `Score`, `Passed`, or `Failed`. ```js theme={"theme":{"light":"github-light","dark":"github-dark"}} // Match sessions where the assistant said it couldn't help. const messages = session.conversation const lastReply = messages.length ? messages[messages.length - 1].content : '' return lastReply.includes("I can't help") ? Passed() : Failed() ``` `Passed()` and `Failed()` are shorthand for a full match (`1`) and no match (`0`). Use `Score(value, feedback)` when you want a strength in between, or a note explaining the verdict. ## Returning a verdict A script returns a strength between `0` and `1`, not a pass or fail. Latitude compares that strength against a threshold (0.5 by default): at or above it, the behavior counts as present and the session joins the signal. | Helper | Returns | Use for | | --------------------------- | ------------------------------- | ------------------------------------------- | | `Score(value, feedback?)` | `value`, from 0 to 1 | A graded strength, optionally with a reason | | `Passed(value?, feedback?)` | `value` if given, otherwise `1` | A clear match | | `Failed(value?, feedback?)` | `value` if given, otherwise `0` | A clear non-match | `feedback` is optional free text. Latitude stores it on the score and uses it when it groups and displays matches, so a short reason pays off later. `passed = true` means the behavior is present, not that the session was good. A signal tracks a behavior, and a matching session is one that exhibits it. A signal for "made-up information" passes when the model hallucinated. ## The session object Every script receives one global, `session`: a read-only snapshot of the conversation being checked. It is frozen, so you can read it but not change it. | Field | Type | Description | | --------------------------------------- | ---------- | ------------------------------------------------------------- | | `id` | `string` | The session id | | `userId` | `string` | The end user's id, empty if none was sent | | `startTime`, `endTime` | `string` | ISO 8601 timestamps | | `duration` | `number` | Total wall-clock time, in nanoseconds | | `timeToFirstToken` | `number` | Time to the first token, in nanoseconds | | `traceCount`, `spanCount`, `errorCount` | `number` | Counts across the session | | `cost` | `object` | `{ input, output, total }`, in microcents | | `tokens` | `object` | `{ input, output, total, cacheRead, cacheCreate, reasoning }` | | `tags` | `string[]` | Tags on the session | | `metadata` | `object` | The string metadata your app sent | | `conversation` | `array` | The transcript, each entry `{ role, content }` | | `traces` | `array` | Per-trace breakdown (see below) | Numbers are in raw units: durations are nanoseconds, costs are microcents (one US cent is 1,000,000 microcents), and token counts are integers. Most scripts read `conversation`, `tags`, `metadata`, and the tool data rather than the cost and token fields. ### Messages `session.conversation` is the deduplicated transcript for the whole session. Each entry is `{ role, content }`. ```js theme={"theme":{"light":"github-light","dark":"github-dark"}} const userTurns = session.conversation.filter((m) => m.role === 'user').length return Score(userTurns > 5 ? 1 : 0, `${userTurns} user turns`) ``` Interpolating the array into a string renders it as `[role] content` lines, one per message, which is handy for prompts: ```js theme={"theme":{"light":"github-light","dark":"github-dark"}} const transcript = `${session.conversation}` // [user] where is my order? // [assistant] let me check that for you... ``` ### Traces and tools `session.traces` breaks the session down by trace. Each trace carries its own rollups plus the models, providers, finish reasons, and tool calls it used. | Field | Type | Description | | -------------------------------------- | ---------- | ------------------------------------------------------ | | `id`, `name`, `status` | `string` | Trace id, name, and status (`ok`, `error`, or `unset`) | | `errorCount`, `spanCount` | `number` | Counts within the trace | | `duration`, `timeToFirstToken` | `number` | Nanoseconds | | `cost`, `tokens` | `object` | Same shape as the session totals | | `models`, `providers`, `finishReasons` | `string[]` | What the trace used | | `tools` | `array` | Tool calls in the trace (see below) | Each tool call has this shape: | Field | Type | Description | | ----------------- | --------- | ---------------------------------------------------------------------- | | `name` | `string` | The tool's name | | `input`, `output` | `string` | The call's arguments and result, possibly truncated for large payloads | | `error` | `boolean` | `true` if the call failed | | `duration` | `number` | Nanoseconds | ```js theme={"theme":{"light":"github-light","dark":"github-dark"}} // Match sessions where any tool call failed. const failed = session.traces.some((t) => t.tools.some((tool) => tool.error)) return failed ? Passed() : Failed() ``` ## Built-in functions Alongside the score helpers, a few functions are always available, and two more appear only when your script calls them. ### semanticSimilarity(query) Returns the highest cosine similarity between `query` and any message in the session, from `0` to `1` (0 when the session has no messages). It is async. ```js theme={"theme":{"light":"github-light","dark":"github-dark"}} const score = await semanticSimilarity('the user is angry or frustrated') return score >= 0.55 ? Passed(score) : Failed(score) ``` Latitude reuses the message embeddings computed when the session was ingested and embeds only your query, so this stays cheap. As a starting point, about 0.4 is broad, 0.55 is balanced, and 0.7 is strict. A session that hasn't been embedded yet is scored once its embeddings are ready. ### llm(prompt, options) Sends a prompt to an LLM and returns a structured object matching `schema`. Latitude manages the model and the system prompt. `schema` is required and must be built with `z` (below). It is async. ```js theme={"theme":{"light":"github-light","dark":"github-dark"}} const result = await llm( `Did the assistant promise a refund in this conversation?\n${session.conversation}`, { schema: z.object({ promised: z.boolean(), quote: z.string() }) }, ) return result.promised ? Passed(1, result.quote) : Failed() ``` ### z A schema builder for shaping `llm()` output and `parse()` input. It mirrors a subset of Zod: `z.string()`, `z.number()`, `z.boolean()`, `z.literal()`, `z.enum()`, `z.array()`, `z.object()`, and `z.union()`, with `.optional()`, `.nullable()`, `.describe()`, `.min()`, `.max()`, and `.int()`. ### parse(value, schema) Validates a value against a `z` schema and returns it, or throws if it doesn't match. Useful for checking JSON your agent produced. ```js theme={"theme":{"light":"github-light","dark":"github-dark"}} const messages = session.conversation const output = messages.length ? messages[messages.length - 1].content : '' try { parse(JSON.parse(output), z.object({ answer: z.string(), confidence: z.number() })) return Failed(0, 'output matched the expected schema') } catch { return Passed(1, 'output was missing or malformed') } ``` Call `llm` and `semanticSimilarity` by name in your source. Latitude reads those names to allocate the right resources and to make the functions available, so building the call dynamically won't work. ### What isn't available The sandbox is deliberately small. There is no network access, no `fetch`, no timers, no `Date.now` or `Math.random`, and no Node or browser APIs. The only import allowed is `zod`. A script's verdict depends only on the session and any `llm()` call, which keeps results reproducible. ## Where scripts run and their limits Detectors run in the background, never inside your app's request path. When a session finishes, Latitude runs the matching detectors on it. A preview runs the same way, over recent sessions, on demand. Latitude picks a resource budget from what your script uses: | The script | Time budget | | ----------------------------------------------- | ----------------- | | Uses neither `llm()` nor `semanticSimilarity()` | about 1 second | | Calls `semanticSimilarity()` | about 15 seconds | | Calls `llm()` | about 120 seconds | Every script gets 64 MiB of memory. These limits are enforced automatically and are generous for typical detectors. Errors are handled at two points: * Invalid JavaScript is caught when you save, so a script that won't compile can't be stored. * A script that throws at runtime, or exceeds its time or memory budget, scores that session as `errored`. Errored sessions show up in the preview. A detector that errors often is flagged as unhealthy. LLM and embedding calls cost money, so use the sampling control in the [Scope step](../signals/create#scope) to check a slice of traffic rather than every session. ## Editing and detaching If you build a detector with conditions or an LLM judge, the Custom script tab shows the exact script Latitude compiled from your settings, read-only. Choose "Edit as custom script" to take it over by hand. This clears the conditions or criteria form and switches the detector to your script. It is a one-way move. Once the script is the source of truth, the settings form is gone. Switching between the conditions and judge tabs before you detach keeps both drafts, so detach only when you're ready to hand-write. ## Related pages * [Detection methods](./detection-methods): conditions and LLM judges, the other two ways to define a detector * [Create a signal](../signals/create): the full creation flow * [Scores](../scores/overview): what a detector produces when it matches * [Sessions](../observability/sessions): what a session is # Detection methods Source: https://docs.latitude.so/evaluations/detection-methods The three ways an evaluation decides whether a session matches a signal: a set of conditions, an LLM judge, or a custom script. When you create a signal, you define how Latitude decides whether a session belongs to it. That check is the signal's evaluation, and you can build it three ways. | Method | How it decides | Cost | Good for | | --------------------------------------- | ----------------------------------------------- | -------------------------------- | ------------------------------------------------------------- | | [Set of conditions](#set-of-conditions) | Deterministic checks on facts about the session | Free, instant | Concrete facts: a phrase, a failed tool, latency over a limit | | [LLM as judge](#llm-as-judge) | An LLM reads each session and decides | One LLM call per checked session | Fuzzy behavior: tone, frustration, made-up answers | | [Custom script](#custom-script) | JavaScript you write | Depends on the script | Logic the other two can't express | All three produce the same kind of detector, and a signal has one active detector at a time. You can switch methods while you build or edit a signal. ## Set of conditions A set of conditions checks concrete facts about a session. Each condition is a small deterministic test. Conditions are free and run instantly, so this is the method to start with. Add up to 10 conditions and choose how they combine: * **All**: every condition must hold (AND). * **Any**: at least one condition must hold (OR). The match selector appears once you have two or more conditions. A single condition just has to hold. ### Condition types | Condition | What it checks | Options | | ------------------- | ------------------------------------------------------------------------------ | ------------------------------------------ | | Text match | A message contains, doesn't contain, matches, or doesn't match text or a regex | Where; operator; value; case sensitive | | Semantic similarity | A message is semantically close to a query | Query; sensitivity, or a custom threshold | | Empty output | The assistant produced no output | None | | Output length | The assistant output's length compared to a value | Characters or words; operator; value | | JSON output | The assistant output is valid or invalid JSON | Valid or invalid | | Metric | A session or trace metric compared to a value | Metric; aggregation; operator; value | | Tool used | A specific tool was called | Tool name | | Tool failed | A tool call ended with an error status | Tool name (optional; empty means any tool) | | Tool call count | How many tool calls happened | Operator; value | | Error | The trace or session ended in an error state | None | | Finish reason | The model's finish reason, such as stop, length, or tool\_calls | Value | A few details are worth knowing: * **Where** (for text match) picks which messages to look at: the last assistant message, any assistant message, any user message, any tool message, or the whole conversation. * **Comparison operators** read as "greater than", "at least", "less than", and "at most". * **Metric** works in display units: duration in milliseconds, cost in dollars. The available metrics are duration, cost, total or input or output tokens, error count, trace count, and span count. Aggregation decides whether a metric is summed across the session or compared per trace. * **Tool failed** and **Error** look at the error status of a tool call or trace, not at the content of a tool's output. A tool that returns the word "error" in an otherwise normal result does not count as failed. * **Semantic similarity** compares messages to a query using the embeddings Latitude already computed for the session. It is the one condition that isn't instant, because it works on embeddings. ### Semantic similarity sensitivity The sensitivity presets map to a similarity threshold: | Preset | Threshold | | -------- | --------- | | Broad | 0.40 | | Balanced | 0.55 | | Strict | 0.70 | Broad matches loosely and catches more. Strict matches only close paraphrases. Start with Balanced and adjust from the preview, or set an exact threshold and operator under Advanced. ## LLM as judge An LLM judge reads each matching session and decides whether the behavior is present, with a short reason. You describe the behavior in plain language in the "A session matches when..." field: > the user got frustrated, repeating themselves, complaining, or giving up before getting a useful answer Use a judge for behavior that is hard to pin to a fixed rule: tone, frustration, whether the answer actually resolved the request, or whether the model made something up. Latitude writes the prompt and manages the model, so you only supply the criteria. Each check sends a session to an LLM, which costs money and takes longer than a condition. On high traffic, lower the sampling rate in the [Scope step](../signals/create#scope) so a judge checks a representative slice instead of every session. ## Custom script A custom script is JavaScript you write that reads a session and returns a score. It can express anything conditions and judges can, plus logic they can't, such as combining several checks or inspecting tool arguments. The [Custom scripts](./custom-scripts) page documents the full API: the `session` object, the built-in functions, and the limits scripts run under. ## Choosing a method * If the behavior is a concrete fact you can name, use a set of conditions. It is free, fast, and easy to reason about. * If the behavior is fuzzy or semantic, use an LLM judge. * If you need logic the builder can't express, write a custom script. You aren't locked in. Editing a signal lets you switch methods, and the Custom script tab always shows the script your conditions or criteria compiled to, so you can start simple and take over by hand later. ## Related pages * [Create a signal](../signals/create): where you pick a method and scope the detector * [Custom scripts](./custom-scripts): the full scripting reference * [Triggers](./triggers): scope and sampling for a detector * [Evaluations overview](./overview): how detectors fit the wider evaluation model # Evaluations overview Source: https://docs.latitude.so/evaluations/overview Understand how evaluations score traffic to monitor your agent's quality over time. # Evaluations overview An evaluation is an automated detector that scores sessions as they arrive. It watches for one behavior or quality criterion, runs on completed traffic, and produces a [score](../scores/overview) each time it checks a session. Those scores feed the same analytics, signal, and alignment workflows as annotations and flaggers. Every signal is backed by an evaluation. When a signal's evaluation matches a session, that session joins the signal. ## What an evaluation has * A name and description: the behavior being detected. * A detection method: how it decides whether a session matches. See [Detection methods](./detection-methods). * A trigger: which sessions it runs on, and at what sampling rate. See [Triggers](./triggers). ## How an evaluation runs 1. A session completes in your project. 2. Latitude checks it against each active evaluation's scope and sampling. 3. Matching evaluations score the session. 4. Each returns a pass or fail verdict with feedback, stored as a score. 5. A passing score adds the session to the evaluation's signal. `passed = true` means the behavior is present, not that the session was good. A signal for a bad behavior passes when that behavior happens. ## Where evaluations come from An evaluation can be created two ways. ### Generated from a signal When Latitude discovers a signal, or when you choose to monitor one, it can generate an evaluation from the signal's description, example traces, annotations, and scores. You don't pick the method. Latitude builds a detector from the evidence and keeps it aligned to human judgment over time. ### Defined by you When you [create a signal](../signals/create) yourself, you define its evaluation directly. You choose one of three [detection methods](./detection-methods): * Set of conditions: deterministic checks, free and instant. * LLM as judge: describe the behavior and let an LLM decide. * Custom script: JavaScript for anything the other two can't express. A detector you define runs exactly as written. It is not automatically realigned to annotations the way a generated one is. See [Alignment](./alignment). ## Choosing a detection method Clear structural failures, such as tool errors, empty responses, or latency over a limit, are a good fit for a set of conditions. Semantic behavior, such as relevance, tone, or whether an answer resolved the request, usually needs an LLM judge. When neither fits, a custom script gives you full control. See [Detection methods](./detection-methods) for the full catalog. ## Evaluation lifecycle * Active: scoring matching sessions in real time. * Paused: sampling set to `0`, configuration preserved. * Archived: read-only and no longer scoring new sessions. * Deleted: removed from management views, while historical results stay in analytics. ## Next steps * [Detection methods](./detection-methods): the three ways an evaluation decides * [Custom scripts](./custom-scripts): the scripting reference * [Triggers](./triggers): scope and sampling * [Alignment](./alignment): how evaluations stay calibrated to human judgment * [Signals](../signals/overview): how evaluation matches become tracked signals # Evaluation triggers Source: https://docs.latitude.so/evaluations/triggers Configure which sessions an evaluation runs on, and how many. # Evaluation triggers An evaluation's trigger decides which sessions it runs on and how many of them. It controls monitoring scope and cost without changing how the evaluation decides a match. ## Scope: which sessions to check By default an evaluation runs on every session in your project. Narrow it with filters. When you [create a signal](../signals/create), the Scope step offers these dimensions: * Tags * Services * Models * Providers * Metadata (any `metadata.*` key your app sends) With filters set, only matching sessions run through the evaluation, and everything else is skipped. An empty filter means every session. Scope uses the same shared filter system as trace views and [saved searches](../search/saved-searches), so a filter you build for search translates directly to an evaluation's scope. You can also open the builder pre-scoped from a search, using "Create signal from this search." ## Sampling: how many to check Sampling is the percentage of matching sessions the evaluation actually runs on, from 0 to 100. * It defaults to 10 percent for a new signal. * Setting it to 0 pauses the evaluation. The configuration is kept, but no sessions are checked. * A [set of conditions](./detection-methods#set-of-conditions) is free and instant, so 100 percent is usually fine. An LLM judge, or a script that calls an LLM, costs money and time per check, so a lower rate keeps costs down while still catching the pattern on a high-traffic project. ## Timing Latitude runs an evaluation as sessions complete, so it acts on finished work rather than partial executions. The exact turn it runs on, and any debouncing for multi-turn sessions, are handled for you. You set the scope and the sampling rate, and Latitude manages the rest. ## Scope, search, and annotations Scope and sampling control automated monitoring. [Search](../search/overview) and [annotations](../annotations/overview) cover human review: use search to inspect relevant sessions, then annotate the ones that need human judgment for alignment or discovery. [Flaggers](../annotations/flaggers) add automatic signal for a fixed list of common categories. ## Next steps * [Detection methods](./detection-methods): how an evaluation decides a match * [Alignment](./alignment): how human annotations calibrate evaluations * [Evaluations overview](./overview): how evaluations work * [Search](../search/overview): build cohorts of sessions to review # Experiments Source: https://docs.latitude.so/experiments/overview Compare two or more slices of your data side by side — every metric across sessions, users, tools, signals, behaviours and more. **Where this fits:** Experiments are part of the **Understand** section. A [search](../search/overview) answers "show me these sessions"; an experiment answers "how do these slices differ from each other?" An **Experiment** is a side-by-side dashboard. You define a **baseline** slice of your data and one or more comparison slices, and Latitude computes every metric it already tracks for each slice, then shows how each comparison moved relative to the baseline — greener when a change is good, redder when it's bad. ## Variants Each slice is a **Variant**: a saved population selector made of three things. * **Filters** — the same session filters you use on the dashboard (model, status, user, tool, metadata, duration, and so on). * **Search query** — an optional free-text or semantic query. * **Time range** — a preset (last day, last week, last month…) or a fixed calendar window. Presets stay live, so a "last 7 days" variant always covers the trailing week. ## Baseline You can mark one Variant as the **baseline**. Every other Variant's metrics are compared against it, allowing you to see easily how your new changes compare to previous versions of your agent. ## What gets compared For every Variant, an experiment computes the full set of analytics Latitude produces, grouped by type: * **Sessions** — count, distinct users, total and average cost and tokens, error rate, cache hit rate, and duration percentiles. * **Users** — distinct users and per-user rollups (sessions, traces, cost, duration, error rate). * **Tools** — calls, distinct tools, usage rate, error rate, and duration percentiles, plus the top tools. * **Signals** — distinct signals, occurrences, affected sessions/traces/users, and cost impact, plus the top signals. * **Behaviours** — observations, distinct clusters, and detected moments, plus the top behaviours. ## Population deviation A Variant whose population differs from the baseline by more than 25% (in session or user count) is flagged, since very different population sizes make the comparisons less reliable. ## Semantic queries A semantic search query returns a ranked sample rather than an exact set, so any Variant whose query has a semantic component is marked **approximate** and its metrics are best-effort. Filter-only and literal/phrase queries are exact. ## Creating an experiment From a project, open **Experiments** and create one with a name and optional description. A new experiment starts with two variants (a baseline and one comparison) so it's useful immediately; edit their filters, query, and time range, then read the comparison. "Import from search" seeds a Variant from a saved search's filters and query. # CLI Source: https://docs.latitude.so/getting-started/cli Install and use the Latitude CLI to manage your Latitude organization right from the terminal ## Overview The **`latitude` CLI** is a single, self-contained binary that exposes your Latitude organization on the command line. Like the [MCP server](/getting-started/mcp), its commands are **generated directly from the Latitude API**, so the command surface automatically stays in sync with the platform. For the live list of commands and their input/output schemas, check the [API reference](https://api.latitude.so/docs). It's built for two audiences: * **Humans**, a fast, scriptable way to inspect and manage projects, traces, datasets, members, keys, and more without leaving the terminal. * **Agents**, a zero-dependency way for an agent (Claude Code, Cursor, Codex, …) to act on Latitude without wiring up an MCP connection, using `--format json` for machine-readable output and `--schema` for machine-readable help. Prefer a network connection? The [MCP server](/getting-started/mcp) exposes the same capabilities over OAuth. ## Installation The CLI ships as a pre-built binary on our [GitHub Releases](https://github.com/latitude-dev/latitude-llm/releases) — grab the latest `cli-vX.Y.Z` release. Download the archive for your platform, extract it, and put the `latitude` binary somewhere on your `PATH`. | OS | Architecture | | ------- | ----------------------- | | Linux | x86-64 (`amd64`) | | Linux | ARM64 (`aarch64`) | | macOS | Intel (`amd64`) | | macOS | Apple Silicon (`arm64`) | | Windows | x86-64 (`amd64`) | ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Set and pick the asset for your OS/arch: latitude--.tar.gz curl -fsSL -o latitude.tar.gz \ https://github.com/latitude-dev/latitude-llm/releases/download/cli-/latitude-macos-arm64.tar.gz tar -xzf latitude.tar.gz chmod +x latitude sudo mv latitude /usr/local/bin/latitude # or any directory on your PATH latitude --version ``` On MacOS the first run may be blocked by Gatekeeper. Allow it with `xattr -d com.apple.quarantine /usr/local/bin/latitude`, or via **System Settings → Privacy & Security**. Download `latitude-windows-amd64.zip`, extract `latitude.exe`, and move it to a directory on your `PATH` (or add its folder to `PATH`). Then: ```powershell theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude --version ``` Then verify it works and, optionally, set up [shell completion](#shell-completion) and the [man page](#man-page). ## Authentication The CLI authenticates with an **organization-scoped API key**. Create one in the Latitude UI under **Settings → Keys → API Keys**. There are two ways to give the key to the CLI: Store the key once in your operating system's secret store (Keychain on MacOS, Secret Service on Linux, Credential Manager on Windows): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude auth login ``` You'll be prompted to paste your API key; it's saved to the keyring under `latitude:ApiKeyAuth` and reused on every subsequent command. Manage it with: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude auth status # show which credential sources are set, for each scheme latitude auth logout # remove the stored key from the keyring ``` Export the key in your shell — handy for CI or ephemeral environments: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} export LATITUDE_API_KEY="lat_..." latitude projects list ``` `LATITUDE_API_KEY` takes precedence over the keyring, so it's easy to override the stored credential per-session. `latitude auth status` shows every source the CLI can see and which one wins. ## Usage The general shape is `latitude [flags]`, where each resource mirrors an area of the Latitude API: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude --help # list every resource latitude projects --help # list the commands for a resource latitude projects list # run one latitude projects create --help # discover a command's flags/arguments ``` Resources include `projects`, `traces`, `datasets`, and more — run `latitude --help` for the full list. Common examples: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude account get # the current org + your role latitude projects list # every project in the org latitude projects create --name "My App" # create a project latitude traces list --help # inspect traces (project-scoped — see --help for the flags) latitude api-keys list ``` A few global flags worth knowing (run `latitude --help` for the full list): * `--query ` — project/filter the response before it's formatted, e.g. `--query "items[].slug"`. * `--dry-run` — validate the request locally without sending it to the API. * `--quiet` / `-q` — suppress success output (errors still print to stderr). * `--debug` — dump the raw HTTP request and response to stderr. ## Output formats Every command accepts `--format` to control how results are rendered. This makes the CLI equally good for humans reading a terminal and for scripts or agents parsing output. | Format | Description | | ------- | ------------------------------------------------------------------------------------------------------------------------- | | `table` | Human-readable table. **Default when stdout is a TTY.** | | `json` | Pretty JSON. **Default when the output is piped.** | | `jsonl` | Newline-delimited JSON (NDJSON) — one compact value per line; arrays are flattened. Ideal for streaming into other tools. | | `yaml` | YAML. | | `csv` | CSV — convenient for spreadsheets. | | `raw` | The unmodified server response bytes. | | `http` | The full HTTP response (status line + headers + body), like `curl -i`. | ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude projects list --format json latitude projects list --format csv > projects.csv latitude traces list --format jsonl | jq . # stream trace-by-trace # Combine with --query to reshape before formatting latitude projects list --query "items[].{slug: slug, name: name}" --format yaml ``` You can also set a default format for a session with the `LATITUDE_OUTPUT` environment variable (e.g. `export LATITUDE_OUTPUT=json`); the `--format` flag always overrides it. ## Environment variables | Variable | Purpose | | ----------------------- | -------------------------------------------------------------- | | `LATITUDE_API_KEY` | API key used to authenticate (alternative to the keyring). | | `LATITUDE_OUTPUT` | Default output format when `--format` is omitted. | | `LATITUDE_BASE_URL` | Override the API base URL (e.g. self-hosted or a mock server). | | `LATITUDE_TIMEOUT_SECS` | Total request timeout, in seconds. | | `LATITUDE_PROXY` | HTTP(S) proxy URL. | | `LATITUDE_CA_BUNDLE` | Path to a PEM file with extra trust roots. | | `LATITUDE_INSECURE=1` | Skip TLS verification (debugging only). | The standard `HTTPS_PROXY` / `HTTP_PROXY` / `NO_PROXY` / `SSL_CERT_FILE` variables are honored as well. ## Shell completion Generate a completion script for your shell — `bash`, `zsh`, `fish`, `powershell`, or `elvish` — and load it to get tab-completion for every resource, command, and flag: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude completion zsh > ~/.latitude-completion.zsh echo 'source ~/.latitude-completion.zsh' >> ~/.zshrc ``` ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude completion bash | sudo tee /etc/bash_completion.d/latitude > /dev/null ``` ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude completion fish > ~/.config/fish/completions/latitude.fish ``` ## Man page The CLI can emit its own manual page in roff format, so `man latitude` works like any native tool: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} latitude man > latitude.1 man ./latitude.1 # or install it system-wide: sudo mv latitude.1 /usr/local/share/man/man1/ && man latitude ``` ## For Agents The CLI pairs naturally with agents: `--format json`/`jsonl` for structured output, `--schema` for machine-readable help, and `latitude generate-skills` to teach an agent the command surface — everything it needs to drive Latitude locally with no MCP connection. Prefer the [MCP server](/getting-started/mcp) when you'd rather connect the agent over the network with OAuth. # Core Concepts Source: https://docs.latitude.so/getting-started/concepts Latitude's core concepts: projects, spans, traces, sessions, users, tools, search, behaviours, scores, signals, evaluations, and monitors. # Core Concepts Latitude is organized around a small set of concepts, grouped by the loop you work in: **Observe** what your agent did, **Understand** what matters, and **Refine** with monitoring and tests. ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Project ├─ Observe │ ├─ Sessions │ │ └─ Traces │ │ └─ Spans (LLM calls, tool calls, retrieval, and more) │ ├─ Users │ └─ Tools ├─ Understand │ ├─ Search and Behaviours over traces │ ├─ Scores attached to traces │ └─ Signals grouped from failed scores, or defined by you └─ Refine ├─ Evaluations that score live traffic └─ Monitors that alert on signals, searches, or raw traffic ``` ## Projects A **project** is the main workspace for one agent, application, or AI feature. Projects keep production data and reliability workflows scoped to the right product boundary. Project-scoped data includes: * spans, traces, and sessions * users and tools * search, saved searches, and behaviours * scores from annotations, evaluations, flaggers, and custom checks * signals grouped from failed scores * evaluations and monitors * project settings such as monitoring behaviour Projects belong to organizations. Organization membership, API keys, and settings control who can access a project and how defaults cascade. ## Spans A **span** is the smallest unit of captured work. It represents one operation inside an agent run, such as an LLM call, tool call, retrieval step, HTTP request, or custom instrumented operation. Spans can capture: * input and output content when available * start time, end time, and duration * success or error status * tags and metadata * provider, model, token usage, and cost for LLM calls Use spans when you need to understand exactly what happened inside a trace. ## Traces A **trace** is one complete interaction from start to finish, composed of one or more spans. For a chat agent, a trace commonly represents a single user turn and the agent work needed to respond. Traces are the main unit used across Latitude: * the trace detail view shows the full execution path * search returns matching traces * annotations, scores, and evaluations attach to traces * flaggers run on completed traces * failed scores on traces feed signal discovery Latitude waits for a trace to stop receiving new spans before treating it as complete. Downstream workflows such as search indexing, flaggers, evaluations, scores, and signal discovery run on completed traces so they do not act on partial executions. ## Sessions A **session** groups related traces into a multi-turn conversation or workflow. Traces always exist; sessions exist when your application sends a stable session id with telemetry. Use sessions when multiple traces belong to the same user conversation, support ticket, workflow, or agent task. For example, each chat turn can be one trace while the whole conversation shares one session. ## Users A **user** is an end user of your agent, identified by the `userId` you send with telemetry. The Users view groups everything one user did: their sessions, errors, cost, and activity over time, along with the signals affecting them. Use it to see who is impacted by a problem and who is most active. ## Tools A **tool** is a function your agent can call. Latitude discovers every tool from your telemetry and tracks per-tool usage, error rate, and latency in the Tools view, so you can see which tools are heavily used, failing, or never called. ## Search **Search** finds traces by meaning, exact text, metadata filters, or a combination of all three. It is the discovery layer for behaviours your team wants to inspect, review, and track. Saved searches let teams bookmark useful cohorts and review them repeatedly. Use search to find traces such as: * users showing frustration * failed payments or onboarding problems * tool loops and tool call errors * hallucinated or incomplete answers * traces from a specific user, model, release, environment, tag, or metadata value ## Behaviours A **behaviour** is a topic Latitude automatically discovers by clustering your sessions by meaning. Behaviours surface what your users are actually doing, with trends and outcome metrics, so you find patterns you did not know to search for. Where search starts from a query, behaviours start from a map of your traffic. ## Scores A **score** is Latitude's common verdict model for trace quality. Scores can come from human annotations, built-in flaggers, automated evaluations, or custom checks submitted through the API. Scores answer questions such as: * Did this trace pass or fail a quality check? * What feedback explains the verdict? * Which source produced the verdict? * Should this failure contribute to signal discovery? Failed, eligible scores are the primary input for signal discovery. Scores also power analytics and help teams compare automated judgments with human feedback. ## Signals A **signal** is a recurring failure pattern Latitude tracks. Instead of leaving failures as isolated examples, Latitude groups similar failed scores into named signals with example traces, trends, affected users, lifecycle states, and linked evaluations. You can also define a signal yourself when you already know the behavior to track. See [Create a signal](../signals/create). Signals created from negative human annotations are also called **issues**; the word is still used in that narrower context. Signals help teams: * triage new and escalating production problems * inspect representative traces for the failure pattern * decide whether a pattern is real, noise, resolved, or ignored * generate evaluations that monitor the same failure mode on future traces * catch regressions after a fix ships Signal states are **new**, **escalating**, and **ongoing**. A signal can have more than one at once, such as new and escalating. ## Evaluations An **evaluation** is an automated monitor that checks traces for a specific behaviour or quality criterion. Evaluations run on completed traces and produce scores, so their results feed the same analytics and signal discovery workflows as annotations, flaggers, and custom checks. Evaluations help teams: * monitor known failure modes on live traffic * measure whether a fix reduced or eliminated a behaviour * catch regressions after a signal is resolved * turn important signals into ongoing monitors * compare automated judgments with human annotations through alignment metrics Latitude can generate evaluations from signals, then use human feedback to keep those evaluations aligned with the team's judgment. ## Monitors A **monitor** watches a target, such as a signal, a saved search, a tool, or your raw traffic, and opens an **incident** when a condition is met. When an incident opens, Latitude notifies you in-app, by email, or in Slack. Every project comes with system monitors that watch the signal lifecycle, and you can create your own. ## How the concepts fit together 1. A **project** receives telemetry from your agent. 2. Agent operations arrive as **spans**. 3. Related spans form completed **traces**, and related traces group into **sessions**. 4. **Search** and **Behaviours** help you find the traces and topics that matter; **Users** and **Tools** break the same activity down by end user and by tool. 5. Annotations, flaggers, **evaluations**, and custom checks create **scores** on traces. 6. Failed scores cluster into **signals** you can triage, monitor, and fix, and you can also define a signal yourself. Signals from negative annotations are called issues. 7. **Monitors** watch signals (and searches, tools, or raw traffic) and open incidents that alert you when something needs attention. ## Next steps * [Start tracing](../telemetry/start-tracing): send your first spans and traces to Latitude. * [Observability](../observability/overview): inspect traces, sessions, users, and tools. * [Search](../search/overview) and [Behaviours](../behaviours/overview): find the traffic that matters. * [Signals](../signals/overview): understand how failed scores become tracked signals. * [Monitors](../monitors/overview): get alerted when something needs attention. # How to use Latitude Source: https://docs.latitude.so/getting-started/how-to-use-latitude The recommended workflow for finding, tracking, and fixing production failures in your AI agent. Latitude works best as a continuous loop for production agents: observe real traffic, understand what is going wrong, and refine your agent until it is fixed and stays fixed. The product is organized around that loop: **Observe, Understand, Refine.** ## The core workflow Send traces from your agent to Latitude. Each interaction becomes a trace of spans (LLM calls, tool calls, retrieval, and more), and multi-turn conversations group into [sessions](../observability/sessions). Send a `userId` and `sessionId` so you can also break activity down per [user](../observability/users) and review reliability, errors, and latency per [tool](../observability/tools). If you have not connected your app yet, follow [Start tracing](../telemetry/start-tracing). Use [Search](../search/overview) to find conversations by meaning, exact text, or metadata filters: frustrated users, tool loops, hallucinations, failed workflows, or anything specific to your product. [Behaviours](../behaviours/overview) goes further, automatically clustering your sessions into the topics users actually hit, so you discover patterns without writing a query. Open traces from search results, behaviours, or the trace list and leave [annotations](../annotations/overview). A thumbs-down with clear feedback tells Latitude this behaviour is worth tracking. [Flaggers](../annotations/flaggers) also annotate common failure categories automatically, such as frustration, refusal, jailbreaking, tool errors, and empty responses. Failed annotations, flagger matches, evaluation failures, and custom scores become [scores](../scores/overview). Latitude groups similar failures into named, prioritized [signals](../signals/overview), each with example traces, affected-user counts, trends, and a lifecycle. Signals created from negative annotations are called issues. Triage signals on the Signals page: set priority, inspect example traces, and resolve noise. [Monitors](../monitors/overview) watch a signal, a saved search, a tool, or your raw traffic and open an incident when something needs attention, notifying you in-app, by email, or in Slack. Generate [evaluations](../evaluations/overview) to keep scoring live traffic for the same failure. Fix the underlying behaviour in your code, prompts, tools, retrieval, or product flow. Turn the failing traces into a [dataset](../datasets/overview) and [regression test](../test-and-fix/regression-testing) the fix, using the [MCP server](./mcp) to drive it from your coding agent so the failure cannot quietly return. Resolve the signal once it is fixed, and the regressed monitor tells you if it comes back. Repeat the loop as new production traffic arrives. ## What to focus on first If you are setting up Latitude for the first time: 1. Connect tracing for one production agent. 2. Add `userId` and `sessionId` so traces group by user and conversation. 3. Search for one failure mode your team already cares about, or browse Behaviours to see what stands out. 4. Annotate representative traces with specific feedback. 5. Watch the Signals page for grouped patterns, and let the built-in monitors notify you. 6. Generate evaluations for the signals you want to score continuously. ## Work with agents and self-host * **MCP**: manage your workspace straight from Claude, Cursor, and other agents through the [MCP server](./mcp). * **Self-hosting**: run Latitude in your own infrastructure, from a [single host](../deployment/overview) to a full cluster. ## Why this works Latitude does not require you to define every possible failure upfront. You discover failures from real traffic, validate them with human review, and turn important patterns into automated monitoring. Over time, the system becomes a living map of what goes wrong in your agent and whether your fixes are working. # Latitude Overview Source: https://docs.latitude.so/getting-started/introduction Understand what Latitude is, why teams use its MIT-licensed open-source platform, and how Observe, Understand, and Refine fit together. # Latitude Overview Latitude is an **open-source, MIT-licensed** platform for improving production AI agents. It helps teams capture real agent traffic, understand what is going wrong, and turn repeated failures into tracked signals you can monitor and fix. The product is organized around the loop you work in every day: **Observe, Understand, Refine**.