Where this fits: Experiments are part of the Understand section. A search answers “show me these sessions”; an experiment answers “how do these slices differ from each other?”
Variants
Each slice is a Variant: a saved population selector made of three things.- Filters: the same session filters you use on the dashboard (model, status, user, tool, metadata, duration, and so on).
- Search query: an optional free-text or semantic query.
- Time range: a preset (last day, last week, last month…) or a fixed calendar window. Presets stay live, so a “last 7 days” variant always covers the trailing week.
Baseline
You can mark one Variant as the baseline. Every other Variant’s metrics are compared against it, allowing you to see easily how your new changes compare to previous versions of your agent.What gets compared
For every Variant, an experiment computes the full set of analytics Latitude produces, grouped by type:- Sessions: count, distinct users, total and average cost and tokens, error rate, cache hit rate, and duration percentiles.
- Users: distinct users and per-user rollups (sessions, traces, cost, duration, error rate).
- Tools: calls, distinct tools, usage rate, error rate, and duration percentiles, plus the top tools.
- Signals: distinct signals, occurrences, affected sessions/traces/users, and cost impact, plus the top signals.
- Behaviors: observations, distinct clusters, and detected moments, plus the top behaviors.
Population deviation
A Variant whose population differs from the baseline by more than 25% (in session or user count) is flagged, since very different population sizes make the comparisons less reliable.Semantic queries
A semantic search query returns a ranked sample rather than an exact set, so any Variant whose query has a semantic component is marked approximate and its metrics are best-effort. Filter-only and literal/phrase queries are exact.Creating an experiment
From a project, open Experiments and create one with a name and optional description. A new experiment starts with two variants (a baseline and one comparison) so it’s useful immediately; edit their filters, query, and time range, then read the comparison. “Import from search” seeds a Variant from a saved search’s filters and query.Compare simulation runs
When you simulate an agent locally against a dataset, tag each run with a version (agentVersion in metadata, plus a simulation tag). Create an experiment whose baseline is simulation + agentVersion = v1 and whose comparison is simulation + agentVersion = v2. The experiment then compares those two slices on every metric Latitude already tracks, including signals and behaviors.