Changelog

​

Numeric similarity evaluation metric

​

Realtime log updates toggle

​

Export all logs

​

Documentation sidebar

​

Per-subscription rate limits

​

Reworked examples documentation

​

Consolidated evaluations API

​

Performance and stability improvements

• Improving the performance of the logs section
• Improving the performance of the prompt editor
• Improving the performance of the project overview section
• Preparing the platform to scale for the future

​

Experiments

​

Evaluations rework

• Evaluations can now be accessed from the sidebar, whenever you select a prompt you’ll see the evaluations associated with it.
• You can now edit and test the prompt of LLM-as-Judge evaluations in a fully featured prompt playground
• Now you can set a minimum (or maximum) scoring threshold for an evaluation to pass
• Redesigned annotation experience for Human-in-the-Loop evaluations:
  • You can now manually evaluate logs for several evaluations inside the logs table with all the filtering options
  • You can now leave instructions to the annotators to help them make the right decision
• Improved prompt refiner and suggestions
  • Refiner now optimizes for a lower or higher evaluation score depending if the evaluation is negative
  • New smarter algorithm to select the best evaluation results to improve the prompt
  • You can now disable automatic suggestions from specific evaluations
• You can now pass custom instructions to guide the evaluation generator
• Evaluations are now versioned along with the project

1. Programmatic Rule: Evaluate responses using algorithmic metrics

• Exact Match: Checks if the response is exactly the same as the expected output
• Regular Expression: Checks if the response matches the regular expression
• Schema Validation: Checks if the response follows the schema
• Length Count: Checks if the response is of a certain length
• Lexical Overlap: Checks if the response contains the expected output
• Semantic Similarity: Checks if the response is semantically similar to the expected output

2. LLM-as-a-Judge: Evaluate responses using an LLM as a judge

• Binary: Judges whether the response meets the criteria
• Rating: Judges the response by rating it under a criteria
• [New] Comparison: Judges the response by comparing the criteria to the expected output
• Custom: Judges the response under a criteria using a custom prompt

3. Human-in-the-Loop: Evaluate responses using a human as a judge

• Binary: Judges whether the response meets the criteria. The resulting score is “passed” or “failed”
• Rating: Judges the response by rating it under a criteria. The resulting score is the rating

​

New Onboarding

• A fresh onboarding experience for newcomers
• PR #1161

​

Quality of live Improvements (part 2)

• Massively improved performance of editor in playground
• Improve performance of overview section
• Fixed charts displaying dates in timestamp format
• Added more details to error when synthetic dataset generation preview fails
• Added an “Improve my prompt” button to trigger a general suggestion improvement from Latitude’s Copilot
• Improved startup latency of MCP server integrations

​

Google SSO support

• Added support for Google single-sign-on (SSO)

​

Added several third-party MCP server integrations

• Added 20+ new integrations from the official list of third-party MCP servers
• PR 1150

​

Removed Telemetry and Traces section

​

New docs

​

Added new providers

• Added support for new providers: xAI, DeepkSeek, Amazon Bedrock, and Perplexity
• PR #1075

​

Quality of life improvements

• In sidebar when creating folder/prompt save it on blur #1044
• If theres only one folder in the project, display it open #1051
• When creating a prompt put focus in the editor #1052
• When open evaluations close parameters. And the other way #1055
•  Collapsed/Expanded parameters are broken #1000
• Move to the center prompt Add message/Reset in playground #1056
• Rename “Publish prompt” to “Share” in playground #1060
• Move toasts to the left side #1069
• PR #1078

​

Improved public Docker images

• Shaved several GBs off the public Docker images
• PR #1068

​

Other Updates

• Added several new MCPs to the list of supported integrations.
• Improved parameter mapping by using selected dataset columns as a fallback.
• Refactored jobs to remove meta programming, potentially fixing a memory leak.
• Introduced an impersonation feature for admin users.
• Addressed a build issue with heapdump in amd64 environments.
• Improved SDK’s error handling to address silent failures.

​

New evaluation types

• Exact Match: Checks if the response is exactly the same as the expected output. The resulting score is “matched” or “unmatched”.
• Regular Expression: Checks if the response matches the regular expression. The resulting score is “matched” or “unmatched”.
• Schema Validation: Checks if the response follows the schema. The resulting score is “valid” or “invalid”.
• Length Count: Checks if the response is of a certain length. The resulting score is the length of the response.
• Lexical Overlap: Checks if the response contains the expected output. The resulting score is the percentage of overlap.
• Semantic Similarity: Checks if the response is semantically similar to the expected output. The resulting score is the percentage of similarity.

​

Revamped dataset management

​

Webhooks

• commitPublished: Triggered when a commit is published in a project. The webhook payload includes details about the commit, such as the commit ID, message, and project information.

​

Other stuff

• We added support for new MCP servers:
  • Exa
  • Apify
  • Audiense
  • Redis
  • Jira
  • Attio
  • Supabase

​

Integrations

​

Triggers

• Time-based triggers: these triggers are triggered based on a specific time of day, for example, you can trigger your prompts at 9AM everyday.
• Email-based triggers: these triggers are triggered based on a specific email address, for example, you can trigger your prompts when a specific email address receives an email.

​

Subagents

​

Other stuff

• Drag and drop files/folders from the sidebar
• Improved editor error reporting
• Improved agentic behavior of agents running in Latitude
• Added option to stop streaming from UI and Typescript SDK
• General performance and stability improvements

​

Built-in tools

• Web search: if enabled, the model will be able to search the web for information.
• Extract web content: great when paired with the web search tool, this will allow you to extract the content of a website.
• Run code: if your prompt requires some code execution, you can now activate this tool and the model will be able to generate and run code in a sandboxed environment.

​

Playground improvements

​

Automatic refiner suggestions

​

Other stuff

• General improvements to stability and performance
• Updated list of available models
• Anthropic’s cache configuration now works as expected
• New endpoint to fetch all available prompts for a project
• We added Vertex AI as a new provider

​

Agents

• Some instructionsincluding a goal and some contextjust like any other prompt
• A set of tools that it can use to interact with the world
• A tool the agent can call to signal that it’s finished its work

• Adding support for using other agents as tools
• Supporting code execution as a tool
• Building a library of built-in tools ready to use (web browsing, search, etc.)
• Agent tracing
• Triggers

​

Full support for tool calling

​

Python SDK

• Running prompts through our gateway
• Pushing logs to Latitude
• Telemetry, automatic tracing of your LLM calls

​

Other stuff

• General improvements to stability and performance
• Added revert/reset commit actions to the Project’s History
• Added full support for OpenAI’s o1 model family
• The response of a step now is automatically parsed as JSON if the JSON output schema is defined.
• Lots of tool-related fixes

​

Latitude Telemetry

• Anthropic
• AWS Bedrock
• Azure OpenAI
• Cohere
• Google AI Platform
• OpenAI
• Vercel AI SDK
• Vertex AI

​

Version history

​

File / Image prompt parameters

​

Other stuff

• General improvements to stability and performance

​

Create prompts from documents

​

Log Filters

​

Other stuff

• We’ve increased our keep alive timeout setting to 10 minutes in order to match it with OpenAI’s default setting. This helps when you are trying to generate a big JSON response. That being said, we always recommend using streaming responses when you expect a large response, as it effectively means you have limitless timeout.
• Updated some old documentation
• Released a new homepage!
• General improvements to stability and performance

​

Public prompts

​

Export logs

​

Collapsible parameters

​

Overview page

​

Compile Latitude prompts locally

const { config, messages } = await latitude.prompts.render({
  prompt,
  parameters: {
    // Any parameters your document expects
  },
})

const response = await openai.chat.completions.create({
  ...config,
  messages,
})

​

Other improvements

• Added tool calls to LLM-as-judge evaluations’ context
• Fixed datasets containing JSON not working correctly in some scenarios
• General improvements in stability and performance

​

PromptL activated in Latitude

​

Refine prompt directly from evaluation logs

​

Evaluation results in Logs

​

Other improvements

• You can now edit a version title and description before publishing it
• You can now rename projects
• Several improvements in stability and performance

​

Human / Code evaluations

​

New prompt template syntax

<step as='researchPhase' provider='OpenAI' model='gpt-4'>
  <user>Research key points about {{ topic }} and create an outline.</user>
</step>

<step as='writing' isolated='true' temperature='0.7'>
    <user>
        Using this outline: {{ researchPhase.outline }}
        Write a detailed article.
    </user>
</step>

​

New parameters section for prompts

​

Prompt analytics

​

Default provider and models

​

More improvements

• You can now get and create prompts from the SDK/API docs
• You can now eject from simple LLM-as-judge evaluations into more complete advanced evaluations that give you complete control over the evaluation prompt
• Updated UI code snippets on how to push logs and evaluations to Latitude
• Several improvements in infrastructure stability and performance
• Several improvements and fixes to UI/UX

​

New evaluations playground

​

Latitude Cookbook

​

Anthropic cache

​

Rust SDK

​

Latitude Typescript SDK v1 released

​

Other improvements

• Dramatically improved performance of the prompt editor on large prompts
• Improved error reporting in the prompt editor
• Long-lived modals no longer close on click-outside or hitting ESC key
• Prompt input parameters are now stored in memory so that you can navigate to other sections and come back without losing the latest inputs you used in a specific prompt

​

Upload external logs

​

Trigger evaluations from SDK

​

JSON API

​

Other improvements

• Improved performance of prompt editor in large prompts
• Added code examples on how to use the SDK to the OSS repository
• Improved and fixed documentation in several places
• Several performance and stability improvements