Human-in-the-Loop (HITL), or Manual Evaluations, involve direct human review and assessment of prompt outputs. This method is essential for capturing nuanced judgments, user preferences, and criteria that are difficult for automated systems to evaluate.

​
How it Works

  1. Target Prompt Run: Your main prompt generates an output (log).
  2. Evaluation Setup: You define a HITL evaluation in Latitude, specifying what kind of feedback is needed (e.g., a numeric score, a boolean flag, a category label, free-form text).
  3. Manual Review: A team member (or designated reviewer) examines the prompt’s input and output within a specific log entry.
  4. Feedback Submission: The reviewer submits their judgment (score, label, text feedback) for that log through:
    • The Latitude Dashboard (Logs or Evaluations section).
    • Programmatically via the Latitude API or SDKs (useful for integrating feedback from end-users or custom review tools). You can see a code example here.
  5. Result Storage: The human-provided evaluation result is stored alongside the corresponding log.

Because HITL evaluations require manual input, they do not support automatic live or batch execution like LLM-as-Judge or Programmatic Rules. Feedback must be submitted individually for each log reviewed.

​
Use Cases

Manual evaluations are best for:

  • Nuanced Judgments: Assessing qualities like subtle tone issues, specific domain accuracy, or overall user satisfaction that automated methods might miss.
  • Capturing User Feedback: Integrating direct feedback (e.g., thumbs up/down, ratings) collected from end-users of your application.
  • Ground Truth Creation: Building a “golden dataset” of high-quality, human-verified examples that can be used to train or test automated evaluation methods (like LLM-as-Judge or fine-tuning models).
  • Safety & Compliance Reviews: Having humans verify adherence to critical safety or ethical guidelines.
  • Exploratory Analysis: Understanding failure modes or quality issues before setting up automated evaluations.

​
Trade-offs

  • Scalability: Cannot be fully automated; requires manual effort for each review.
  • Time & Cost: Can be slower and more resource-intensive than automated methods.
  • Subjectivity & Consistency: Judgments can vary between reviewers. Clear guidelines and reviewer calibration are important.

​
How to Perform Manual Reviews & Capture Feedback

​
1. Setting Up the HITL Evaluation

  • Navigate to your prompt’s “Evaluations” tab.
  • Click “New Evaluation”.
  • Choose “Human-in-the-Loop” as the type.
  • Configure the expected feedback type: Numeric (define range), Boolean, Text (define categories or allow free text).
  • Provide clear instructions or criteria for reviewers if needed (in the description field).

​
2. Reviewing Logs in the Dashboard

  • Go to the “Logs” section or the specific HITL evaluation’s dashboard.
  • Select a log entry to review.
  • Examine the prompt input, output, and context.
  • Find the section for the HITL evaluation.
  • Enter the score, select the label, or type the feedback according to the evaluation setup.
  • Submit the evaluation result for that log.

​
3. Capturing Feedback via API/SDK

  • Use the Latitude API or SDK endpoints/methods for submitting evaluation results.
  • Pass the log_id, the evaluation_id (for the specific HITL evaluation), and the evaluation result (score, boolean, text).
  • This allows you to build custom review interfaces or pipe in feedback collected directly from your application’s users.

Check how to annotate a log. A log is the result of running your prompt. So the person can annotate that result and tell if it was good or bad, or provide a score.

​
Viewing Evaluation Results

Manually submitted results appear alongside other evaluation results:

  • Logs View: Attached to the individual log entry.
  • Evaluations Tab: Aggregated statistics and distributions for the HITL evaluation.

​
Next Steps