Status: proposed evaluation method

This is a practical test design for evaluating assistant behavior. It is not a completed study, a customer result, a published academic paper or a claim of product superiority.

Fixture and task

Start a synthetic supplier comparison, then change one budget limit while keeping the date and capacity fixed.

Procedure

  • Record when the correction was issued
  • Identify the existing task explicitly
  • Compare the final output with the updated brief

What to score

Score the changed requirement, preserved requirements and any duplicate action separately.

Limits and responsible interpretation

Timing and provider behavior can affect the outcome. This page describes a test design and contains no measured ranking.

Publish the date, exact product setup, source pack, sample size and failures with any future results. Keep expected answers with the evaluator. Do not combine unmatched API and consumer-app runs into one ranking.

Related reading