Status: proposed evaluation method

This is a practical test design for evaluating assistant behavior. It is not a completed study, a customer result, a published academic paper or a claim of product superiority.

Fixture and task

Use a fictional invoice sheet with duplicate IDs, a credit note, a missing amount and two currencies.

Procedure

  • Define the correct business rule first
  • Inspect formulas and exclusions
  • Record whether uncertainty is visible in the final answer

What to score

Score arithmetic, source traceability and handling of missing or mixed data separately.

Limits and responsible interpretation

A correct total from an incorrect method should not receive full credit. This is a proposed evaluation method with no measured results.

Publish the date, exact product setup, source pack, sample size and failures with any future results. Keep expected answers with the evaluator. Do not combine unmatched API and consumer-app runs into one ranking.

Related reading