Status: proposed evaluation method

This is a practical test design for evaluating assistant behavior. It is not a completed study, a customer result, a published academic paper or a claim of product superiority.

Fixture and task

Create two fictional agreements with known changes to dates, amounts and obligations, plus one deliberately unclear scan.

Procedure

  • Keep expected changes with the evaluator
  • Give the assistant only the source documents
  • Review each reported change against both versions

What to score

Score found changes, false changes, source references and correctly reported reading limits.

Limits and responsible interpretation

The fixture is an evaluation aid, not legal advice or a customer case. No product scores are published on this page.

Publish the date, exact product setup, source pack, sample size and failures with any future results. Keep expected answers with the evaluator. Do not combine unmatched API and consumer-app runs into one ranking.

Related reading