Status: proposed evaluation method

This is a practical test design for evaluating assistant behavior. It is not a completed study, a customer result, a published academic paper or a claim of product superiority.

Fixture and task

Use synthetic source files and ask for a sourced memo plus a spreadsheet, with a fixed checklist of required fields.

Procedure

  • Keep the source pack and requirements identical
  • Record progress separately from completion
  • Inspect the actual deliverable and missing fields

What to score

A complete result must contain the requested artifacts and supported facts. A partial result is labelled and scored as partial.

Limits and responsible interpretation

Do not equate elapsed time or output length with quality. No cross-product performance results are claimed here.

Publish the date, exact product setup, source pack, sample size and failures with any future results. Keep expected answers with the evaluator. Do not combine unmatched API and consumer-app runs into one ranking.

Related reading