Status: proposed evaluation method

This is a practical test design for evaluating assistant behavior. It is not a completed study, a customer result, a published academic paper or a claim of product superiority.

Fixture and task

Use a fictional person with three preferences. Change one preference explicitly, then ask for an output that depends on all three.

Procedure

  • Keep the same fixture and task across products
  • Run fresh conversations and record the setup
  • Score the changed constraint and unchanged constraints separately

What to score

A successful correction uses the new detail and preserves the other requirements. A stale detail or an invented preference is a failure.

Limits and responsible interpretation

Do not report a universal memory ranking from this one scenario. This page is a proposed test method; no measured results are presented.

Publish the date, exact product setup, source pack, sample size and failures with any future results. Keep expected answers with the evaluator. Do not combine unmatched API and consumer-app runs into one ranking.

Related reading