Status: proposed evaluation method

This is a practical test design for evaluating assistant behavior. It is not a completed study, a customer result, a published academic paper or a claim of product superiority.

Fixture and task

Use a controlled public test page with a sequence of unchanged, matching and unavailable states. Request notifications only for a stated condition.

Procedure

  • Set the cadence and conditions explicitly
  • Record observations and deliveries separately
  • Include an unchanged repeat and an access failure

What to score

Score correct findings, duplicate alerts, missed observations and honest coverage limits.

Limits and responsible interpretation

A periodic watch is not continuous monitoring. This protocol has not been presented as a completed production study.

Publish the date, exact product setup, source pack, sample size and failures with any future results. Keep expected answers with the evaluator. Do not combine unmatched API and consumer-app runs into one ranking.

Related reading