Evidence Record

Evidence summary

NISTIR 7804 reports unassisted and assisted task effectiveness separately, treats an assist as task failure for validation, requires errors, failures, omissions, and difficulties to be recorded, and says time is relevant primarily when time is critical to the task.

Exact claim supported or contradicted

Supports a procedural-pilot outcome vector led by unassisted correct completion, with assistance and critical errors as separate guardrails and time as a conditional secondary measure.

Source provenance

Official NIST interagency report developed with a multidisciplinary group for electronic-health-record usability validation.

Relevant excerpt or data

The annotated validation table on pages 55–56 separates unassisted completion, assisted completion, time, errors/difficulties, and assists. It directs investigators to analyze failures and critical-error patterns rather than collapse them into a single score.

Interpretation

“Finished” must mean the predefined goal state was achieved without facilitator help and without a critical error. Assistance remains diagnostically useful but cannot be counted as equivalent success. Faster is better only when speed is part of the stated goal or when accuracy and safety are already satisfied.

Limitations

The report concerns safety-sensitive interactive health systems, so its critical-error emphasis may exceed the needs of a low-risk synthetic document task. It specifies reporting structure, not psychometric reliability for this repository’s future instrument.

Counterevidence

In exploratory formative work, an assist can allow continued observation and reveal later problems; coding it as failure should not end the session.

Reproduction or verification notes

Inspect NISTIR 7804, Table 2 and surrounding discussion on PDF pages 55–56.