JevMade Sign in
← Back to guides

JevMade field notes / Conceptual research preprint

Cheap AI checks do not replace judgment

Richard Hill asks whether cheaper machine evaluation creates more demand for checks while shifting the harder organizational decisions elsewhere.

Original by Richard HillEvaluation

Listen to this guide

JevMade’s plain-English explanation

0:00 /

Our summary

When checking something becomes cheaper, organizations may check more things rather than simply spend less. Richard Hill uses Jev to explore that possibility. His paper distinguishes applying a scoring rule from deciding which evidence matters, what the rule should measure and who has authority to act on the result.

Extra checks can share the same missing evidence or repeat the same mistake. A model cannot recover an important warning that disappeared from its input. Hill asks whether cheaper checks could move the harder work toward choosing rules, preserving evidence, understanding disagreement and deciding which exceptions need a person's attention.

The paper proposes questions for research, not measured Jev adoption or organizational savings. Demand would grow only if useful needs remain and other costs do not outweigh cheaper checking. Repeated checks do not automatically provide independent evidence, and a high score does not itself give permission for an important action.

Key takeaways

  1. Keep scoring against a rule separate from choosing the rule and authorizing what happens next.
  2. Count the cost of gathering evidence, reviewing exceptions and correcting errors, not just the price of a model request.
  3. Treat agreement cautiously when evaluators share evidence or errors; more answers need not mean more independent support.

Increased demand for checks is an untested possibility, not a measured result. Vendor figures are not independent scientific evidence. This explanation rewrites Richard Hill's preprint in our own words and is licensed under CC BY-SA 4.0, linked in Credits and license. Reuse must credit Hill and this adaptation, identify changes and retain the same license.

arXiv · Original published

Read the original guide Opens the author’s site in a new tab.

Keep this for later

Sign in to bookmark experiments, guides and videos, and keep notes only you can see.

Continue to sign in

We’ll bring you back to this listing.