Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

How Accurate Is the Intelligence?

ArcaneMetriX reads card text to produce tags describing a card’s roles and synergies explaining how two cards work together. We check whether those claims are true, including the exact sentence shown in the app.

Current results

Fresh post-fix samples reviewed on November 2, 2026, drawn uniformly from 186 preconstructed Commander decks covering about 6,300 distinct cards:

ClaimCorrect + defensible (95% interval)Correct only
Card tags — 2,000 random claims99.25% (98.8–99.5%)85.15%
Synergy pairings and their sentences — 800 random claims97.38% (96.0–98.3%)85.88%

A defensible row is accepted under an explicit modelling convention. For example, creature-token makers can count as support for death payoffs. The headline includes these rows; the correct-only column excludes them.

Current samples: correct, defensible and wrong claims

How we check

Reviewers read the complete card texts, including other faces and restrictions such as “nontoken” or “another.” A tag must match its published definition. A synergy must be a real pairing and have a true explanation: a useful interaction described incorrectly still fails.

Each measurement uses a fresh random seed. Independent AI reviewers check the claims, and flagged verdicts are adjudicated against the written grading rules. The sample, seed, verdicts and reasons are kept in the audit ledgers, including initial verdicts changed during adjudication. Automatic checks also look for known error shapes and missing mechanic coverage; their open findings remain recorded alongside the review.

Progress and grading

Our goal remains 99.5% for both headline measures in two consecutive fresh samples under the current grading. We have not reached it. Card tags are close: the last four fresh samples read 98.95–99.40%. Synergies have held at 97.4–97.9% over the same four samples. Most of the remaining synergy errors are single unusual card pairs, each different from the last sample’s, so fixing the ones a sample finds barely moves the next one.

The November 1–2 work had no release between its rounds, so the charts show only its latest post-fix measurement. Intermediate results remain in the audit records. The new stricter grading marker identifies the tighter checks on published definitions, qualifiers and exact sentences. Older measurements used earlier grading and are not directly comparable.

Card-tag accuracy over time

Synergy accuracy over time

What the numbers do not promise

  • Every card: the precision sample covers these preconstructed decks, not all of Magic.
  • Complete coverage or good rankings: a true tag can still be missing, and a real synergy can still be weak. Precision does not measure either problem.
  • Perfect grading: the reviewers are AI agents. Confidence intervals describe sampling uncertainty, not reviewer mistakes.

You can edit card tags and report questionable claims.