How Accurate Is the Intelligence?
ArcaneMetriX has what we call an “intelligence layer”. This is a deterministic, advanced oracle text parser that reads every card and turns it into two kinds of data you see all over the app: tags on a single card (Sol Ring is “Ramp”) and synergies between two cards in your deck (Grave Pact wants creatures to die and Viscera Seer sacrifices them). This page explains how we check those claims, what the current numbers are, and what they do and do not promise so you can decide for yourself how much weight to put on them.
Note
The numbers below are measured, not estimated. Every figure comes from a random sample that a person reviewed by hand against the actual cards, and the review ledgers are saved in the project’s repository so any single call can be reevaluated.
The current numbers
| Claim | How it was checked | Result |
|---|---|---|
| A tag is a true statement about its card | 2,000 random (card, tag) pairs reviewed by hand | 99.95 % correct (95 % interval 99.72 – 99.99 %) |
| A synergy is a real pairing and its description is true | 800 random synergies reviewed by hand, twice, on two consecutive builds | 99.50 % correct both times (pooled interval 99.02 – 99.75 %) |
| Every rules mechanic on the cards we analyze has a tag | All 175 mechanics present were counted, not sampled | 175 of 175 |
Some rows are marked defensible: the claim is true, but a stricter reader could argue with the description’s wording. For example, Skullclamp’s “Card Draw” tag is defensible because the draw needs a creature to die first. We count those in our favor in the headline, and we publish the strict figure (correct only) alongside it in the ledgers so nothing is hidden in the definition.
How a claim gets checked
The development team cannot check every claim the intelligence layer makes. The app makes about 34,000 tag claims and 68,000 synergy claims across just the the 186 preconstructed Commander decks that make up the measured population. So we try to do what a careful editor does: pull a random sample, read the cards, and mark each claim right or wrong. This is about as accurate as we can get for now, however, we are continuing to improve the system over time. If you see any tag or synergy inconsistencies, let us know!
Two rules keep this honest:
- Every synergy is asked two questions. Is the pairing real, given the full text of both cards (including the predicate restrictions like “nontoken”, “another”, “you control”, “with power 4 or greater”)? And is the sentence we show you actually true given the data? A correct pairing explained with a wrong sentence still counts as wrong.
- Every round uses a fresh sample with a new random seed. Re-checking the rows that prompted a fix would only prove that those rows were fixed. Drawing new rows is the only way to learn whether the rate of mistakes moved.
When an invalid row is found, we do not patch that one card. We find the shape of the mistake (say, “a card that only affects an opponent’s creatures was matched against a card in your own deck”) fix it for every card that shape touches, add a test so it cannot come back, and add a detector to the census (below) so we can count how many synergies that shape was affecting.
The road to 99.5 %
Synergy sentences are the harder claim, and they started a long way from where they are now.
The first measured sample was 86.5 % correct, and most of those errors were one template bug in how the sentence was worded. The middle of the chart shows a plateau around 93 – 95 % where fixes kept landing and the number would not move: each sample turned up different rare mistakes, so a sample could tell us what kinds of mistakes existed but not how many of each there were.
What broke the plateau was the census. Every mistake shape we had ever found was written down as an automatic detector and run over every synergy the engine produces, not a sample of them. That gave each shape an exact count, showed that the real problem was in shapes nobody had seen yet, and let us fix by size rather than by anecdote. The last stretch came from three rules that removed whole classes of error at once, most importantly: if the engine cannot verify a requirement from a card’s printed properties, it drops the synergy rather than guessing, and the sentence never states a condition the engine did not actually check.
Two consecutive fresh samples of 800 then landed exactly on 99.50 %, the target we set, and every wrong row in both was fixed before the work was closed.
At close, all original 36 mistake shapes ever found count zero across all 67,878 synergies. The 56 edges (0.08 %) still flagged belong to a few documented open shapes and none of them produced a wrong row in the last two samples.
Tags followed the same method and reached 99.95 % after nine rounds:
What these numbers do not promise
- They cover the cards we measured. The population is 186 preconstructed Commander decks which is only 6,288 distinct cards across every recent set. A card outside that set has not been sampled. We did run the engine over the full 21,952-card database to find cards it says nothing about: 60, of which 57 were lands whose mana ability appears only as reminder text (since fixed).
- Right is not the same as complete. These figures say the tags and synergies we show are true. They do not say we show every one that would be true. We plan to continue expanding coverage over various mechanics and strategies throughout Magic’s history, but turns out there are a LOT of synergies that develop over 30 years between +30,000 cards. If you see something we missed that you want to see worked on, let us know in our Discord.
- True is not the same as important. How strong a synergy is is a separate question and a separate, ongoing piece of work.
- The reviewer is fallible. Verdicts have been revised when a stricter reading showed one was wrong, always with the reason recorded in our ledgers. However, though we know a lot about Magic, we can occasionally be wrong so keep this in mind. Again, this is an ongoing process, so as we continue iterating we will push towards more synergy coverage with even higher accuracy. Your feedback helps!
If you find a wrong tag or synergy
You can remove or replace any tag on a card in your deck (see Card Tags), and every synergy explanation is written so you can check it against the cards yourself. When you find one that is wrong, tell us through Community & Feedback. A single reported card usually points at a shape of mistake, and fixing the shape is what moves these numbers.