Change log
Changes to the vendor table, the judge and the model set, each with the first edition scored under it.
| Instrument | Change | First edition under it |
|---|---|---|
| Judge | From the November 2026 Edition every category is read by a model trained for this task. Earlier editions keep the labels they were published with. | November 2026 Edition |
| The record | The pages, the standings and the method files are published under CC BY 4.0, free to cite. Until 4 October 2026 the September and October 2026 records were offered for download under CC BY 4.0; copies obtained then keep those terms. From that date no edition's raw labels or full responses are offered for download; they are sent on request, under the record license: for the research or reporting it was sent for, with commercial use, redistribution and model training by agreement. | October 2026 Edition |
| Two tiers | The public index runs on the standard tier from the September 2026 Edition: fourteen models from twelve labs and three buyer sizes. The flagship models run per category for subscribers and are never merged into the public standing. | September 2026 Edition |
| Categories | 69 categories added on 4 October 2026, from the categories G2 lists for this index's buyers that it did not ask. Their first edition, so no movement is read for them until the next. | October 2026 Edition |
| Vendor table v2026-11.3 | Freshdesk With Freddy AI, Freshdesk Freddy AI and Freshdesk Freddy fold into Freshdesk: the same help desk named with its built-in AI (David, 9 November 2026). | November 2026 Edition (re-scored) |
| Vendor table v2026-11.2 | One company, several names (docs/same-company-products-plan.md): 43 spellings of one product folded into its name, and 859 category readings so a company's other names for its offering in a category count for that product there. | November 2026 Edition (re-scored) |
| Vendor table v2026-11.1 | The November 2026 Edition's products in the vendor tables: alias_extend added the products named twice or more that no table name resolved (the per-edition routine, docs/entity-layer.md) | November 2026 Edition (re-scored) |
| Vendor table v2026-10.8 | Two companies whose one offering was counted under several names in a category added on 4 October 2026, folded in that category only (reviewed by hand, 5 October 2026): in master data management, Informatica's MDM under its platform, edition and Salesforce-era names reads as Informatica MDM (Informatica Cloud BDM, big-data integration, stays apart); in communications archiving, Proofpoint's archive under its Digital Communications Governance, Nuclei and archiving names reads as Proofpoint Enterprise Archive (Proofpoint Essentials Email Archive, the small-business product, stays apart). | November 2026 Edition (re-scored) |
| Vendor table v2026-10.7 | One company, several names (docs/same-company-products-plan.md): 22 spellings of one product folded into its name, and 1115 category readings so a company's other names for its offering in a category count for that product there. | November 2026 Edition (re-scored) |
| Vendor table v2026-10.6 | The products named in the categories added from the G2 gap (October 2026 Edition, added 4 October), named twice or more, entered from the judge's spellings. | November 2026 Edition (re-scored) |
| Vendor table v2026-10.5 | One company, several names (docs/same-company-products-plan.md): 10 spellings of one product folded into its name, and 564 category readings so a company's other names for its offering in a category count for that product there. | November 2026 Edition (re-scored) |
| Vendor table v2026-10.4 | Publishers excluded, the spellings the first pass missed: 3 names of sites that publish software roundups and rankings (config/source_classes.json, publisher) and sell nothing they rank (fitgap.com, gitnux.org, worldmetrics.org), found by sweeping every edition's raw labels for the listed publishers. | November 2026 Edition (re-scored) |
| Vendor table v2026-10.3 | Publishers excluded: 5 names of sites that publish software roundups and rankings (config/source_classes.json, publisher) and sell nothing they rank, read by the judge as products because an answer named its source (axis-intelligence.com, dupple.com, expertinsights.com, toolradar.com, zipdo.co). | November 2026 Edition (re-scored) |
| Vendor table v2026-10.2 | The October 2026 Edition's new names folded (13 spelling, suffix and domain variants of one product under its usual name). | November 2026 Edition (re-scored) |
| Vendor table v2026-10.1 | Products named in the October 2026 Edition run (fourteen models, three segments), two or more labels | November 2026 Edition (re-scored) |
| Vendor table vv2026-09.4 | One reading rule from the shared resolver, no table rows: a combined answer whose parts do not all resolve counts once, for its first part when that part resolves. | November 2026 Edition (re-scored) |
| Vendor table vv2026-09.3 | Two reading rules from the shared resolver, no table rows: a parenthetical anywhere in a name and a trailing edition or version number are read past; each part of a combined answer gets the tier and edition strip. | November 2026 Edition (re-scored) |
| Vendor table vv2026-09.2 | The five security categories whose first reading did not parse: attack surface, DLP, EDR, SD-WAN, threat intelligence | November 2026 Edition (re-scored) |
| Vendor table vv2026-09.1 | Spelling, prefix and tier variants folded and bare vendor names read per category, after a category-by-category reading of the seeded table | November 2026 Edition (re-scored) |
| Vendor table vv2026-09.0 | IT launch table, seeded from the September smoke and baseline runs | November 2026 Edition (re-scored) |
| Vendor table v2026-09-17.0 | Empty table at the start of the IT index. | November 2026 Edition (re-scored) |
Earlier judge checks
A sample of an edition's answers read again by other models under the same rubric. Same first choices is the share of answers where a reader names the same first choice or choices. The judge reading the answers again sets the ceiling.
September 22, 2026, 480 answers from the September 2026 Edition, 40 per model, re-read under the same rubric and schema. Reference: the stored labels from Claude Opus 5.
| Reader | Answers | Same first choices | Same labels | Kappa |
|---|---|---|---|---|
| Claude Opus 5.5 | 480 | 83% | 85% | 0.8 |
Opus 5.5 on its release day, 22 September 2026: the judge candidate at 40% less to run
September 16, 2026, 480 answers from the September 2026 Edition, 40 per model, re-read under the same rubric and schema. Reference: the stored labels from Claude Opus 5.
| Reader | Answers | Same first choices | Same labels | Kappa |
|---|---|---|---|---|
| Claude Opus 5 (the judge, again) | 480 | 93% | 93% | 0.908 |
| GPT-6 Astra | 480 | 81% | 76% | 0.669 |
| Gemini 3.1 Pro | 437 | 74% | 79% | 0.704 |
The first check on the standard tier: forty answers per model over the twelve models of the September 2026 Edition, read again by the judge and by two readers from other labs.
September 14, 2026, 240 answers from the September 2026 Edition, flagship answers, 40 per model, re-read under the same rubric and schema. Reference: the stored labels from Claude Opus 5.
| Reader | Answers | Same first choices | Same labels |
|---|---|---|---|
| Claude Opus 5 (the judge, again) | 240 | 93% | 92% |
| Claude Sonnet 5 | 240 | 72% | 78% |
| Claude Haiku 4.5 | 240 | 65% | 70% |
All three readers agreed on which products were named and on positive against negative; the cheaper models moved the line between first choice and alternative, which is the line the share is computed from. The judge stays Opus 5, and its own 7% first-choice flip on a repeat pass is part of the published noise floor.