Change log
Changes to the vendor table, the judge and the model set, each with the first edition scored under it.
| Instrument | Change | First edition under it |
|---|---|---|
| Judge | From the November 2026 Edition every category is read by a model trained for this task. Earlier editions keep the labels they were published with. | November 2026 Edition |
| The record | The pages, the standings and the method files are published under CC BY 4.0, free to cite. Until 4 October 2026 the September and October 2026 records were offered for download under CC BY 4.0; copies obtained then keep those terms. From that date no edition's raw labels or full responses are offered for download; they are sent on request, under the record license: for the research or reporting it was sent for, with commercial use, redistribution and model training by agreement. | October 2026 Edition |
| Two tiers | The public index runs on the standard tier from the September 2026 Edition: fourteen models from twelve labs and three buyer sizes. The flagship models run per category for subscribers and are never merged into the public standing. | September 2026 Edition |
| Categories | 39 categories added on 4 October 2026, from the categories G2 lists for this index's buyers that it did not ask. Their first edition, so no movement is read for them until the next. | October 2026 Edition |
| Vendor table v2026-11-09.1 | Freshdesk Freddy AI and Freshdesk Freddy fold into Freshdesk, as in the IT table (David, 9 November 2026). | November 2026 Edition (re-scored) |
| Vendor table v2026-11-05.2 | One company, several names (docs/same-company-products-plan.md): 22 spellings of one product folded into its name, and 412 category readings so a company's other names for its offering in a category count for that product there. | November 2026 Edition (re-scored) |
| Vendor table v2026-11-05.1 | The November 2026 Edition's products in the vendor tables: alias_extend added the products named twice or more that no table name resolved (the per-edition routine, docs/entity-layer.md) | November 2026 Edition (re-scored) |
| Vendor table v2026-11-04.3 | A phrase in Chinese (a certain paid advertising platform) named once by one model is not a product; it had no address and its page overwrote the vendor directory | November 2026 Edition (re-scored) |
| Vendor table v2026-11-04.2 | A phrase in Chinese (some social media management tools) named once by one model is not a product; it had no address and its page overwrote the vendor directory | November 2026 Edition (re-scored) |
| Vendor table v2026-11-04.1 | The Index's own former name excluded (David, 4 November 2026) | November 2026 Edition (re-scored) |
| Vendor table v2026-10-05.2 | One company, several names (docs/same-company-products-plan.md): 26 spellings of one product folded into its name, and 543 category readings so a company's other names for its offering in a category count for that product there. | November 2026 Edition (re-scored) |
| Vendor table v2026-10-05.1 | The products named in the categories added from the G2 gap (October 2026 Edition, added 4 October), named twice or more, entered from the judge's spellings. | November 2026 Edition (re-scored) |
| Vendor table v2026-10-03.3 | One company, several names (docs/same-company-products-plan.md): 20 spellings of one product folded into its name, and 426 category readings so a company's other names for its offering in a category count for that product there. | November 2026 Edition (re-scored) |
| Vendor table v2026-10-03.2 | Publishers excluded, the spellings the first pass missed: 3 names of sites that publish software roundups and rankings (config/source_classes.json, publisher) and sell nothing they rank (fitsmallbusiness.com, toolradar.com, zipdo.co), found by sweeping every edition's raw labels for the listed publishers. | November 2026 Edition (re-scored) |
| Vendor table v2026-10-03.1 | Publishers excluded: 12 names of sites that publish software roundups and rankings (config/source_classes.json, publisher) and sell nothing they rank, read by the judge as products because an answer named its source (business.com, dupple.com, f6s.com, fitgap.com, gitnux.org, influencermarketinghub.com, thecmo.com, thedigitalprojectmanager.com, wifitalents.com, worldmetrics.org, zipdo.co). | November 2026 Edition (re-scored) |
| Vendor table v2026-10-01.5 | The October 2026 Edition's new names folded: spelling, suffix and domain variants of one product under its usual name (53 folds: Coffee.ai, Prospeo.io, PostAffiliatePro, Loyalty Lion, Vena Solutions and the like), and category readings for the bare company names the run surfaced where the company has one product in that category (Oracle in six categories, Zoho in seven, Google, Adobe and LinkedIn in two each). | November 2026 Edition (re-scored) |
| Vendor table v2026-10-01.4 | Products named in the October 2026 Edition run (fourteen models, three segments, 73 public categories), two or more labels | November 2026 Edition (re-scored) |
| Vendor table v2026-10-01.3 | Residue from the 10-01.2 fold: Arize AI is the company Arize's own spelling (it had its own entry from the AI visibility tracking category), so it folds into Arize and takes Arize's category readings (Arize AX in AI agent observability and evaluation platforms, Arize Phoenix in conversational AI testing and QA platforms); Arize Phoenix + AX is removed so the combined answer counts once, for Arize Phoenix. | November 2026 Edition (re-scored) |
| Vendor table v2026-10-01.2 | The Cekura private run's products folded (c99 conversational AI testing and QA platforms, c100 AI agent observability and evaluation platforms): spellings and suffixes of one product under its usual name (Hamming AI, Bespoken AI, Galileo AI, Fiddler AI, Maxim AI, Rhesis AI, TestMu AI, Cekura.ai, Coval.ai), Botium and its box under Cyara Botium, Phoenix under Arize Phoenix, Opik under Comet Opik, W&B Weave's spellings under Weights & Biases Weave, Datadog's and Splunk's agent and LLM observability names under the product, Traceloop's OpenLLMetry under Traceloop, self-hosted and cloud Langfuse under Langfuse; combined answers removed so the first part counts; and category readings for the bare company names (Cyara, Arize, Comet, Weights & Biases, W&B, Datadog) where one product is meant. | November 2026 Edition (re-scored) |
| Vendor table v2026-10-01.1 | Products named in the private run for Gane's Cekura report (c99 conversational AI testing and QA platforms, c100 AI agent observability and evaluation platforms), two or more labels, so the bundle folds through the table | November 2026 Edition (re-scored) |
| Vendor table v2026-09-25.1 | One reading rule, no table rows: a combined answer whose parts do not all resolve counts once, for its first part when that part resolves (the reading room's pick the first, note the second, by rule). | November 2026 Edition (re-scored) |
| Vendor table v2026-09-24.5 | Decisions from the entity lane, the owner reading the first 40 spelling items of the September edition: 26 aliases, one new product (Adobe Experience Cloud), two duplicate entries folded (Zoom Webinar into Zoom Webinars, Salesforce Maps/Planning into Salesforce Maps), one combined spelling folded, and a reading for BigCommerce in ecommerce platforms. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-24.4 | Two reading rules, no table rows: a parenthetical anywhere in a name is read past (Google Tag Manager (GTM) 360 & Server-Side GTM had been counted raw), and so is a trailing edition or version number (Google Optimize 360, Kentico 13); each part of a combined answer gets the tier and edition strip. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-24.3 | Residue from the 24.2 fold of the private categories: the business-account variants of one product (Veezu, Cabubble, Arnold Clark), FlexNet Manager under Flexera One, Zuora Billing under Zuora, and readings for the bare names (ManageEngine, Stripe, Smartrak) in the categories where they mean one product. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-24.2 | The private categories' products folded: spellings, tiers, regional and business-programme variants of one product under its usual name, and category readings for the bare names (Thales, Snow, Enterprise, Nexus, Uber, Stripe) that mean one product in these categories. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-24.1 | Products named in the private runs for the licensing, monetization, usage-based billing and UK fleet categories (c74, c95 to c98), so the briefs fold through the table instead of a hand list | November 2026 Edition (re-scored) |
| Vendor table v2026-09-23.1 | Zentitle: the reversed slash form and the platform name join its aliases, and it is listed in all three Nalpeiron categories (licensing and entitlement management, software monetization and licensing platforms, and the new usage-based billing and metering) | November 2026 Edition (re-scored) |
| Vendor table v2026-09-16.3 | Software monetization and licensing: a bare Oracle here is not a candidate. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-16.2 | Software monetization and licensing, from the Nalpeiron one-off: the three spellings of Nalpeiron's Zentitle are one product, Revenera's FlexNet spellings are Revenera, and a bare Flexera in this category means its Revenera licensing line; Flexera One stays the asset-management product | November 2026 Edition (re-scored) |
| Vendor table v2026-09-16.1 | Software monetization and licensing: readings for the products the Nalpeiron one-off surfaced | November 2026 Edition (re-scored) |
| Vendor table v2026-09-15.2 | Fold generic-suffix spellings from the September standard run into their vendors; category readings for bare parent names where one product is meant | November 2026 Edition (re-scored) |
| Vendor table v2026-09-15.1 | Judge raw names from the September 2026 standard run (twelve models, three segments), groups of two or more labels | November 2026 Edition (re-scored) |
| Vendor table v2026-09-12.3 | The Neon named in conversation intelligence is Neon Mobile, the consumer app that paid users to record their calls and went offline in October 2025, which Claude named as a product to avoid in the negative variant. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-12.2 | Duplicate cleanup of the vendor table after the judge-driven extensions. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-12.1 | Category-scoped readings for the twelve bare parent names left in the v2026-09-10.1 review (Salesforce, Meta, LinkedIn, Oracle, SAP, Adobe, Zoho, Sinch, Freshworks, Google, Acquia, Thryv), plus the bare Twilio, Microsoft and HubSpot labels a second pass over the edition and calibration exports found: 57 readings added, each to the product the parent sells in that category, chosen from the evidence quotes. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-10.2 | Gane listed in email marketing as well as marketing automation. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-10.1 | Vendors named in the Gemini answers collected 2026-09-10 and in the redone Perplexity pairs | November 2026 Edition (re-scored) |
| Vendor table v2026-09-09.1 | Full-scope edition: the vendor table extended from the judge's raw names across the 59 categories added on 2026-09-09; one vendor per spelling group, existing vendors extended into new categories, bare parent names left for category-scoped readings. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-08.5 | Tier-suffix rule (a product tier counts as the product). | November 2026 Edition (re-scored) |
| Vendor table v2026-09-08.4 | Categories 13 (marketing automation and campaign management) and 14 (email marketing) added with a conflict disclosure: 49 vendors, category-scoped readings for HubSpot, Adobe, Oracle, Zoho, Microsoft, Salesforce, SAP, Twilio and Intuit, and existing vendors extended into the two categories. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-08.3 | status: discontinued on eleven products so a positive label on a dead product is reportable. | November 2026 Edition (re-scored) |
| Vendor table v2026-09-08.2 | Category-scoped aliases for bare parent and multi-product names (Salesforce, HubSpot, Adobe, Oracle, SAP, Semrush, Ahrefs, Salesloft, Outreach, Zoom, Clari, Gong). | November 2026 Edition (re-scored) |
| Vendor table v2026-09-08.1 | Seeded 38 CDP-centric vendors, then 438 vendors from the September 2026 Edition normalization backlog across all twelve categories. | November 2026 Edition (re-scored) |
Earlier judge checks
A sample of an edition's answers read again by other models under the same rubric. Same first choices is the share of answers where a reader names the same first choice or choices. The judge reading the answers again sets the ceiling.
September 22, 2026, 480 answers from the September 2026 Edition, 40 per model, re-read under the same rubric and schema. Reference: the stored labels from Claude Opus 5.
| Reader | Answers | Same first choices | Same labels | Kappa |
|---|---|---|---|---|
| Claude Opus 5.5 | 480 | 83% | 85% | 0.8 |
Opus 5.5 on its release day, 22 September 2026: the judge candidate at 40% less to run
September 16, 2026, 480 answers from the September 2026 Edition, 40 per model, re-read under the same rubric and schema. Reference: the stored labels from Claude Opus 5.
| Reader | Answers | Same first choices | Same labels | Kappa |
|---|---|---|---|---|
| Claude Opus 5 (the judge, again) | 480 | 93% | 93% | 0.908 |
| GPT-6 Astra | 480 | 81% | 76% | 0.669 |
| Gemini 3.1 Pro | 437 | 74% | 79% | 0.704 |
The first check on the standard tier: forty answers per model over the twelve models of the September 2026 Edition, read again by the judge and by two readers from other labs.
September 14, 2026, 240 answers from the September 2026 Edition, flagship answers, 40 per model, re-read under the same rubric and schema. Reference: the stored labels from Claude Opus 5.
| Reader | Answers | Same first choices | Same labels |
|---|---|---|---|
| Claude Opus 5 (the judge, again) | 240 | 93% | 92% |
| Claude Sonnet 5 | 240 | 72% | 78% |
| Claude Haiku 4.5 | 240 | 65% | 70% |
All three readers agreed on which products were named and on positive against negative; the cheaper models moved the line between first choice and alternative, which is the line the share is computed from. The judge stays Opus 5, and its own 7% first-choice flip on a repeat pass is part of the published noise floor.