Skip to content
Test Automation Vendors / 263 cases

Note for buyers

How this ranking grades a case: A, B, C and X with real examples

Research desk, testautomationvendors.com · published 2026-11-12 · data as of 2026-11-11 · 4 records

18
grade A
108
grade B
137
grade C
0
removed as X

The dataset holds A 18, B 108 and C 137 cases; 0 cases were removed as X. A needs a named client plus a metric with a stated baseline and method.

Counts come from verified records. A baseline is the before value a vendor states for a metric; quoted claims carry their grade and baseline flag.

baseline:
The measured value before the work started, against which a later value is compared. source

Examples per grade

deviqa-2 (DeviQA, Agorapulse): grade A. Named client and at least one metric with a stated baseline and a stated method or period.

a1qa-1 (a1qa, anonymous client): grade B. Anonymous client with a metric that states its baseline.

a1qa-7 (a1qa, Anaplan): grade B. Named client without a baselined metric.

a1qa-6 (a1qa, anonymous client): grade C. Anonymous client and no metric with a stated baseline.

Removed cases

No case was graded X in this run.

Questions

Who assigns the grade?

A verifier step that re-opens each source page and matches quotes; grades are computed from the flags, not set by hand.

Can a vendor ask for a regrade?

Yes, with a link to a source that states the missing field; regrades are logged in the changelog.

Do C cases count?

Yes, at weight 0.3 in case volume.

What makes a case X?

The page is gone, a metric contradicts another source, or the case describes development rather than testing.

Records used

This post reads 4 records; the newest verifiedOn date among them sets the data-as-of date.

Show record ids

deviqa-2, a1qa-1, a1qa-7, a1qa-6

More from the blog

Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.