Note for buyers
How to read a test automation case study: the fields that are usually missing
Research desk, testautomationvendors.com · published 2026-11-12 · data as of 2026-11-12 · 263 records
- 222 of 263
- cases omit start year
- 188
- omit a metric with a baseline
- 172
- omit the client name
Across 263 verified cases, the field missing most often is start year (222 cases), then a metric with a baseline (188) and the client name (172). Each case file on this site lists its missing fields under What this case does not tell you.
What stands out
- 01a1qa publishes 61 of the 263 verified cases; the three vendors with the most publish 153, so a reader is looking at a concentrated set of sources.
- 02209 of 639 published metrics state a before value, and 58 also state how or over what period they were measured.
- 0318 of 263 reach grade A, which needs a named client and one metric with both a before value and a stated method.
- 04The field left out most often is start year: 222 of 263 cases do not state it.
Counts come from verified records. A baseline is the before value a vendor states for a metric; quoted claims carry their grade and baseline flag.
- baseline:
- The measured value before the work started, against which a later value is compared. source
Missing fields, counted
Counts are cases where the field is not stated on the source page.
| Field | Missing in (of 263) |
|---|---|
| start year | 222 |
| a metric with a baseline | 188 |
| the client name | 172 |
| duration | 154 |
| a CI tool | 136 |
| team size | 114 |
Reading order
Start with the client name, then each metric's before value, then the measurement period, then team size and duration. A metric without a before value is a claim.
Case files that show this layout: deviqa-2 (DeviQA, grade A); deviqa-3 (DeviQA, grade A); deviqa-9 (DeviQA, grade A).
The evidence ladder
Each row counts the verified cases that make that disclosure; the rows are separate counts, ordered from the most basic disclosure to grade A. Of 263 verified cases, 18 reach grade A, and the rarest disclosure below it is "reach grade a" (18 cases).
The right-hand column gives the same steps for all 263 verified cases, so a reader can set this subset against the whole dataset.
Table view
| Item | Count |
|---|---|
| Name a listed framework | 263 |
| Name the client | 91 |
| Publish at least one metric | 210 |
| State a baseline for a metric | 75 |
| State baseline and method for one metric | 35 |
| Reach grade A | 18 |
| Total | 692 |
| Step | Verified cases | All verified cases |
|---|---|---|
| Name a listed framework | 263 of 263 | 263 of 263 |
| Name the client | 91 of 263 | 91 of 263 |
| Publish at least one metric | 210 of 263 | 210 of 263 |
| State a baseline for a metric | 75 of 263 | 75 of 263 |
| State baseline and method for one metric | 35 of 263 | 35 of 263 |
| Reach grade A | 18 of 263 | 18 of 263 |
Worked example: case deviqa-2
DeviQA published this case for Agorapulse. It reaches grade A, the top grade, and still leaves fields out; the table marks each field as stated or not disclosed.
- “Only 15% of tests were automated”before 15% after 90-95% DeviQA deviqa-2 grade A, verified baseline baseline yes method yes
- “90-95% of tests are automated”before Only 15% of tests were automated after 90-95% of tests DeviQA deviqa-2 grade A, verified baseline baseline yes method no
| Field | On the page |
|---|---|
| Client | stated: Agorapulse |
| Industry | stated: Social media management SaaS |
| Frameworks | stated: Playwright |
| Languages | stated: TypeScript |
| CI tools | stated: GitHub Actions |
| Team size | stated: 2 |
| Duration in months | not disclosed |
| Metrics with baseline and method | 1 of 2 |
Questions
Which field is missing most often?
start year, in 222 of 263 cases.
Is a case without metrics useless?
It still shows which frameworks a vendor has worked on and, if named, for whom.
Where are the missing fields listed?
On every case file, under What this case does not tell you.
Should I ask the vendor for missing fields?
The fields listed are the ones a vendor can supply on request; the site records only what is published.
Records used
This post reads 263 records; the newest verifiedOn date among them sets the data-as-of date.
Show record ids
a1qa-1, a1qa-2, a1qa-3, a1qa-4, a1qa-5, a1qa-6, a1qa-7, a1qa-8, a1qa-9, a1qa-10, a1qa-11, a1qa-12, a1qa-13, a1qa-14, a1qa-15, a1qa-16, a1qa-17, a1qa-18, a1qa-19, a1qa-20, a1qa-21, a1qa-22, a1qa-23, a1qa-24, a1qa-25, a1qa-26, a1qa-27, a1qa-28, a1qa-29, a1qa-30, a1qa-31, a1qa-32, a1qa-33, a1qa-34, a1qa-35, a1qa-36, a1qa-37, a1qa-38, a1qa-39, a1qa-40, a1qa-41, a1qa-42, a1qa-44, a1qa-60, a1qa-61, a1qa-62, a1qa-63, a1qa-64, a1qa-65, a1qa-66, a1qa-67, a1qa-68, a1qa-69, a1qa-70, a1qa-71, a1qa-72, a1qa-73, a1qa-74, a1qa-75, a1qa-76, a1qa-77, abstracta-1, abstracta-2, abstracta-3, abstracta-4, abstracta-5, abstracta-6, abstracta-7, betterqa-1, bugraptors-4, bugraptors-6, codoid-1, codoid-2, codoid-3, codoid-4, codoid-5, codoid-6, codoid-7, codoid-8, codoid-9, codoid-10, codoid-11, codoid-12, codoid-13, codoid-14, codoid-15, codoid-16, codoid-17, codoid-18, codoid-19, codoid-20, codoid-21, codoid-22, codoid-23, codoid-24, codoid-25, codoid-26, codoid-27, codoid-28, codoid-29, codoid-30, codoid-31, codoid-32, codoid-33, codoid-34, codoid-35, codoid-36, codoid-37, codoid-38, codoid-39, codoid-40, deviqa-1, deviqa-2, deviqa-3, deviqa-4, deviqa-5, deviqa-6, deviqa-7, deviqa-8, deviqa-9, deviqa-10, deviqa-11, deviqa-12, deviqa-13, deviqa-14, deviqa-15, deviqa-16, deviqa-17, deviqa-18, deviqa-19, deviqa-20, deviqa-21, deviqa-22, deviqa-23, deviqa-24, deviqa-25, deviqa-26, deviqa-27, deviqa-28, deviqa-29, deviqa-30, deviqa-31, deviqa-32, deviqa-33, deviqa-34, deviqa-35, deviqa-36, deviqa-37, deviqa-38, deviqa-39, deviqa-40, deviqa-41, deviqa-42, deviqa-43, deviqa-44, deviqa-45, deviqa-46, deviqa-47, deviqa-48, deviqa-49, deviqa-50, deviqa-51, deviqa-52, impactqa-1, impactqa-2, impactqa-3, impactqa-4, qa-madness-1, qa-madness-2, qa-wolf-1, qa-wolf-2, qawerk-1, qawerk-2, qawerk-3, qawerk-4, qawerk-5, qawerk-6, qawerk-7, qawerk-8, qualitest-1, qualitest-2, qualitest-3, qualitest-4, qualitest-5, qualitest-6, qualitest-7, qualitest-8, qualitest-9, qualitest-10, qualitest-11, qualitest-12, qualitylogic-1, qualitylogic-2, qualitylogic-3, testdevlab-1, testdevlab-2, testdevlab-3, testdevlab-4, testfort-1, testfort-2, testfort-3, testfort-4, testfort-5, testfort-6, testfort-7, testfort-8, testfort-9, testfort-10, testfort-11, testfort-12, testfort-13, testfort-14, testfort-15, testfort-16, testfort-17, testfort-18, testfort-19, testfort-20, testfort-21, testfort-22, testfort-23, testfort-24, testfort-25, testfort-26, testfort-27, testfort-28, testfort-29, testfort-30, testingxperts-1, testingxperts-2, testingxperts-3, testingxperts-4, testingxperts-5, testingxperts-6, testingxperts-7, testingxperts-8, testingxperts-9, testingxperts-10, testrig-1, testrig-3, testrig-6, testrig-7, testrig-8, testrig-9, testrig-10, testrig-11, thinksys-1, thinksys-2, thinksys-3, thinksys-4, thinksys-5, thinksys-6, thinksys-7, thinksys-8, thinksys-9, zuci-systems-1, zuci-systems-2, zuci-systems-3, zuci-systems-4, zuci-systems-5, zuci-systems-6, zuci-systems-7, zuci-systems-8
More from the blog
Note for buyers
Why a percentage without a baseline is a claim, not a result
246percentage claims
246 percentage claims in verified automation cases, and how many state the starting value.
data as of 2026-11-12 · 126 records
Note for buyers
What an automation coverage figure can mean: the denominators seen in published cases
88coverage claims
88 coverage claims sorted by what they measure: test cases, regression effort, features or code.
data as of 2026-11-12 · 76 records
Note for buyers
Questions to ask a vendor about a migration case before you rely on it
11migration cases
What 11 published migration cases disclose and leave out, turned into questions for the vendor.
data as of 2026-11-12 · 11 records
Related pages
Scores and grades describe published evidence as of the verifiedOn date on each item. They do not measure service quality. Verify with the vendor before contracting.