Heimdl
Heimdl and cubic
cubic reviews the change, Heimdl checks the tests behind it
Two different questions
cubic asks
“Does this change look right?”
Heimdl asks
“If this change were wrong, would a test fail?”
Across the 2 pull requests cubic reviewed, the tests missed 58% of the changes
open-circle/valibot #1523
MediumUnknown keys dropped from validation results57changes38caught19survived0uncovered4tests suggestedlooseObject.ts:204survived
HighContractRequest version repair can be emptied, tests stay green63changes12caught4survived47uncovered1tests suggestedfixBrokenRequestVersion.ts:37survived
MediumUnknown keys dropped from validation results57changes38caught19survived0uncovered4tests suggestedlooseObject.ts:204survived- !dataset.issues || !config.abortEarly+ !dataset.issues && !config.abortEarlySuggested test, looseObject should include unknown keys when there are issues and abortEarly is falseexpect(result.value).toEqual({ key: 123, extra: 'bar' });hoppscotch/hoppscotch #6513HighContractRequest version repair can be emptied, tests stay green63changes12caught4survived47uncovered1tests suggestedfixBrokenRequestVersion.ts:37survived- "test-runner"+ ""Suggested test, fixBrokenRequestVersion should not mutate request field when type is test-runnerexpect(result[0].doc.request).toBe(null);Running both
Keep cubic for review, add Heimdl for proof your tests hold the change
Other comparisons
Missed, the share of changes on the pull requests a reviewer reviewed that no test caught, survived plus uncovered
From Heimdl’s published audits of public pull requests, a reviewer is listed where it commented on the pull request