Heimdl
Heimdl and Greptile
Greptile says safe to merge, Heimdl checks the tests would notice if it were not
Two different questions
Greptile asks
“Does this change look right?”
Heimdl asks
“If this change were wrong, would a test fail?”
Across the 2 pull requests Greptile reviewed, the tests missed 84% of the changes
Greptile, #8160“Safe to merge.”
infisical/infisical #8160
HighAuthenticationLDAP auth check can be switched off, tests stay green71changes9caught11survived51uncovered1tests suggestedidentity-auth-template-service.ts:521survivedHighContractRequest version repair can be emptied, tests stay green63changes12caught4survived47uncovered1tests suggestedfixBrokenRequestVersion.ts:37survived
HighAuthenticationLDAP auth check can be switched off, tests stay green71changes9caught11survived51uncovered1tests suggestedidentity-auth-template-service.ts:521survived- fieldPatch && template.authMethod === IdentityAuthTemplateMethod.LDAP+ falseSuggested test, LDAP URL change without bindPass throws when fieldPatch is presentawait expect(patchTemplate(service, { url: 'ldap://new.example.com' })).rejects.toThrow(
"Changing an LDAP auth template's URL requires supplying bindPass. ..."
);hoppscotch/hoppscotch #6513HighContractRequest version repair can be emptied, tests stay green63changes12caught4survived47uncovered1tests suggestedfixBrokenRequestVersion.ts:37survived- "test-runner"+ ""Suggested test, fixBrokenRequestVersion should not mutate request field when type is test-runnerexpect(result[0].doc.request).toBe(null);Running both
Keep Greptile for review, add Heimdl for proof your tests hold the change
Other comparisons
Missed, the share of changes on the pull requests a reviewer reviewed that no test caught, survived plus uncovered
From Heimdl’s published audits of public pull requests, a reviewer is listed where it commented on the pull request