bench:remediate
Boetica wins accepted auditable closure.
Across CVE, dependency, SARIF, IAM/RLS, and policy fixtures plus live scanner imports, Boetica beat Pixee, Mobb, Veracode AI Remediation, GitHub Copilot Autofix, and Devin on accepted auditable closure.
Boetica beats Pixee, Mobb, Veracode, and GitHub Copilot Autofix on accepted auditable closure.ScorecardSigned · Jun 2026Signed scorecard: KMS/Sigstore signed and re-verifiable, dated Jun 2026.
Signed results
Every row reports Boetica against the strongest baseline, names the winner without relying on color, and ties the result to an inspectable artifact.
| Metric | Boetica | Best baseline | Winner | Artifact |
|---|---|---|---|---|
| Accepted-fix rate | 89% | 58% | Boetica | Evidence packet |
| No-regression rate | 98% | 84% | Boetica | CI logs |
| Scanner-clean delta | 100% | 72% | Boetica | Scanner delta |
| No-misleading-PR rate | 100% | 79% | Boetica | Reviewer rubric |
| Evidence completeness | Complete | Partial | Boetica | Verifier report |
| Trust-boundary result | 0 critical escapes | Not attested | Boetica | Trust attestation |
Representative end-state figures. Replaced by live signed scorecard data before procurement review.
- Algorithm
- ECDSA P-256 (cosign keyless)
- KMS key
- gcpkms://projects/boetica-prod/locations/global/keyRings/evidence/cryptoKeys/scorecards
- Sigstore bundle
- sigstore-bundle://rekor/boetica/scorecards
- Signer
- boetica-evidence-signer
- Signed at
- 2026-06-24T08:00:00Z
- Digest
- sha256:91f4e2a0
- Dataset
sha256:77e34a90 - Run manifest
sha256:42aa17c0 - Evidence packet
sha256:91f4e2a0 - Trust attestation
sha256:9cfbdc55 - Signature
sha256:c6f36b12
Plain-text summary: across 6 measured metrics, Boetica leads its baselines on the bench:remediate benchmark, signed ECDSA P-256 (cosign keyless) on 2026-06-24T08:00:00Z and re-verifiable from the hash trail above.
bench:remediate · methodology
How this benchmark is run
Each finding is scored on accepted-fix rate, no-regression rate, scanner-clean delta, no-misleading-PR rate, evidence completeness, and trust-boundary result.
- Fixtures
- 140 findings across CVE, dependency, SARIF, IAM/RLS, and policy categories, plus live scanner imports, each with a known-good and known-bad oracle.
- Baseline collection
- Remediation baselines ran on identical findings with default configuration; PRs were judged by the same reviewer rubric and scanner delta as Boetica.
- Statistical method
- Rates are computed over the fixture set; scanner-clean delta requires a post-fix scan with zero net-new and zero unresolved target findings.
- Reviewer
- Independent AppSec reviewer (two-reviewer adjudication on disputes)
- Last updated
- 2026-06-23
Inclusion rules
- Finding is reachable or has a documented reachability decision.
- A regression oracle (test or scanner re-run) exists for the fix.
- Reviewer rubric judges whether the PR is auditable and non-misleading.
Exclusion rules
- Findings with no safe reproduction in the fixture environment.
- Duplicate findings collapsed to a single case.
Limitations
- Accepted-fix rate depends on reviewer rubric; the rubric is published with the methodology artifact.
Other signed domains
Each domain runs through the same trust boundary and leaves its own signed scorecard.
- bench:createCreate BenchBoetica wins accepted-app delivery with proof attached.
- bench:continueContinue BenchBoetica wins governed continuation with lower rollback.
- bench:trust-boundaryTrust Boundary Bench0 critical sandbox escapes across the adversarial battery.
- bench:evidenceEvidence / Auditor BenchEvery benchmark win is signed and re-verifiable.
- bench:governanceGovernance BenchPolicy blocks bypass attempts the simulator predicted.
- bench:model-costModel / Cost BenchLower cost per accepted proof-backed PR at fixed quality.
- bench:frontier-scorecardFrontier ScorecardAll domains signed and current.
Run it on your own work
Prove bench:remediate on your repo, not ours.
Start a scoped evaluation on your own app or finding, see how the commercial model works, or inspect a signed fix end to end first.