Where static analysis is easy vs hard: a walk through 2,740 OWASP tests
The OWASP Benchmark's 2,740 labelled cases show a clear pattern: SAST nails pattern-local bugs (weak crypto at 100% precision) and struggles with anything that needs dataflow (SQL injection at 39% recall). Here's the whole scorecard — including where our own engine scores zero.
TL;DR — The OWASP Benchmark is 2,740 labelled Java test cases across 11 vulnerability classes. Running our engine over all of them and reading the results by category reveals a clean rule: a scanner's accuracy tracks how local the bug's signal is. Weak-crypto bugs (one function call) score 100% precision; injection bugs (untrusted input traced to a sink) drop to 39–57% recall; and two classes we don't yet cover score a flat zero. We publish the whole scorecard — strengths and zeros — because the shape is the interesting part.
The benchmark
The OWASP Benchmark v1.2 is the standard test suite for SAST tools: 2,740 Java cases, each labelled true (a real vulnerability) or false (a safe look-alike). The headline metric is the Benchmark Score = true-positive rate − false-positive rate (Youden's J) — it rewards catching real bugs and punishes crying wolf equally.
We measured Zennoxa Shield across the full suite (commit 9d182dd, 2026-07-21):
Benchmark Score +0.547, precision 92.4%, recall 60.0%, FPR 5.3%. But the
overall number hides the story. Here's every category, ranked:
| Vulnerability class | Recall | Precision | Benchmark Score |
|---|---|---|---|
| Weak randomness | 100.0% | 100.0% | +1.000 |
| Weak cipher | 80.8% | 100.0% | +0.808 |
| Weak hash | 69.0% | 100.0% | +0.690 |
| Cross-site scripting | 57.3% | 92.2% | +0.516 |
| XPath injection | 46.7% | 100.0% | +0.467 |
| Command injection | 57.9% | 82.0% | +0.451 |
| Path traversal | 53.4% | 85.5% | +0.445 |
| SQL injection | 39.0% | 84.1% | +0.303 |
| Trust boundary | 47.0% | 79.6% | +0.237 |
| LDAP injection | 0.0% | — | +0.000 |
| Insecure cookie | 0.0% | — | +0.000 |
The pattern: locality predicts accuracy
Read that table top to bottom and a gradient appears.
Top — "syntactic" bugs (100% precision). Weak randomness, weak cipher, weak
hash. The signal is local: it's a single call — new Random(), DES,
MessageDigest.getInstance("MD5"). You don't need to know where any data came
from; the presence of the API is the finding. So a scanner can be both complete
and precise here — 100% precision on all three. If a tool can't nail these, be
worried.
Middle — injection (39–57% recall). XSS, command injection, path traversal, SQL injection. These are not local: a bug exists only if untrusted input reaches a dangerous sink. The scanner has to trace data from a source (a request parameter) through assignments and helper methods to the sink (a query, a shell, a file path). Miss one hop and you miss the bug (lower recall); over-approximate and you flag safe code (lower precision). SQL injection is the hardest here at 39% recall — SQL gets built through the most indirection.
Bottom — the zeros. LDAP injection and insecure-cookie flags score 0.0 for us: we don't ship rules for them yet. We're showing that rather than hiding it. A Benchmark Score of exactly 0 means "didn't play" — not a subtle failure, a coverage gap. (Precision is undefined when a tool makes zero findings in a class, hence the "—".)
Why this matters if you're choosing or trusting a scanner
- A single headline number is misleading. "92% precision" is true for us overall, but it's 100% on weak crypto and undefined on LDAP injection. Always ask for the per-class breakdown.
- Recall is where SAST quietly fails. Precision gets marketed; recall gets buried. On the hardest injection classes even a well-tuned engine catches well under half in this suite. SAST is a filter, not a guarantee — pair it with review and test coverage for the injection-heavy parts of your app.
- "Syntactic" classes are table stakes. If a scanner misses weak crypto/hash/ random, that's a red flag — those are the easy ones.
Honest limitations
- This is one benchmark, in Java. Real codebases have framework magic and cross-file flows the Benchmark's self-contained cases don't fully exercise; scores elsewhere will differ.
- We optimise precision-first, which caps recall by design — a tool tuned the other way would trade our low 5.3% FPR for higher recall and more noise.
- The two zeros are real coverage gaps, not measurement artifacts.
Reproduce it
git clone --depth 1 https://github.com/OWASP-Benchmark/BenchmarkJava
OWASP_BENCH=$PWD/BenchmarkJava make bench-owasp
You get the same per-category scorecard. The full ranked numbers are in
data.json, and every dated run is filed in our
Benchmark Archive.
For how other engines score on the same suite, see OWASP's own published
scorecards — we cite theirs rather than measuring other tools ourselves.
Zennoxa Research publishes reproducible security data. This study reports only our own engine's results on a public dataset.
Shield reports its recall per vulnerability class, so you know which bug types it actually catches before you rely on it.
See what Shield finds in your code
Free to start, no credit card. One pass covers SAST, secrets, vulnerable dependencies, containers and IaC — ranked by what’s actually reachable, not just severity.
Scan your repo freePrefer the terminal? The offline CLI runs with no account