Agent-boundary benchmarks your CISO can reproduce
02 - FAQ's
Frequently asked questions
Scoring & methodology
What does Pass vs Partial vs Fail mean?
Pass — blocked or masked before model/tool execution. Partial — detected or HTTP/content-only, not structural MCP enforce at the agent hop. Fail — payload reached execution unchanged.
Why are some matrix cells N/A?
N/A = outside that product’s layer, not a hidden fail. Lakera classifies prompt content (B1), not MCP tools/call (B2/B3). Kong enforces routes/plugins — no session lineage (B3). OneCLI is doc-verified on egress until local proxy is available.
How is audit quality scored in B8?
0–3 scale: 3 = allow/mask/block + findings + requestId; 2 = decision + timestamp; 1 = log/boolean; 0 = no metadata. Blekline scores 3; partial vendors typically 2.
Stack & competitors
Why are Lakera and Kong Partial on some rows?
Lakera flags prompt content — Pass on B1-style injection, N/A on tool enforce. Kong applies route plugins — Partial when the gateway hop exists but tool-arg mask or lineage is out of scope. Partial = detects or routes, not structural enforce at the agent boundary.
How is this different from an LLM firewall or prompt filter?
Firewalls inspect text. B1–B8 score the full agent path — mask, destructive tools/call, lineage, egress (B6), time-to-govern (B7), audit metadata (B8). Complementary, not the same layer.
Does Blekline replace Kong or Okta?
No. Kong = API routes. Okta/Veza = people and access. Blekline = what agents do at runtime. Regulated teams typically need all three. These benchmarks show where each layer stops.
Reproduction & diligence
Can we reproduce these results in our environment?
Yes — gitignored env.benchmark, pnpm benchmark:run, dated JSON with git SHA. CI runs --quick on every build.
Will enforcement slow down our agents?
B4 charts p99. Blekline local enforce targets <10 ms; cloud mask adds RTT. Lakera/Kong include API/gateway hops — compare at the layer each product owns.
Who sees our data when we run a reproduction?
Your keys only — never commit secrets. Cloud mask over TLS; audit defaults to metadata only. Sidecar = in-VPC enforce. EEA-first SaaS; US-primary on enterprise addendum.
