Featured · Open source · Python
GuardRail-as-a-Service — prompt-injection detection for production LLM serving
A two-tier pipeline that filters prompts before they reach the model. Tier 1 is a deterministic Aho-Corasick + regex matcher that clears the common path in under 0.1 ms; Tier 2 is a DeBERTa-v3 ONNX classifier that only ever sees what Tier 1 lets through.
Tier 1's blocklist was tuned by inspecting the attacks it missed on my own corpus,
so its 1.000 in-sample recall is partly memorisation. Split 60/40 and refitted on the
training half only, it scores 0.7895 on held-out data — a 21-point
overstatement if I quoted the in-sample figure. The repo says so in its own README,
labels the contaminated row “not a result”, and stores every number as reproducible
JSON under results/.
Measured performance
| Evaluation | Tier 1 recall | FPR | Reading |
|---|---|---|---|
| In-house corpus (in-sample) | 1.000 | 0.000 | Fit, not generalisation |
| Held-out split, train-fitted rules | 0.7895 | 0.000 | Best estimate of real performance |
Open-Prompt-Injection (source="data") |
1.000 | 0.006 | External benchmark |
BIPIA (source="data") |
0.480 | 0.017 | Where it underperforms |
| NotInject (benign only, over-defense) | — | 0.000 | No over-blocking at Tier 1 |
Latency p50 / p95 / p99 — 0.14 ms / 26.21 ms / 32.11 ms. Tier 1 alone: p50 0.09 ms.
Engineering decision
Tier 2 ships monitor-only
On the NotInject over-defense benchmark the classifier blocks 40.4% of benign prompts
against Tier 1's 0.0%, and the over-defense is not threshold-separable. So it scores
and logs a TIER2-FLAG but does not block. The cost is a recall
ceiling, stated explicitly rather than hidden; the blocking mode stays reachable via
POST /validate?tier=2.
Engineering decision
Double scan, and fail open
Tier 1 scans normalised text first (keeping |,
_, - so markers like
<|im_start|> still match), then a de-leeted, separator-stripped
variant — which is what catches 1gn0r3 prev10us 1nstruct10ns.
If the Tier 2 model errors, the request is allowed: availability over coverage, deliberately.
Baseline comparison — read carefully
| Metric | GuardRail | Llama-Guard-3-8B |
|---|---|---|
| Recall | 1.0000 | 0.1316 |
| Precision | 1.0000 | 1.0000 |
| Latency p50 | 0.16 ms | 1708 ms |
This is not “7× better than LlamaGuard”. Llama-Guard-3 is a content-safety classifier whose S1–S13 taxonomy does not include prompt injection — it missed 165/190 attacks because it is not an injection detector. The defensible claim is that content-safety guardrails do not transfer to prompt-injection detection. The latency figure is local Ollama inference on a GTX 1650, not a hosted API round-trip, and the results JSON records that so it cannot be misquoted downstream.