Obfuscated-Shell Safety Leaderboard
Can a model see the real intent of a shell command when it is disguised? A safety gate that
only reads plain text is blind to the same command hidden in base64, hex, eval, or an interpreter.
This board rewards reading the decoded intent through the disguise: denying the dangerous while still
allowing the harmless that merely looks encoded.
Ranked by Balanced Accuracy. Click any column header to sort. Official split: a private held-out set of 221 commands, scored 3× with majority vote. The model answers for itself; no AI judge.
| # | Model | Type | Balanced Acc ▼ | Risky Recall ▼ | Benign Pass ▼ | Obf Gap ▼ |
|---|
Higher Balanced Accuracy, Risky Recall and Benign Pass are better; Obfuscation Gap is best near 0.
How to read this leaderboard
Each model acts as a shell-safety gate: it sees one command (often disguised) and answers ALLOW or DENY. We compare that answer to the known correct label.
| Column | What it means | Better |
|---|---|---|
| Balanced Accuracy | The headline: the average of catching the dangerous and passing the safe. A model that denies everything lands at 50, so it cannot be gamed by blocking. | higher |
| Risky Recall | Of the truly dangerous commands, how many it denied. | higher |
| Benign Pass | Of the harmless commands, how many it correctly allowed. | higher |
| Obfuscation Gap | How much worse the model does on disguised commands than on plain ones. Near 0 means it reads through the disguise; a large gap means it only handles plain text. | near 0 |
The one thing to look at
Almost every model denies dangerous commands well (Risky Recall is high across the board). The real difference is Benign Pass: does the model understand a disguised command, or does it block anything that looks encoded? Low Obfuscation Gap with high Benign Pass means the model actually decodes intent; a large gap with low Benign Pass means it reads only plain text and blocks disguises out of caution.
Scoring
A public development set is provided for reference and tuning; the official score is computed only on the private held-out set. The held-out set is kept off the hub, so no model can be tuned to the test. Each command is scored three times and we take the majority answer, because hosted models are not perfectly repeatable even at temperature 0. There is no AI judge; answers are compared to fixed correct labels.
The dataset
Built on FINAL-Bench/Obfuscated-Shell-Gate:
shell commands, each either genuinely dangerous or harmless, many wrapped in a layer of obfuscation (base64,
hex, octal, rot13, reverse, IFS and variable splitting, quote/backslash breaking, process substitution,
interpreter decoders, and multi-layer chains). Every obfuscated command is verified in a real shell to decode
back to its plain form, and nothing is runnable malware: all network hosts use the unreachable
.invalid domain and destructive examples target a sandbox path.
Submit a model
To add a model to the board, open a post in the Community tab of this Space (the 💬 button at the
top-right) with the model id (an OpenRouter id like vendor/model, or a Hugging Face model
id) and, if it is served through a specific endpoint, that endpoint.
We run the frozen scoring protocol on the private held-out set and append a verified row. Every model is scored by us under the same protocol, so self-reported numbers are not added; only our runs appear. Posting in the Community tab keeps a public, time-stamped record of every submission.
Catch the dangerous vs. allow the harmless
Every model sits near the top (it catches dangerous commands). The spread is horizontal: how many disguised-but-harmless commands it still allows. The top-right corner is the goal — block the bad without blocking the merely-encoded good.
Who reads through the disguise — Obfuscation Gap
How much worse a model does on disguised commands than on plain ones. Near 0 (green) means it decodes intent; a large bar (red) means it mostly handles plain text and blocks disguises out of caution.
The real separator — Benign Pass
Allowing the harmless is where models split apart, from ~97% down to ~16%. This single column decides most of the ranking.