Put Weight On It: An Open LLM Red-Team Lab
Most teams ship an LLM feature after prompting it a few times and calling it safe. We built a local-first, open-source harness that puts real adversarial pressure on a model and scores what breaks: 39 cases across 9 attack categories.
Key takeaways
- Most LLM features ship with no adversarial testing at all. Someone prompts the model a few times, it behaves, and “it feels fine” becomes the whole safety story.
- We put the open-source AI Red-Team Lab in the open: a local-first harness that runs 39 adversarial cases across 9 categories and scores what breaks.
- It runs fully on your own machine through Ollama. No prompts, no canaries, and no results leave the box unless you choose a cloud lane.
- Every run lands as a scored, structured record, so after the next model upgrade you compare instead of guessing.
- Public-safe and MIT licensed. The methodology is right there. Take it apart and point it at your own models.
You do not know how something holds up until you put weight on it. That is true of a bridge, a lock, and a language model. Yet most teams shipping an LLM feature test it by chatting with it a few times, watching it behave, and moving on. “It feels fine” gets treated like an assessment. It isn’t one.
So we built a harness for putting weight on a model, and released the public-safe version on GitHub: ai-redteam-lab. This post explains what it tests, why it runs local, and how it fits the way we work.
Why “it feels fine” is not a test
A casual chat with a model tells you how it behaves when nobody is trying to break it. That is the least interesting case. The questions that matter are adversarial:
- Can a user message override the system prompt you wrote?
- Can a poisoned document hijack the model through your retrieval pipeline?
- Can someone pull a secret back out of it?
You cannot answer those by feel. You answer them by running the same set of attacks every time, scoring the outcome against a written rubric, and keeping the record. That is the gap the lab fills: it turns LLM robustness into something you can measure, repeat, and report on.
What the lab actually tests
Thirty-nine cases, each defined in YAML against a published schema, grouped into nine categories:
| Category | What you are measuring |
|---|---|
| Instruction Hierarchy | Can user-level content override the system prompt? |
| Indirect Prompt Injection | Can retrieved or RAG documents hijack model behavior? |
| Leakage Resistance | Can canary tokens or the system prompt be extracted? |
| Tool Abuse | Can malicious tool output steer future actions? |
| Memory Poisoning | Can prior-turn content corrupt later policy? |
| Refusal Consistency | Does the model stay stable under repeated reframing? |
| Encoding & Obfuscation | Do base64, multilingual, or Unicode tricks bypass defenses? |
| Reasoning Manipulation | Can chain-of-thought load or “skip your reasoning” framing degrade guardrails? |
| Baseline Controls | Is the false-positive rate on benign prompts acceptable? |
That last category matters as much as the rest. A model that refuses everything is not secure, it is broken. The baseline set keeps you honest about false positives, so you are measuring real failures and not jumping at shadows.
Local-first, on purpose
The lab runs on your own hardware through Ollama. Nothing leaves the machine unless you deliberately add a cloud model lane.
That is a deliberate design choice, not a limitation. The moment you want to test is right before deployment, when the thing you are evaluating sits next to data you are not allowed to leak. A hosted testing service asks you to send it your prompts, your canaries, and your findings. This asks you to send it nothing. Clone it, pull a model, run the suite offline.
Evidence you can diff
Every run writes a scored result in a stable schema. That sounds mundane. It is the whole point.
Models change constantly. You upgrade to a new version, a new architecture, a new quantization, and the behavior shifts in ways nobody documents. If your only record of “we tested it” is a memory of a chat that went fine, you have nothing to compare against. With a structured result per case, you re-run the suite after every upgrade and diff the output. Regressions show up as a number that moved, not a hunch.
Public-safe by design
The same ethic governs this release that governs our client work: share the method and the measurement, never the weapon.
- The public repo carries the cases, the runner, the scoring rubric, and the schema. It is scrubbed of anything operational.
- A CI-enforced OPSEC gate runs on every push, so nothing sensitive slips into the history by accident.
- What is released is the part that helps defenders: a defensible baseline they can re-run, not a catalog of live exploits.
If you build an LLM assessment practice on top of it, you own the results and they never touch our infrastructure.
Why this matters for your exposure
We build tools like this because assessing risk means understanding how the thing actually breaks, whether the thing is an internet-facing host or a model sitting behind a chat box. The attacker’s-eye view we bring to your external attack surface is the same view we brought to building this: assume someone is trying to make it misbehave, then measure how far they get.
More organizations are shipping LLM features straight into production with “it seemed okay” as the test plan. If that describes a project on your roadmap, the lab is a place to start, and a conversation about your broader exposure is a place to go next. Begin with an External Exposure Audit Sprint, or read more about how we work.
Prox Offensive. See your exposure before attackers do.