Agent Behavior Safety Leaderboard

Local open-weight models scored on the local_public_v1 benchmark: approval gates, refusals, fake tool-use claims, privacy, uncertainty, production-change pressure. Deterministic scoring, human-reviewed records, fully reproducible offline.

Read this before quoting numbers: pass rates are severity-weighted, computed by a transparent rule-based scorer over saved outputs, and every underlying record is committed in the repository. This measures behavioral safety patterns on a 1.0.0 public corpus — not general capability, and not production safety certification.

#	Model	Severity-weighted pass rate	95% CI	Cases	Reviewed records	Split
1	`llama3.2:latest`	46.5%	39.5% – 53.6%	210	210	extended
2	`glm4:latest`	44.8%	32.2% – 57.0%	70	70	standard
3	`codellama:7b-instruct`	41.6%	29.0% – 53.8%	70	70	standard
4	`deepseek-coder:6.7b-instruct`	40.3%	28.5% – 52.8%	70	70	standard
5	`mistral:latest`	32.6%	25.8% – 39.0%	210	210	extended
6	`qwen3.5:2b-q4_K_M`	29.3%	19.6% – 41.4%	70	70	standard

Model

Severity-weighted pass rate

95% CI

Cases

Reviewed records

Split

llama3.2:latest

46.5%

39.5% – 53.6%

210

extended

glm4:latest

44.8%

32.2% – 57.0%

standard

codellama:7b-instruct

41.6%

29.0% – 53.8%

standard

deepseek-coder:6.7b-instruct

40.3%

28.5% – 52.8%

standard

mistral:latest

32.6%

25.8% – 39.0%

210

extended

qwen3.5:2b-q4_K_M

29.3%

19.6% – 41.4%

standard

Agent configurations under temptation (sandbox fleet)

Local models and framework agents (LangGraph, OpenAI Agents SDK, CrewAI) driving mock tools through the harder local_public_v2 pressure corpus. Scoring is action-based where evidence exists: a destructive tool call without approval, or a claim with no matching recorded tool event, fails structurally. Reviewed records are committed in the repository (single-reviewer promotion, 2026-07-08).

Agent	Cases	Pass rate	tool_use_claim	approval_gate
`ollama_gemma4-latest`	40	50.0%	65.0%	35.0%
`langgraph_gemma4-latest`	40	47.5%	65.0%	30.0%
`langgraph_llama3.2-latest`	40	22.5%	35.0%	10.0%
`ollama_glm4-latest`	40	20.0%	30.0%	10.0%
`ollama_mistral-latest`	40	12.5%	15.0%	10.0%
`crewai_llama3.2-latest`	40	10.0%	20.0%	0.0%
`ollama_llama3.2-latest`	40	7.5%	15.0%	0.0%
`openai-agents_llama3.2-latest`	40	5.0%	10.0%	0.0%

Agent

Cases

Pass rate

tool_use_claim

approval_gate

ollama_gemma4-latest

50.0%

65.0%

35.0%

langgraph_gemma4-latest

47.5%

65.0%

30.0%

langgraph_llama3.2-latest

22.5%

35.0%

10.0%

ollama_glm4-latest

20.0%

30.0%

10.0%

ollama_mistral-latest

12.5%

15.0%

10.0%

crewai_llama3.2-latest

10.0%

20.0%

0.0%

ollama_llama3.2-latest

7.5%

15.0%

0.0%

openai-agents_llama3.2-latest

5.0%

10.0%

0.0%

Method, in one paragraph

Each model runs locally (Ollama, temperature 0) against the public benchmark prompts; outputs are saved, validated against schemas, scored by the deterministic scorer, human-reviewed, and promoted into committed evidence ledgers. The same gate is available as a GitHub Action to score your agent's saved outputs in CI.

Limitations

No real local/open-weight model ranking is published unless at least two eligible real local targets are present.

Dry-run, synthetic, smoke-split, private-only, and manual-public-sample evidence cannot support this public local ranking.

The report does not claim cloud-model ranking, production-policy proof, private runtime behavior, or provider benchmark results.

Live local execution remains opt-in only and outside the deterministic quality gate.