Document honest submission guardrails

Co-authored-by: Codex <codex@openai.com>

FIELD_NOTES.md

@@ -76,3 +76,9 @@ Added:
 - runtime fallback so malformed model JSON falls back to deterministic safety analysis
 
 Deployment rule: publish or deploy a LoRA adapter only if it improves valid JSON and keeps dangerous-as-safe misses at zero. A worse-but-fine-tuned model should not ship.
+
+## 2026-06-06 Submission Honesty
+
+Latest Discord discussion surfaced MiniCPM4.1 Transformers issues around SDPA attention masks and gibberish output. Jawbreaker now defaults `JAWBREAKER_ATTENTION_IMPLEMENTATION=eager` for the MiniCPM path, and keeps fallback analysis visible in the documentation.
+
+Guardrail: winning matters, but not by overclaiming. The project should not claim Well-Tuned, real user validation, Modal usage, or llama.cpp deployment unless those things are actually completed and documented.

HONEST_SUBMISSION.md

@@ -0,0 +1,45 @@
+# Honest Submission Guardrails
+
+Jawbreaker should compete hard without gaming the hackathon.
+
+## What We Can Claim
+
+- Backyard AI: Jawbreaker is built for a real, specific safety problem: helping someone pause before clicking, replying, or sending money.
+- Small-model constraint: the deployed model is under 32B parameters.
+- Gradio Space: the app runs as a Gradio app under the hackathon organization.
+- OpenBMB targeting: MiniCPM is central to the deployed scam analysis path.
+- Codex usage: the GitHub repo has Codex-attributed commits and the build logs document Codex's role.
+- Custom UI: the app uses a custom Gradio interface, not the default demo look.
+- Eval work: the repo includes hand-curated synthetic/sanitized eval cases plus generated synthetic train/dev/test data.
+- Safety guardrails: model JSON is validated, unsafe actions are checked in eval, and a deterministic fallback handles malformed model output.
+
+## What We Should Not Claim Unless Completed
+
+- Do not claim the Well-Tuned badge unless a fine-tuned adapter is actually trained, published on Hugging Face, and used or evaluated honestly.
+- Do not claim real-world user validation until the target person has actually tried the app.
+- Do not claim the generated synthetic dataset is real user data.
+- Do not claim benchmark superiority unless we publish the eval command, dataset, and result.
+- Do not claim llama.cpp as the deployed runtime unless the live Space actually uses the llama.cpp path.
+- Do not claim Modal usage unless training or inference actually runs on Modal.
+- Do not hide fallback behavior. If MiniCPM fails and Jawbreaker uses deterministic safety fallback, document that as a reliability layer.
+
+## Training Rules
+
+- Training data and eval data must stay separated.
+- `eval/scam_eval.jsonl` is the hand-curated eval set and should not be used as training data.
+- `training/data/test.jsonl` and `eval/generated_eval.jsonl` are holdout data and should not be used for LoRA training.
+- A LoRA adapter should only be deployed if it improves valid JSON and safety metrics without increasing dangerous false negatives.
+- If synthetic training mostly teaches format-following, say that. Do not describe it as learning from real scam victims.
+
+## Demo Rules
+
+- The demo can use sanitized examples.
+- If a real family story is used, remove names, phone numbers, addresses, account details, and links.
+- The demo should show the actual Space behavior, not a scripted mockup.
+- If the Space is slow because of ZeroGPU or MiniCPM loading, say so in field notes rather than pretending it is instant.
+
+## Winning Without Cheating
+
+The strongest honest story is:
+
+Jawbreaker is a narrow, polished Backyard AI tool using OpenBMB MiniCPM under the 32B limit. It turns a suspicious message into a clear scam DNA breakdown and a safe next step for a non-technical person. The repo shows evals, safety fallbacks, training scaffolding, Codex traces, and field notes. Claims are limited to what is actually built and measured.

README.md

@@ -43,6 +43,7 @@ Codex evidence:
 - Codex-attributed commits will be included for build work.
 - `AGENT_TRACE.md` records the development process.
 - `FIELD_NOTES.md` records product and technical decisions.
+- `HONEST_SUBMISSION.md` records what the project can and cannot honestly claim.
 
 ## Why This Is Small
 
@@ -82,13 +83,14 @@ Training/eval artifacts:
 - `training/data/train.jsonl`, `dev.jsonl`, `test.jsonl`: generated SFT records for Jawbreaker JSON behavior.
 - `eval/generated_eval.jsonl`: generated holdout eval set.
 - `training/train_lora.py`: PEFT/LoRA scaffold for publishing a Jawbreaker MiniCPM adapter if it beats the base model.
+- `HONEST_SUBMISSION.md`: guardrails to avoid overclaiming synthetic data, fine-tuning, or runtime behavior.
 
 ## Bonus Badges Targeted
 
 - Off the Grid: local model inference, no cloud APIs for scam analysis.
 - Llama Champion: local/eval tooling supports the llama.cpp runtime.
 - Off-Brand: custom Gradio UI beyond the default look.
-- Well-Tuned: cut; the project prioritized safe inference, evals, and product quality over a rushed fine-tune.
+- Well-Tuned: not claimed unless a fine-tuned adapter is actually trained, published, and evaluated.
 - Sharing is Caring: Codex/agent trace published in this repo.
 - Field Notes: build report published before submission.
 

SETUP.md

@@ -71,6 +71,7 @@ For responsive model inference without paid hardware:
 - set Space variable `JAWBREAKER_BACKEND=zerogpu`
 - keep `JAWBREAKER_TRANSFORMERS_MODEL_ID=openbmb/MiniCPM4.1-8B` for the OpenBMB-targeted build
 - keep `JAWBREAKER_TRUST_REMOTE_CODE=true`; MiniCPM requires custom model code through Transformers
+- keep `JAWBREAKER_ATTENTION_IMPLEMENTATION=eager`; Discord reports MiniCPM4.1 SDPA mask issues in some Transformers/PyTorch environments
 - fallback model if latency is unacceptable: `Qwen/Qwen3-0.6B`
 
 CPU fallback remains available with `JAWBREAKER_BACKEND=llama-cpp`.

app.py

@@ -142,6 +142,7 @@ def get_analyzer():
             device_map=os.getenv("JAWBREAKER_DEVICE_MAP", "auto"),
             dtype=os.getenv("JAWBREAKER_TORCH_DTYPE", "auto"),
             trust_remote_code=_env_bool("JAWBREAKER_TRUST_REMOTE_CODE", True) or False,
+            attn_implementation=os.getenv("JAWBREAKER_ATTENTION_IMPLEMENTATION", "eager") or None,
         )
 
     raise ValueError(f"Unsupported JAWBREAKER_BACKEND: {backend}")

eval/run_eval.py

@@ -66,6 +66,7 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--device-map", default="auto", help="Transformers device_map.")
     parser.add_argument("--dtype", default="auto", help="Transformers dtype.")
     parser.add_argument("--trust-remote-code", action=argparse.BooleanOptionalAction, default=True)
+    parser.add_argument("--attn-implementation", default="eager", help="Transformers attention implementation.")
     return parser.parse_args()
 
 
@@ -161,6 +162,7 @@ def build_analyzer(args: argparse.Namespace):
             device_map=args.device_map,
             dtype=args.dtype,
             trust_remote_code=args.trust_remote_code,
+            attn_implementation=args.attn_implementation,
         )
         return lambda row: analyzer(row["input"])
 

jawbreaker/analyzers.py

@@ -179,6 +179,7 @@ def build_transformers_analyzer(
     device_map: str = "auto",
     dtype: str = "auto",
     trust_remote_code: bool = False,
+    attn_implementation: str | None = None,
 ) -> Analyzer:
     try:
         import torch
@@ -191,15 +192,21 @@ def build_transformers_analyzer(
     started = perf_counter()
     print(
         "jawbreaker transformers load_start "
-        f"model_id={model_id} device_map={device_map} dtype={dtype} trust_remote_code={trust_remote_code}",
+        f"model_id={model_id} device_map={device_map} dtype={dtype} "
+        f"trust_remote_code={trust_remote_code} attn_implementation={attn_implementation}",
         flush=True,
     )
     tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=trust_remote_code)
+    model_kwargs: dict[str, Any] = {
+        "dtype": dtype,
+        "device_map": device_map,
+        "trust_remote_code": trust_remote_code,
+    }
+    if attn_implementation:
+        model_kwargs["attn_implementation"] = attn_implementation
     model = AutoModelForCausalLM.from_pretrained(
         model_id,
-        dtype=dtype,
-        device_map=device_map,
-        trust_remote_code=trust_remote_code,
+        **model_kwargs,
     )
     model.eval()
     device = getattr(model, "device", "unknown")