shadowrock-io/Harrier-OSS-v1-0.6B-Community-FP8

Harrier OSS v1 0.6B — Community FP8

Unofficial community quantization; no Microsoft affiliation or endorsement.

FP8 W8A8 quantization of microsoft/harrier-oss-v1-0.6b, pinned to revision f9b9dc8d. Microsoft developed the underlying Qwen3 embedding model. This build changes the precision of its transformer linear layers and packages the result for vLLM. Checkpoint size decreases from 1.19 GB to 0.70 GB; this is a storage comparison, not a measurement of total runtime memory.

What is quantized

• FP8_DYNAMIC through llm-compressor 0.13.0 / compressed-tensors 0.18.0: per-channel E4M3 weight scales and dynamic per-token E4M3 activations.
• 196 transformer linear projections. Embeddings, normalization, and the pooling/normalization heads retain full precision.
• Last-token pooling and L2 normalization follow the upstream model configuration.
• The historical export used 448 public or authored calibration samples. This scheme does not require data-driven weight-scale calibration.

See provenance, recipe, and the module inventory.

Benchmarks / validation

These are retained measurements from August 18, 2026. We compared FP8 with the same model served as BF16 through vLLM using the same evaluation wrapper. They are our paired baseline, not Microsoft's official leaderboard results. This documentation update rechecks the stored scores and fixture identities; it does not report a new inference run.

Task	BF16 via vLLM	FP8 via vLLM	FP8 − BF16
STSBenchmark	0.825287	0.825317	+0.000030
STS17	0.813517	0.813170	-0.000347
SciFact	0.728410	0.730090	+0.001680
NFCorpus	0.324940	0.325040	+0.000100

STSBenchmark and STS17 use their MTEB main scores; STS17 averages 11 subsets. SciFact and NFCorpus use nDCG@10. Mean absolute difference across the four tasks: 0.000539, below the existing 0.005 threshold. Every absolute task difference is below 0.01. BF16 results · FP8 results.

The fixture comparison covers 190 cases: 189 normal and one long-input stress case.

Fixture metric	Measured	Existing threshold
Normal-case cosine, 5th percentile	0.997672	≥ 0.995
Normal-case cosine, minimum	0.995669	≥ 0.99
Long-input stress cosine	0.997974	≥ 0.95

The retained results pass these thresholds. The 270M fixture thresholds were revised after the initial FP8 observations; they were not an independent preregistered acceptance test. This update preserves the historical thresholds. Per-case results · fixture identities · thresholds.

Recheck the published evidence

From a downloaded copy of this repository:

python scripts/validate_evidence.py

The verifier checks evidence hashes, the exact case-ID set, stress-case separation, per-subset score means, and acceptance thresholds. It uses Python's standard library. Fixture source-text hashes and the original golden-embedding hash appear in the fixture manifest. Dataset text, binary embeddings, and full inference/quantization harnesses are not included in this release.

Evaluation scope

Tests used an RTX 5070 Ti (16 GB), vLLM 0.26.0, llm-compressor 0.13.0, compressed-tensors 0.18.0, and torch 2.11.0+cu130. The fixture engine used BF16 compute, eager execution, and an 8192-token limit. These are the original recorded conditions, not a compatibility guarantee for newer runtimes.

The MTEB summaries retain scores and subsets, but not a complete environment lock or dataset snapshot record. The historical wrapper passes text to vLLM without explicit per-task prompt selection. These measurements should not be treated as full reproduction of Microsoft's evaluation protocol. They establish neither latency nor throughput improvements.

Serving / usage

Use a vLLM installation with support for this compressed-tensors FP8 format and compatible GPU kernels. The usage below follows the tested 0.26.0 serving path.

from vllm import LLM

llm = LLM(
    model="shadowrock-io/Harrier-OSS-v1-0.6B-Community-FP8",
    runner="pooling",
    trust_remote_code=True,
    dtype="bfloat16",
    max_model_len=8192,
)

documents = ["The observatory opens its roof after sunset."]
query = (
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\n"
    "Query: when does the observatory open its roof"
)
document_vectors = llm.embed(documents)
query_vectors = llm.embed([query])

Apply the upstream instruction prefix to queries once; passages need no prefix. vLLM reads last-token pooling and L2 normalization from the supplied configuration. Trailing whitespace can change the last token and therefore the embedding.

Caveats

• Four benchmark tasks and this fixture set do not establish quality across all languages, domains, or context lengths.
• The single stress case cannot characterize long-document quality.
• FP8 numerical differences remain; validate retrieval quality for your own corpus.
• No speed, energy, or universal hardware-compatibility claim accompanies this release.
• No NVFP4 companion release is available for this model.

Intended use & limitations

Use the embeddings for retrieval, semantic similarity, clustering, and related applications covered by the upstream model card. Microsoft's intended-use and safety guidance still applies. Our measurements cover only the cases and tasks disclosed above.

Attribution & citation

Quantization, validation, and packaging by Matt Busi at ShadowRock. Cite Microsoft's original model for the architecture and training.

License

MIT. See LICENSE and the upstream model's license declaration.

About ShadowRock

ShadowRock is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at shadowrock.io or on LinkedIn.