Norobara-ZLoss-8x7B

This is an experimental instruct-tuned mistralai/Mixtral-8x7B-v0.1-based model trained using Charles Goddard's ZLoss and Megablocks-based fork of transformers.

It primarily uses the Capybara and No Robots datasets (thus the name). The goal was to create an uncensored general instruction following model, as well as test various loss implementations while we figure out how the heck to train Mixtral properly.

Exl2 Quants

Quants courtesy of TheBloke:

• GPTQ
• GGUF
• AWQ

Additional Exl2 Quants courtesy of LoneStriker:

• 2.4bpw
• 3.0bpw
• 3.5bpw
• 3.75bpw
• 4.0bpw
• 5.0bpw
• 6.0bpw

Usage:

The intended prompt format is a modified multi-turn Alpaca instruction format:

### Instruction:
{system prompt}

### Input:
{user message}

### Response:
{model response}

### Input:
{user message}

### Response:
{model response}

(etc.)

Bias, Risks, and Limitations

The model will show biases present in the base model. No ethical alignment was applied to prevent the generation of toxic or harmful outputs (in fact the opposite, with examples from toxic-DPO included), so generate at your own risk.

Training Details

This model was trained as a QLora adapter for 3 epochs using a single H100 GPU for around 13 hours.