Instructions to use thesven/Chatty-McChatterson-3-mini-128k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use thesven/Chatty-McChatterson-3-mini-128k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="thesven/Chatty-McChatterson-3-mini-128k", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("thesven/Chatty-McChatterson-3-mini-128k", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("thesven/Chatty-McChatterson-3-mini-128k", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use thesven/Chatty-McChatterson-3-mini-128k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "thesven/Chatty-McChatterson-3-mini-128k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thesven/Chatty-McChatterson-3-mini-128k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/thesven/Chatty-McChatterson-3-mini-128k

SGLang

How to use thesven/Chatty-McChatterson-3-mini-128k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "thesven/Chatty-McChatterson-3-mini-128k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thesven/Chatty-McChatterson-3-mini-128k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "thesven/Chatty-McChatterson-3-mini-128k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thesven/Chatty-McChatterson-3-mini-128k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use thesven/Chatty-McChatterson-3-mini-128k with Docker Model Runner:
```
docker model run hf.co/thesven/Chatty-McChatterson-3-mini-128k
```

Chatty-McChatterson-3-mini-128k

Model Details

Model Name: Chatty-McChatterson-3-mini-128k Base Model: microsoft/Phi-3-mini-128k-instruct
Fine-tuning Method: Supervised Fine-Tuning (SFT)
Dataset: ultrachat_200k
Training Data: 12884 conversations selected for being 512 input tokens or less Training Duration: 4 hours
Hardware: Nvidia RTX A4500
Epochs: 3

Training Procedure

This model was fine-tuned to provide better instructions on code.

The training was conducted using PEFT and SFTTrainer on select conversations from the Ultra Chat 200k dataset. Training was completed in 3 epochs (19326 steps) over a span of 4 hours on an Nvidia A4500 GPU.

The dataset comprised of a filterd list of rows from the Ultra Chat 200k dataset, where the prompt template was 512 tokens or less.

Intended Use

This model is designed to improve the overall chat experience and response quality.

Getting Started

Instruct Template

<|system|>
{system_message} <|end|>
<|user|>
{Prompt) <|end|>
<|assistant|>

Transfromers

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name_or_path = "thesven/Chatty-McChatterson-3-mini-128k"

# BitsAndBytesConfig for loading the model in 4-bit precision
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
)

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name_or_path,
    device_map="auto",
    trust_remote_code=False,
    revision="main",
    quantization_config=bnb_config
)
model.pad_token = model.config.eos_token_id

prompt_template = '''
<|user|>
What is the name of the big tower in Toronto?.<|end|>
<|assistant|>
'''

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.1, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=256)

generated_text = tokenizer.decode(output[0, len(input_ids[0]):], skip_special_tokens=True)
print(generated_text)

Downloads last month: 4

Safetensors

Model size

4B params

Tensor type

F32

Dataset used to train thesven/Chatty-McChatterson-3-mini-128k

Collection including thesven/Chatty-McChatterson-3-mini-128k

Finetune

Collection

10 items • Updated Mar 2