Instructions to use Epiculous/Crunchy-onion with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Epiculous/Crunchy-onion with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Epiculous/Crunchy-onion")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Epiculous/Crunchy-onion")
model = AutoModelForCausalLM.from_pretrained("Epiculous/Crunchy-onion", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Epiculous/Crunchy-onion with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Epiculous/Crunchy-onion"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Epiculous/Crunchy-onion",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Epiculous/Crunchy-onion

SGLang

How to use Epiculous/Crunchy-onion with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Epiculous/Crunchy-onion" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Epiculous/Crunchy-onion",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Epiculous/Crunchy-onion" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Epiculous/Crunchy-onion",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Epiculous/Crunchy-onion with Docker Model Runner:
```
docker model run hf.co/Epiculous/Crunchy-onion
```

I have no idea what you did, but this model's great.

by CamiloMM - opened Feb 21, 2024

Discussion

CamiloMM

Feb 21, 2024

Been using the Q4_K_M of this (by @LoneStriker ) and it's pretty awesome. And runs fast enough on a 24GB card + RAM, unlike say a 70B L2 model. And the context length is just sweet.

MinP with DynaTemp, ChatML, running on Kobold via ST. I get around ~10 T/s.

I'm constantly blown away by the ever-increasing progress of locally-runnable AI, and it's because of people like you that keep pushing the envelope with these finetunes. Can you imagine if we had Stable Diffusion, but no finetunes or LoRAs? LLaMA2-Chat, all by itself? The AI landscape would look bleak and gray.

Epiculous

Owner Feb 25, 2024

Glad you enjoyed it!

HR1777

Feb 25, 2024

Been using the Q4_K_M of this (by @LoneStriker ) and it's pretty awesome. And runs fast enough on a 24GB card + RAM, unlike say a 70B L2 model. And the context length is just sweet.

@CamiloMM , Could you please give us more information about the ability of this model to keep up at longer inputs? Did you notice any obvious difference between this model and other models in handling long inputs?

CamiloMM

Feb 25, 2024

•

edited Feb 25, 2024

@CamiloMM , Could you please give us more information about the ability of this model to keep up at longer inputs?

I only test models up to 8K tokens of input + smart context in Kobold (I think this means it incurs the cost of 16K tokens internally, but not sure if I got that right), so I can't be sure about that. It does hold up at that range, though (Llama2 with rope scaling usually started being nonsense at some point) It should be capable of 32K tokens, have you had issues?

HR1777

Feb 25, 2024

I have not had the opportunity to try this model yet, but your comment has piqued my interest. My aim is to utilize language models to revise extensive content, specifically articles spanning around 2,500 words. Based on your experience, do you believe this model can effectively process and edit such lengthy inputs (approximately 8K tokens) while preserving crucial information? Although I have experimented so many models, none of them couldn't help me. I am curious to see if this model can accomplish this task. Essentially, I would like to provide the model with a long input and request that it expands and enhances the content.

CamiloMM

Feb 27, 2024

@HR1777 based on your requirements, I'd actually recommend Nous-Hermes-2-Mixtral-8x7B-DPO which you can try out for free on HuggingChat to see if it is smart in the stuff you wanna ask him.

The biggest feature of Crunchy-Onion for me, was that it's absolutely uncensored and fun. It'll take a character card in SillyTavern and roll with it, no questions asked. But for more serious tasks your best bet is probably similar to Nous Hermes. Do also check out Dolphin and Bagel.

These will all have 32K tokens of context as they are 8x7B, and they will be pretty smart, also mostly uncensored but without any bias for trying to be fun (Crunchy Onion is more RP-oriented).

8x7B is very fast by itself, but I recommend also trying exl2 versions of these if you can. Those, will be crazy fast. On a 24GB card, you can run the 3bpw no problem, maybe 3.5bpw.

HR1777

Feb 27, 2024

@CamiloMM , Thank you so much for your time and great suggestions. I am quite confident that the models you introduced are better suited for my requirements than the previously tested ones, as I have come across positive reviews about them on other websites as well. By the way, if you are searching for a completely uncensored model, you may want to try this one: https://huggingface.co/Undi95/Unholy-v2-13B.

CamiloMM

Feb 28, 2024

@CamiloMM , Thank you so much for your time and great suggestions. I am quite confident that the models you introduced are better suited for my requirements than the previously tested ones, as I have come across positive reviews about them on other websites as well. By the way, if you are searching for a completely uncensored model, you may want to try this one: https://huggingface.co/Undi95/Unholy-v2-13B.

Glad to help! In the same vein/size of Unholy 13B, I'd recommend PsyMedRP 20B (also available in 13B), also made by Undi95. It has MedLLaMA_13B in the mix which helps it understand some themes better (anatomy, psychology, etc). That said, if you can run bigger models, it's even better!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment