Instructions to use deepvk/llava-saiga-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepvk/llava-saiga-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="deepvk/llava-saiga-8b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("deepvk/llava-saiga-8b")
model = AutoModelForImageTextToText.from_pretrained("deepvk/llava-saiga-8b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use deepvk/llava-saiga-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepvk/llava-saiga-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepvk/llava-saiga-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/deepvk/llava-saiga-8b

SGLang

How to use deepvk/llava-saiga-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepvk/llava-saiga-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepvk/llava-saiga-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepvk/llava-saiga-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepvk/llava-saiga-8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use deepvk/llava-saiga-8b with Docker Model Runner:
```
docker model run hf.co/deepvk/llava-saiga-8b
```

GGUF version

by maria-ai - opened Nov 17, 2024

Discussion

maria-ai

Nov 17, 2024

Can't make GGUF version. Is it possible?
https://huggingface.co/spaces/ggml-org/gguf-my-repo
ERROR:hf-to-gguf:Model LlavaForConditionalGeneration is not supported

SpirinEgor

deep vk org Nov 18, 2024

Hi!
Try to use official repo with detailed instructions: https://github.com/ggerganov/llama.cpp/blob/master/examples/llava/README.md

maria-ai

Nov 20, 2024

@SpirinEgor

I used it too. But i got errors (i tried to fix it but i can't):
python examples/llava/llava-surgery-v2.py -m llava-saiga-8b

No tensors found. Is this a LLaVA model?

SpirinEgor

deep vk org Nov 22, 2024

I see the problem

Official repo use the legacy model organisation, i.e. its state dict stores vision tower under model.vision_tower and projector under model.projector. You can see it in proj_criteria method, for example. But there are plenty hard-coded names :(
Our implementation is synchronised with 🤗 and use other mapping in state dict. You can explore it with safetensor viewer on hub.

Therefore, if you want to convert this model to GGUF (and probably any other llava model on hf), you need to create your own llava-surgery script that separate vision tower (Clip), project, and LM (LLaMA). And then convert each part to GGUF version.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment