Instructions to use YeungNLP/LongQLoRA-Llama2-7b-8k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use YeungNLP/LongQLoRA-Llama2-7b-8k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="YeungNLP/LongQLoRA-Llama2-7b-8k")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("YeungNLP/LongQLoRA-Llama2-7b-8k")
model = AutoModelForCausalLM.from_pretrained("YeungNLP/LongQLoRA-Llama2-7b-8k")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use YeungNLP/LongQLoRA-Llama2-7b-8k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "YeungNLP/LongQLoRA-Llama2-7b-8k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YeungNLP/LongQLoRA-Llama2-7b-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/YeungNLP/LongQLoRA-Llama2-7b-8k

SGLang

How to use YeungNLP/LongQLoRA-Llama2-7b-8k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "YeungNLP/LongQLoRA-Llama2-7b-8k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YeungNLP/LongQLoRA-Llama2-7b-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "YeungNLP/LongQLoRA-Llama2-7b-8k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YeungNLP/LongQLoRA-Llama2-7b-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use YeungNLP/LongQLoRA-Llama2-7b-8k with Docker Model Runner:
```
docker model run hf.co/YeungNLP/LongQLoRA-Llama2-7b-8k
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

LongQLoRA: Efficient and Effective Method to Extend Context Length of LLMs

Technical Report

Technical Report: LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models

Introduction

LongQLoRA is a memory-efficient and effective method to extend context length of Large Language Models with less training GPUs. On a single 32GB V100 GPU, LongQLoRA can extend the context length of LLaMA2 7B and 13B from 4096 to 8192 and even to 12k. LongQLoRA achieves competitive perplexity performance on PG19 and Proof-pile dataset after only 1000 finetuning steps, our model outperforms LongLoRA and is very close to MPT-7B-8K.

Evaluation perplexity on PG19 validation and Proof-pile test datasets in evaluation context length of 8192:

Model	PG19	Proof-pile
LLaMA2-7B	>1000	>1000
MPT-7B-8K	7.98	2.67
LongLoRA-LoRA-7B-8K	8.20	2.78
LongLoRA-Full-7B-8K	7.93	2.73
LongQLoRA-7B-8K	7.96	2.73

Downloads last month: 199

Model tree for YeungNLP/LongQLoRA-Llama2-7b-8k

Quantizations

1 model

Paper for YeungNLP/LongQLoRA-Llama2-7b-8k

LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models

Paper • 2311.04879 • Published Nov 8, 2023