LoRA: Low-Rank Adaptation of Large Language Models
Paper • 2106.09685 • Published • 66
BART-large LoRA for Dialogue Summarization
A LoRA adapter (~4.7 MB) for facebook/bart-large, fine-tuned on SAMSum to summarize English chat-style conversations.
Live demo: https://huggingface.co/spaces/Msoldier-ai/parsbert-qa-and-samsum · Code: https://github.com/MmdDevAi/summarization-bart-lora
Project scope
This model was built as a learning project to get hands-on with LoRA fine-tuning and evaluation, with the help of AI coding assistants. The final model is a single training run with standard hyperparameters (no hyperparameter search), so the scores below are a baseline rather than a tuned result.
Usage
from peft import PeftModel from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tok = AutoTokenizer.from_pretrained("facebook/bart-large") base = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-large") model = PeftModel.from_pretrained(base, "Msoldier-ai/bart-large-samsum-lora").merge_and_unload().eval()
text = "Anna: Are we still meeting at 6?\nTom: Yes, at the cafe near the station." x = tok(text, return_tensors="pt", max_length=512, truncation=True) out = model.generate( **x, num_beams=4, max_length=128, min_length=15, length_penalty=2.0, no_repeat_ngram_size=3, early_stopping=True, ) print(tok.decode(out[0], skip_special_tokens=True))
Training
| Setting | Value |
|---|---|
| Base model | facebook/bart-large |
| Dataset | SAMSum |
| Method | LoRA (SEQ_2_SEQ_LM), PEFT 0.20.0 |
| Rank / alpha / dropout | 8 / 32 / 0.05 |
| Target modules | q_proj, v_proj |
| Epochs | 3 |
| Learning rate | 2e-4 |
| Batch size | 4 per device × 4 gradient accumulation = 16 |
| Weight decay | 0.01 |
| Precision | fp16 |
| Model selection | best checkpoint by validation loss |
| Hardware | Google Colab, NVIDIA T4 |
Results
Evaluated on the SAMSum test split (beam search with 4 beams, max length 128).
| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum |
|---|---|---|---|
| 51.26 | 26.39 | 42.44 | 47.24 |
Intended use
- Demos, learning, and as a baseline for English chat-dialogue summarization.
- Not intended for production or high-stakes uses (legal, medical, HR decisions). Summaries can miss key facts or misattribute actions, so review them before relying on them.
- Dialogues may contain personal information, so avoid submitting sensitive data to the public demo.
Limitations
- English only, trained on casual chat-style dialogues.
- With several speakers, it sometimes attributes an action to the wrong person.
- On very short dialogues it may add redundant or incorrect sentences.
- Less reliable on other domains (meetings, call transcripts, news).
- SAMSum is released under a non-commercial license, so check it before any commercial use.
References
- BART: Lewis et al., 2019 (arXiv:1910.13461)
- SAMSum: Gliwa et al., 2019 (arXiv:1911.12237)
- LoRA: Hu et al., 2021 (arXiv:2106.09685)
- Downloads last month
- 114
Model tree for Msoldier-ai/bart-large-samsum-lora
Base model
facebook/bart-large