serdarcaglar/primary-school-math-question-multi-lang

SetFit with sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 128 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
non_math	'Güneş sisteminde kaç gezegen vardır?' 'What is the largest ocean on Earth?' 'What is the intake of energy and nutrients by living organisms called?'
math	"Bir düzensiz çokgenin kenar uzunlukları 5 cm, 8 cm, 7 cm ve 6 cm'dir. Çokgenin çevresi kaç cm'dir?" '27 ÷ 3 = ?' 'Bir altıgenin bir iç açısının ölçüsü 120° ise, tüm iç açıların toplamı kaç derecedir?'

Evaluation

Metrics

Label	Accuracy
all	1.0

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("serdarcaglar/primary-school-math-question-multi-lang")
# Run inference
preds = model("Bir üçgenin tabanı 12 cm, yüksekliği 8 cm'dir. Üçgenin alanı kaç cm²'dir?")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	10.3435	33

Label	Training Sample Count
math	459
non_math	129

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0007	1	0.1982	-
0.0340	50	0.1009	-
0.0680	100	0.0054	-
0.1020	150	0.002	-
0.1361	200	0.001	-
0.1701	250	0.0023	-
0.2041	300	0.0002	-
0.2381	350	0.0013	-
0.2721	400	0.0004	-
0.3061	450	0.0004	-
0.3401	500	0.0002	-
0.3741	550	0.0001	-
0.4082	600	0.0001	-
0.4422	650	0.0002	-
0.4762	700	0.0001	-
0.5102	750	0.0001	-
0.5442	800	0.0002	-
0.5782	850	0.0002	-
0.6122	900	0.0001	-
0.6463	950	0.0005	-
0.6803	1000	0.0001	-
0.7143	1050	0.0001	-
0.7483	1100	0.0001	-
0.7823	1150	0.0001	-
0.8163	1200	0.0001	-
0.8503	1250	0.0001	-
0.8844	1300	0.0	-
0.9184	1350	0.0002	-
0.9524	1400	0.0001	-
0.9864	1450	0.0001	-

Framework Versions

• Python: 3.10.12
• SetFit: 1.0.3
• Sentence Transformers: 3.0.1
• Transformers: 4.39.0
• PyTorch: 2.4.0+cu121
• Datasets: 2.21.0
• Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}