variant-tapt_grouped_llrd_grouped_txt-LR_2e-05

This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on the Mardiyyah/TAPT-PDBE-V1 dataset. It achieves the following results on the evaluation set:

• Loss: 1.0877
• Accuracy: 0.7598

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.2	1.0	9	1.2315	0.7372
1.1988	2.0	18	1.1789	0.7469
1.1858	3.0	27	1.2256	0.7414
1.15	4.0	36	1.1462	0.7498
1.1719	5.0	45	1.1476	0.7490
1.1317	6.0	54	1.1450	0.7536
1.1332	7.0	63	1.1061	0.7563
1.1144	8.0	72	1.1648	0.7470
1.1076	9.0	81	1.1224	0.7502
1.1046	10.0	90	1.1075	0.7616
1.093	11.0	99	1.0575	0.7647
1.0771	12.0	108	1.1003	0.7601
1.0731	13.0	117	1.1110	0.7552
1.0761	14.0	126	1.1021	0.7583
1.0856	15.0	135	1.0896	0.7572
1.0586	16.0	144	1.0765	0.7636
1.0448	17.0	153	1.0927	0.7578
1.0586	18.0	162	1.0717	0.7628
1.0551	19.0	171	1.1044	0.7570
1.0391	20.0	180	1.0854	0.7602
1.0377	21.0	189	1.0820	0.7625
1.0389	22.0	198	1.0733	0.7612
1.0328	23.0	207	1.0721	0.7609
1.0146	24.0	216	1.0660	0.7680
1.0192	25.0	225	1.0863	0.7620
1.0272	26.0	234	1.0792	0.7582
1.0286	27.0	243	1.0847	0.7597
1.035	28.0	252	1.0971	0.7550
1.0208	29.0	261	1.0522	0.7701
1.0153	30.0	270	1.0948	0.7577
1.0149	31.0	279	1.0692	0.7633
1.012	32.0	288	1.0748	0.7633
1.0186	33.0	297	1.0568	0.7683
0.9948	34.0	306	1.0866	0.7630
1.0128	35.0	315	1.0747	0.7618
1.0022	36.0	324	1.1023	0.7563
1.0071	37.0	333	1.0571	0.7684
1.0116	38.0	342	1.0667	0.7603
1.0066	39.0	351	1.0694	0.7583
0.9944	40.0	360	1.0528	0.7638
1.0086	41.0	369	1.0723	0.7634
0.9846	42.0	378	1.0724	0.7659
0.9723	43.0	387	1.0760	0.7626
1.0	44.0	396	1.1026	0.7603
0.9822	45.0	405	1.0688	0.7651
0.9952	46.0	414	1.0800	0.7616
0.9873	47.0	423	1.0989	0.7606
0.98	48.0	432	1.0886	0.7594
0.9876	49.0	441	1.0835	0.7587
0.9832	50.0	450	1.0555	0.7656
0.973	51.0	459	1.1021	0.7583
0.9726	52.0	468	1.0678	0.7652
0.9592	53.0	477	1.0955	0.7573
0.975	54.0	486	1.0890	0.7629
0.9886	55.0	495	1.1057	0.7607
0.9692	56.0	504	1.0802	0.7556
0.9715	57.0	513	1.1034	0.7590
0.9622	58.0	522	1.0644	0.7637
0.9886	59.0	531	1.0552	0.7629
0.9837	60.0	540	1.0870	0.7614
0.9721	61.0	549	1.0611	0.7643
0.9615	62.0	558	1.0561	0.7671
0.9564	63.0	567	1.0735	0.7600
0.9581	64.0	576	1.0920	0.7594
0.9426	65.0	585	1.0937	0.7618
0.9754	66.0	594	1.1217	0.7581
0.9415	67.0	603	1.0475	0.7629
0.9427	68.0	612	1.0621	0.7652
0.9481	69.0	621	1.0302	0.7711
0.961	70.0	630	1.1056	0.7551
0.9573	71.0	639	1.0640	0.7626
0.9458	72.0	648	1.1082	0.7548
0.9418	73.0	657	1.0844	0.7581
0.9326	74.0	666	1.0544	0.7656
0.9395	75.0	675	1.0929	0.7617
0.9407	76.0	684	1.0888	0.7651
0.9541	77.0	693	1.0825	0.7600
0.9391	78.0	702	1.0542	0.7630
0.9215	79.0	711	1.0896	0.7591
0.9613	80.0	720	1.0846	0.7593
0.9514	81.0	729	1.1061	0.7564
0.9346	82.0	738	1.0376	0.7656
0.9444	83.0	747	1.0703	0.7669
0.9337	84.0	756	1.0705	0.7597
0.9456	85.0	765	1.0861	0.7583
0.9363	86.0	774	1.0777	0.7613
0.926	87.0	783	1.0810	0.7583
0.9303	88.0	792	1.0740	0.7597
0.9668	88.9143	800	1.0833	0.7599

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0