CeLLaTe-tapt_ulmfit_dropout-LR_2e-05

This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1297
• Accuracy: 0.7472

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.426	1.0	21	1.2594	0.7324
1.405	2.0	42	1.2467	0.7365
1.3782	3.0	63	1.2079	0.7407
1.3614	4.0	84	1.2269	0.7358
1.3302	5.0	105	1.2186	0.7397
1.3407	6.0	126	1.2298	0.7372
1.3142	7.0	147	1.1692	0.7422
1.3043	8.0	168	1.1879	0.7477
1.2867	9.0	189	1.1543	0.7456
1.273	10.0	210	1.1645	0.7487
1.2353	11.0	231	1.1340	0.7530
1.2552	12.0	252	1.1548	0.7458
1.2649	13.0	273	1.1796	0.7421
1.2546	14.0	294	1.1249	0.7503
1.2589	15.0	315	1.1655	0.7422
1.2143	16.0	336	1.1486	0.7463
1.2234	17.0	357	1.1934	0.7386
1.2034	18.0	378	1.1431	0.7515
1.2183	19.0	399	1.1518	0.7462
1.2094	20.0	420	1.1478	0.7482
1.1699	21.0	441	1.1046	0.7528
1.175	22.0	462	1.1525	0.7458
1.1684	23.0	483	1.1512	0.7462
1.1566	24.0	504	1.1233	0.7524
1.1808	25.0	525	1.1751	0.7438
1.1737	26.0	546	1.1341	0.7468
1.1754	27.0	567	1.1424	0.7529
1.1624	28.0	588	1.1671	0.7442
1.1412	29.0	609	1.1675	0.7406
1.1699	30.0	630	1.1478	0.7499
1.1309	31.0	651	1.1678	0.7488
1.1596	32.0	672	1.1422	0.7477
1.131	33.0	693	1.1264	0.7516
1.1422	34.0	714	1.1328	0.7488
1.1428	35.0	735	1.1617	0.7428
1.1379	36.0	756	1.1618	0.7471
1.1491	37.0	777	1.1310	0.7514
1.1334	38.0	798	1.1507	0.7465
1.1153	39.0	819	1.1212	0.7506
1.1392	40.0	840	1.0955	0.7595
1.1094	41.0	861	1.1670	0.7438
1.1322	42.0	882	1.1925	0.7410
1.1319	43.0	903	1.1508	0.7459
1.1202	44.0	924	1.1277	0.7511
1.1223	45.0	945	1.1551	0.7502
1.1199	46.0	966	1.1411	0.7466
1.1105	47.0	987	1.1702	0.7452
1.1013	48.0	1008	1.1395	0.7486
1.1339	49.0	1029	1.1975	0.7396
1.1186	50.0	1050	1.1667	0.7469
1.1078	51.0	1071	1.1962	0.7400
1.0944	52.0	1092	1.1565	0.7497
1.1137	53.0	1113	1.1655	0.7460
1.0994	54.0	1134	1.1924	0.7465
1.0878	55.0	1155	1.1722	0.7461
1.0987	56.0	1176	1.1313	0.7520
1.1115	57.0	1197	1.1533	0.7495
1.1148	58.0	1218	1.1544	0.75
1.0921	59.0	1239	1.1420	0.7530
1.0926	60.0	1260	1.1345	0.7531
1.0914	61.0	1281	1.1660	0.7493
1.1003	62.0	1302	1.1202	0.7496
1.1161	63.0	1323	1.1612	0.7472
1.0775	64.0	1344	1.1715	0.7435
1.0852	65.0	1365	1.1480	0.7469
1.102	66.0	1386	1.2095	0.7425
1.0872	67.0	1407	1.1536	0.7489
1.0647	68.0	1428	1.1590	0.7429
1.0972	69.0	1449	1.1403	0.7516
1.0845	70.0	1470	1.1698	0.7478
1.0737	71.0	1491	1.1356	0.7506
1.0774	72.0	1512	1.1739	0.7408
1.0813	73.0	1533	1.1556	0.7480
1.0678	74.0	1554	1.1336	0.7547
1.0767	75.0	1575	1.1454	0.7478
1.0852	76.0	1596	1.1427	0.7432
1.0896	77.0	1617	1.1400	0.7518
1.0798	78.0	1638	1.1641	0.7436
1.0745	79.0	1659	1.1264	0.7520
1.1071	80.0	1680	1.1352	0.7486
1.0665	81.0	1701	1.1544	0.7531
1.0565	82.0	1722	1.1254	0.7551
1.0965	83.0	1743	1.1742	0.7491
1.0715	84.0	1764	1.1154	0.7544
1.0651	85.0	1785	1.1519	0.7457
1.0827	86.0	1806	1.1722	0.7452
1.0904	87.0	1827	1.1895	0.7428
1.0697	88.0	1848	1.1616	0.7461
1.0693	89.0	1869	1.1217	0.7554
1.0733	90.0	1890	1.1338	0.7462
1.0806	91.0	1911	1.1403	0.7512
1.0803	92.0	1932	1.1469	0.7502
1.0726	93.0	1953	1.1279	0.7533
1.082	94.0	1974	1.1460	0.7472
1.0766	95.0	1995	1.1729	0.7411
1.0706	96.0	2016	1.1616	0.7459
1.1028	97.0	2037	1.1496	0.7507
1.0615	98.0	2058	1.1214	0.7521
1.0624	99.0	2079	1.1538	0.7449
1.0889	100.0	2100	1.1297	0.7472

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0