Use the right tokenizer
#10
by Nick088 - opened
As its a finetune of the Google Flan T5 Small model, the tokenizer its the same, so there isn't really a reason to use the base model repo tokenizer when you can just use the one of your own model as its like a duplicate one
Good catch, ty!
roborovski changed pull request status to merged