BOS token in front of every text chunk
#22
by
websterbei - opened
tokenizer will automatically add BOS token to the front of each text chunk, just checking if this is the intended behavior? (i.e. how it was trained)
tokenizer will automatically add BOS token to the front of each text chunk, just checking if this is the intended behavior? (i.e. how it was trained)