How to run inference without vLLM? (e.g., with standard transformers)
Hi,
I'm really interested in using the LightOnOCR-1B-1025 model for a project. I've read the model card, but all the provided examples for inference rely on setting up a vLLM server.
Is it possible to run this model for inference directly using the standard Hugging Face transformers library (for example, by loading it with AutoModel or a specific model class)? The model card mentions "Transformers integration is coming soon for training," but I wasn't sure about inference.
If anyone has a working code snippet for running a single-image inference without vLLM, I would greatly appreciate it!
Thanks!
Hey there!
I see that the transformers integration is on its way: https://github.com/huggingface/transformers/pull/41621
When merged it would be possible to use it with standard transformers 🤗
Hello,
now we have added an inference notebook using transformers that runs on Colab : https://huggingface.co/lightonai/LightOnOCR-1B-1025/blob/main/notebook.ipynb