hi i tried llama 3 instruct version both using hugging face and ollama, huggingface version is about 10 times slower, both running on gpu. do you know what is the problem ?

First assumption would be the quantisation that’s being used. Ollama typically defaults to the smallest q4 quantised version of the model, and if you go and download the fp16 version of the model manually, for example, this can skew the results to make it seem like one is faster than the other. You …

well i applied quantization too , here is the code : bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, ) tokenizer = AutoTokenizer.from_pretrained("./llama3", …

I see. My next guess would be that maybe the tokenizer isn’t a fast tokenizer? The fast tokenizer are written in Rust vs. the standard tokenizer which are written in Python so that could be the source of potential slowdown. Maybe print the type of the tokenizer and test if the fast property is set t…

thanks but i printed it’s type and it was transformers.tokenization_utils_fast.PreTrainedTokenizerFast and tokenizer.is_fast returns True. so i don’t think that’s the problem. By the way, how much time does it take for you to generate, for example, a sentence of 15 words? This generated sentence: …

Same issue here. I have not yet tried directly using GGUF files via transformers and use Quanto (instead of BNB) but see the same performance differences. Edit: does not help in (our?) case since transformers can currently only import GGUF. Working directly with llama_cpp seems to be essentially t…

i used a GGUF file for hugging face but it attempts to convert it back to normal format so nothing changed in terms of performance, it also converts quantized gguf files back to dequantized version. on a different note, major problem of ollama was that it could not handle multiple requests and inst…

I did not measure at all but it feels as if the Python bindings for cpp llama are essentially the same speed as ollama. Integrating HF transformers or raw inputs is pretty tricky though so I abandoned that for now. GGUF in HF - yes, that’s what I meant with “just import”. Sorry for being unclear. M…

Hi, I have observed the same as you. I think it stems from certain optimization going in the background of ollama or llama.cpp. In my case I am running the models on A10 GPU 24GB running Bfloat16 versions and they are relatively fast BUT when my co-worker in the research group uses ollama on laptop…

i switched to VLLM and used a quantized model, it is as fast as ollama and can handle multiple requests this is my code in case anybody is struggling with the speed class Vllm_Openai(Base_QuestionAnswerer): def __init__(self, re_run_server=False): super().__init__() print("scr…

Llama3 so much slow compared to ollama

🤗Transformers

John6666 February 28, 2025, 10:04am 16

If you don’t specify anything in particular, it seems that when you use Llama 3 with Ollama, Llama 3 Instruct is used instead of Llama 3 (Base). This is probably the same for other models as well. If you want to get output similar to Ollama, use Instruct.

Topic		Replies	Views
Ollama + Llama-3.2-11b-vision-uncensored like 22 Beginners	1	1868	December 10, 2024
Llama 3 Instruct taking too long all of a sudden 🤗Transformers	1	1550	June 9, 2024
Llama 3 performance is 4 mins. can get it in seconds? Models	2	703	March 24, 2025
How to Configure LLaMA-3:8B on HuggingFace to Generate Responses Similar to Ollama? 🤗Transformers	7	1974	October 7, 2024
500 Internal Server Error with Ollama Beginners	4	4395	April 11, 2026

Llama3 so much slow compared to ollama

Related topics