Inference Endpoints creation

iarbel
Sun, 14 Jan 2024 11:05:39 +0000

+1 for that. text-generation-inference supports loading a PeftModel directly with the adapter, without the need to merge weights before. I’d expect the Inference Endpoints to support this option, however I couldn’t find a way to do it. Specifically for quantized models this is very important, as it’s not possible to merge LoRA weights into the base

Read full topic