quantized models don't have .to

ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
pull/450/head
AlexanderLavelle 1 year ago committed by GitHub
parent 8f91a7dd9b
commit 78b1e83c3f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

@ -181,7 +181,7 @@ class HuggingfaceLLM(AbstractLLM):
quantization_config=bnb_config, quantization_config=bnb_config,
*args, *args,
**kwargs, **kwargs,
).to(self.device) )
else: else:
self.model = AutoModelForCausalLM.from_pretrained( self.model = AutoModelForCausalLM.from_pretrained(
self.model_id, *args, **kwargs self.model_id, *args, **kwargs

Loading…
Cancel
Save