ExLlama’s “Expected Scalar Type Half” Error

The model loads, the first forward pass raises, and the message names two dtypes without saying which tensor is which. It is a real dtype mismatch and it has a small number of possible entry points.

The string and its mirror image

RuntimeError: expected scalar type Half but found Float

This is PyTorch’s own message, raised by the ATen dispatcher, not by ExLlama. It means an operation was resolved to its float16 implementation and one of its arguments arrived as float32.

You will also meet the reverse, expected scalar type Float but found Half, and the distinction is worth holding on to because it points the opposite way. The first says the kernel is Half and your input is Float — the usual case with ExLlama, where the quantized layers are fp16. The second says the surrounding graph is fp32 and something handed it a Half tensor, which is what happens when a quantized module is dropped into an otherwise full-precision model.

Neither message names the offending tensor. The frame directly above the raise does, so read the traceback from the bottom up until you reach the last line that belongs to your own code or to a wrapper: that is the boundary the wrong dtype crossed.

Why there is no float32 path at all

PyTorch will happily promote dtypes in ordinary arithmetic. It does not do so here, and the reason is structural rather than a design preference.

ExLlama’s quantized matrix multiply is a custom CUDA kernel. It reads 4-bit weights packed into 32-bit integers, unpacks and dequantizes them into registers, and multiplies against the activation vector — and it does that in half precision throughout, because half precision is what makes the arithmetic fit the tensor cores that give the kernel its speed. The kernel is templated on fp16 and never instantiated for fp32. When the dispatcher looks for an implementation matching a Float argument, there is nothing to dispatch to, so it raises instead of silently doing something slower.

That is the useful thing to know about this error: it is not recoverable by configuration, because there is no configuration that creates a kernel that was never compiled. Every genuine fix is a cast at the point where the wrong dtype entered. Newer members of the family add bfloat16 support in places, which changes which dtypes are acceptable but not the principle — a Float tensor still has no kernel.

Four places the wrong dtype enters

• A LoRA adapter saved in float32. The most common cause by some distance. Training frequently keeps adapter weights in fp32 for numerical stability, and the saved adapter_model.safetensors carries that dtype. Applying it over an fp16 quantized base puts an fp32 tensor directly into the forward pass. The tell is that the base model alone runs fine and the error appears the moment the adapter is attached.
• Inputs built without a dtype. Embeddings, position tensors or a manually constructed inputs_embeds default to float32 because that is torch’s default. A tokenizer’s integer ids are fine; anything you build with torch.tensor(...) or torch.zeros(...) and hand to the model is not.
• A config that says float32. A torch_dtype of float32 in config.json, or a loader called without an explicit dtype, materialises the non-quantized parts of the model — embeddings, norms, the language-model head — in fp32 while the quantized layers stay fp16. The mismatch then happens at the seam between them.
• An autocast or precision switch in a wrapper. Some front ends expose a flag that disables half precision globally. It is intended for CPU or for hardware without fp16 support, and turning it on with an ExLlama loader selected produces this error immediately.

Finding and fixing the boundary

Start by printing dtypes rather than guessing, which takes one command and eliminates three of the four causes:

import torch
from safetensors.torch import safe_open

with safe_open("adapter_model.safetensors", framework="pt") as f:
    for k in list(f.keys())[:4]:
        print(k, f.get_tensor(k).dtype)

# torch.float32  -> this is your cause

Then apply the cast at whichever boundary it turned out to be:

1. Adapter in fp32. Convert it once and save the converted copy rather than casting on every load: {k: v.to(torch.float16) for k, v in state.items()}. Casting at load time also works and costs a moment of extra memory.
2. Hand-built inputs. Construct them with the dtype explicitly — torch.zeros(n, d, dtype=torch.float16, device="cuda") — rather than casting afterwards, so a later refactor cannot reintroduce it.
3. Config or loader. Pass torch_dtype=torch.float16 explicitly at load. Do not rely on the config file; the checkpoints on public hubs disagree with each other about this field.
4. A precision flag in a front end. Turn it off and reload. If you needed it for another reason, you need a different loader, not a different flag.

One thing to resist: wrapping the call in torch.autocast to make the error disappear. Autocast will insert casts around operations it knows about, which can indeed silence this, but it does so by inserting conversions you did not choose in places you did not look, and the next dtype error will be further from its cause. Cast at the boundary you identified and leave the rest alone.

A second thing to resist: converting the whole model to fp32 so that everything matches. It resolves the mismatch by making the quantized layers unusable — you would be running a 4-bit checkpoint through a path that dequantizes everything, using more memory than the unquantized model and running slower than either. If fp32 is genuinely required for some reason, load the unquantized weights and do not use ExLlama at all.

One more case that presents identically and is not a dtype problem at all. If the model is split across two GPUs, or partially offloaded to CPU, a tensor can be the right dtype and the wrong device, and some code paths surface that as a dtype complaint from whichever operation fails first rather than as a device error. Print tensor.device alongside tensor.dtype while you are debugging; it costs one extra field and eliminates a whole afternoon of chasing casts that were never wrong.

If the error arrived alongside a warning that the ExLlama kernels were not installed at all, deal with that first — the missing-kernel warning means you are on a fallback path with different dtype behaviour, and fixing the dtype under a fallback kernel fixes the wrong problem.