How to Load Split GGUF Files and When to Merge Them

Check the three files before you start

Here is a complete example. Each numbered file is a shard, meaning it holds part of the weights for the same model. The prefix identifies the model and quantization, and of-00003 tells you how many files to expect.

example-Q4_K_M-00001-of-00003.gguf
example-Q4_K_M-00002-of-00003.gguf
example-Q4_K_M-00003-of-00003.gguf

If you have files 00001 and 00003, download 00002 from the same model repository, revision, and quantization. A file ending in .gguf.part or .gguf.crdownload may still be downloading. Check the publisher’s expected file sizes or hashes before using a file you suspect was cut short.

On Windows, open PowerShell in the download folder. This check prints the names that are missing from the example set. Replace example-Q4_K_M with the actual prefix. No output means all three expected filenames exist. File integrity is a separate check.

1..3 | ForEach-Object {
    $name = 'example-Q4_K_M-{0:00000}-of-00003.gguf' -f $_
    if (-not (Test-Path -LiteralPath $name -PathType Leaf)) { $name }
}

When the middle shard is absent, the check prints example-Q4_K_M-00002-of-00003.gguf. Download that file, then rerun the check. A valid header on the first file tells you about that file alone. You can inspect it with the GGUF metadata reader.

Load the first shard in llama.cpp

Pass shard 00001 to llama-cli. For this numbered format, llama.cpp uses it to find the rest of the set in the same folder. The examples assume the executables are on your PATH. If an executable is in your current PowerShell folder, use its local path, such as .\llama-cli.exe.

llama-cli -m ./example-Q4_K_M-00001-of-00003.gguf -p "Name one planet."

If the log says it failed to load a GGUF split, start with the path named in the error. Check that the file exists and is readable. If the path exists, compare its size or hash with the publisher’s. If the loader says to start with the first split, change -m to point at 00001.

Import the complete set into Ollama

With a version that supports split GGUF imports, put a Modelfile next to the shards. Give FROM a wildcard that matches this set alone. Keep other models and quantizations in separate folders.

FROM ./example-Q4_K_M-*-of-00003.gguf

Run these from the folder containing the Modelfile. Wait for the create command to succeed before running the model.

ollama create split-gguf-demo -f ./Modelfile
ollama run split-gguf-demo "Name one planet."

See the Windows Ollama import guide for the full setup and path checks. If your Ollama version cannot import split files, check the documentation for that release or use the merge workflow below.

Merge when your app needs one file

Keep the shards if your runtime already opens them. Merge when you have an app or workflow that expects a single GGUF. Allow room for a new output file roughly the size of the combined shards and choose a fresh output filename.

llama-gguf-split --merge ./example-Q4_K_M-00001-of-00003.gguf ./example-Q4_K_M-merged.gguf

Give the tool the first shard and the output path. Keep the originals until you have loaded the merged file in the intended app and generated a reply. Merging keeps the same model weights. If the model is too large for your GPU, check your full model and context budget in the VRAM calculator.

The filenames and commands use an example set. Follow the linked documentation for the versions you have installed.

Sources and further reading

• llama.cpp model loader
• llama.cpp GGUF merge tool
• Ollama GGUF import documentation