Practical guide

How to Import a GGUF Model into Ollama on Windows

If you already have a GGUF on disk, point a Modelfile at it, run ollama create, then open the new model by name. Here’s the complete Windows workflow, including how to tell which step failed.

By InfoIsland Editorial · · Documentation reviewed

A blue data block sliding into a desktop computer’s docking tray.
Original AI-generated editorial illustration; not a product screenshot.

You need Ollama running on Windows and a complete GGUF for a model architecture your version supports. For the first run, use a small text instruct model so you can separate an import problem from a memory problem. If you’re unsure what you downloaded, open it in the GGUF metadata reader and look for general.architecture. The reader can identify header fields, while Ollama will determine whether it can actually load the model.

1. Put the Modelfile beside your GGUF

Make a working folder such as C:\Models\gguf-demo. For this example, place your single GGUF there and use the name model.gguf in the Modelfile. You can keep the original filename and edit the line to match instead.

Download the example Modelfile. Save it in the same folder as the GGUF. The download may arrive as Modelfile.txt; either filename works as long as you pass the actual name to -f.

FROM ./model.gguf
PARAMETER num_ctx 2048

The first line picks the local weights. The second sets a modest starting context for this example. Begin with the default chat behavior Ollama picks for your model. If the model card specifies a custom template, review it after you’ve confirmed the basic import works.

2. Create the model and get a reply

Open PowerShell. In this example the downloaded file is named Modelfile. Adjust the folder and filenames as needed. Both Test-Path calls should print True before you continue.

Set-Location 'C:\Models\gguf-demo'
ollama --version
Test-Path -LiteralPath '.\model.gguf' -PathType Leaf
Test-Path -LiteralPath '.\Modelfile' -PathType Leaf
ollama create local-gguf-demo -f .\Modelfile

When the create command finishes successfully, run the new local name:

ollama show local-gguf-demo
ollama run local-gguf-demo "Give me one sentence about the Moon."

ollama show confirms the model is registered. A generated reply confirms it can run with this prompt and these settings. If it takes a long time or falls back to the CPU, check where Ollama loaded it before importing it again.

What if the download has several GGUF files?

A suffix like -00001-of-00003.gguf means you need all three parts from the same release and quantization. Keep the original names together in one folder. Ollama’s current import docs show a FROM wildcard for split files. For example, use FROM ./model-q4-*.gguf if that prefix matches exactly your shard set. Avoid a broad *.gguf pattern in a folder containing different models or variants. If an older build rejects the syntax, check the docs for that build before changing how the files are stored.

Use the split GGUF guide to check for a missing part, load all three files, or merge them for an app that requires one file.

If it fails, start at the failing step

  • PowerShell can’t find Ollama. Open a fresh terminal after installation and check that the desktop app is running.
  • Create can’t find a file. Check both filenames with Get-ChildItem -Name. Windows Explorer may hide extensions. Double-check the path in FROM as well as the path after -f.
  • Create reports an unsupported architecture or invalid model. Verify that you have the model weights and every required shard. A successful metadata read doesn’t validate the tensor data. Check the publisher’s file sizes or checksums when available.
  • Create succeeds but the first prompt fails. Check the runtime error, available memory, and the VRAM estimate. Keep the same model while changing one setting so you can tell what helped.

This walkthrough follows the linked Ollama documentation. The commands are a reproducible example for your own file and Ollama version.

Sources and further reading