llama.cpp

llama.cpp 是保加利亚程序员 Georgi Gerganov 用 C/C++ 语言实现的大型语言模型本地推理引擎，使用 MIT 许可，首个正式版本发布于 2023 年；同年作者基于该项目底层提炼出跨平台的 ggml 张量库，随后庞大的开源社区作为主要推动力并主导了其对多平台硬件加速的后续开发。该项目旨在让更多人在消费级硬件上使用和实验 LLM。

ollama 和 llama.cpp 相比，前者是成品车，后者是高性能引擎。

Quick Start

在 https://github.com/ggml-org/llama.cpp/releases 下载压缩包后解压

允许打开该目录下所有二进制文件 xattr -d com.apple.quarantine llama-bxxx/*

设置登录 https://huggingface.co 并手动下载指定模型 Qwen/Qwen3.5-9B

shell

pip install -U "huggingface_hub[cli]"
# Create a token from https://huggingface.co/settings/tokens
hf auth login

mkdir -p ~/models/
hf download Qwen/Qwen3.5-9B --local-dir ~/models/

运行指定模型 Qwen/Qwen3.5-9B

shell

# Use a local model file
llama-cli -m path/to/my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf Qwen/Qwen3.5-9B

# Launch OpenAI-compatible API server
llama-server -hf Qwen/Qwen3.5-9B

Reuse ollama downloaded models

先用 ollama CLI 下载模型，通过尺寸和文件时间戳定位到对应模型文件

shell

ollama pull deepseek-r1:14b

du -sh $HOME/.ollama/models/blobs/*

498M    $HOME/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa
4.9G    $HOME/.ollama/models/blobs/sha256-a3de86cd1c132c822487ededd47a324c50491393e6565cd14bafa40d0b8e686f
6.1G    $HOME/.ollama/models/blobs/sha256-dec52a44569a2a25341c4e4d3fee25846eed4f6f0b936278e3a3c900bb99d37c
8.4G    $HOME/.ollama/models/blobs/sha256-6e9f90f02bb3b39b59e81916e8cfce9deb45aeaeb9a54a5be4414486b907dc1e
 16G    $HOME/.ollama/models/blobs/sha256-83c54730a5fea8a0958598c01617c1419c431e93b33bacf980b49a420c798926
 18G    $HOME/.ollama/models/blobs/sha256-f36b668ebcd329357fac22db35f6414a1c9309307f33d08fe217bbf84b0496cc

ollama list
NAME                  ID              SIZE      MODIFIED
deepseek-r1:14b       c333b7232bdb    9.0 GB    About an hour ago
qwen3.6:27b-q4_K_M    a50eda8ed977    17 GB     9 hours ago
qwen3:8b              500a1f067a9f    5.2 GB    2 weeks ago
qwen3:0.6b            7df6b6e09427    522 MB    3 weeks ago
qwen3.5:latest        6488c96fa5fa    6.6 GB    3 weeks ago
codellama:34b         685be00e1532    19 GB     3 weeks ago

新建一个软连接，名字更易读

shell

mkdir $HOME/models

ln -s $HOME/.ollama/models/blobs/sha256-6e9f90f02bb3b39b59e81916e8cfce9deb45aeaeb9a54a5be4414486b907dc1e \
$HOME/models/deepseek-r1_14b

命令行测试

shell

./llama-cli \
-m $HOME/models/deepseek-r1_14b \
-p "answer shortly: 1 + 1 = ?" \
-n 512

作为 HTTP 服务启动

shell

./llama-server \
-m $HOME/models/deepseek-r1_14b \
--host 0.0.0.0 \
--port 8080 \
-c 4096 \
-ngl 99

测试 HTTP 接口

shell

curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "ollama/deepseek-r1:14b",
  "messages": [
    {
    "role": "user",
    "content": "answer shortly: 1 + 1 = ?"
    }
  ]
}'

llama.cpp ​

Quick Start ​

Reuse ollama downloaded models ​

llama.cpp

Quick Start

Reuse ollama downloaded models