单卡P40调优Ornith 1.5长上下文推理
开发者 sargetun123 称，其通过自定义 llama.cpp 分支、Q4_0 权重、q8_0 KV 缓存和 MTP 草稿，在单张 24GB Tesla P40 上运行 Ornith 1.5 35B-A3B。该配置支持完整 262k 上下文；未启用 MTP 时，短提示解码速度为 60.8 tokens/s，100k 上下文下降至 29.1 tokens/s，而启用 MTP 的实际服务器处理 101k 提示时生成速度为 35–39 tokens/s。开发者还称，关闭 ECC 可释放 1.5GB 显存，并使解码速度提升约 9%。
该报告详细记录了在较老消费级硬件上运行 35B MoE 模型和长上下文的调优方法，以及相关显存、缓存与内核调整对吞吐表现的影响。
i Got a bit obsessed with my P40 this week trying to max speed on Ornith 1.5 35B-A3B (qwen3.5 MoE arch) without going below Q4 weights or q8 KV. Shrinking quants to win a benchmark has not really been useful to me so ai wanted a logical middle ground that was still “fast”
Setup: one Tesla P40 24GB on PCIe x1 (yes, x1), Windows, my own llama.cpp fork. Q4_0 built from the official BF16, MTP draft (n=2), full 262k context, q8_0 KV.
What actually moved it:
- ECC off +1.5 GB VRAM and ~9% decode. - q8_0 KV, not q4_0. On Pascal q4 KV falls apart with depth (30.6 vs 42.2 t/s at 32k). - Check your MTP head. The first upload of this model, and quants built from it, have an untrained MTP head. `blk.40.nextn.shared_head_norm` mean ~1.02 = untrained, ~2.93 = trained. Draft acceptance went from ~45% to 60-85%. - int8 dp4a for the KQ half of the flash-attention tile kernel. Pascal has no tensor cores and crippled fp16, but dp4a is fast. Prefill +26% at 32k, +41% at 100k. - MTP prefill window. The MTP head is a full-attention layer, so on long prompts it costs ~13% prefill. Letting it read only the last 8k of the prompt gets that back (121k prompt: 327 -> 373 t/s). Output doesn't change, the main model still verifies every draft.