Ornith 1.5 tuned for long-context inference on one Tesla P40
Developer sargetun123 reported running Ornith 1.5 35B-A3B on a single 24GB Tesla P40 using a custom llama.cpp fork, Q4_0 weights, q8_0 KV cache and MTP drafting. The reported setup supports a full 262k context and reached about 60.8 tokens/s decode without MTP at zero context and 29.1 tokens/s at 100k context, while a real server with MTP generated 35–39 tokens/s for a 101k-token prompt. The developer also reported that disabling ECC freed 1.5GB of VRAM and improved decoding by about 9%.
The report provides a detailed, source-attributed case study for running a 35B MoE model with long context on older consumer hardware, including the memory and kernel changes associated with its throughput results.
i Got a bit obsessed with my P40 this week trying to max speed on Ornith 1.5 35B-A3B (qwen3.5 MoE arch) without going below Q4 weights or q8 KV. Shrinking quants to win a benchmark has not really been useful to me so ai wanted a logical middle ground that was still “fast”
Setup: one Tesla P40 24GB on PCIe x1 (yes, x1), Windows, my own llama.cpp fork. Q4_0 built from the official BF16, MTP draft (n=2), full 262k context, q8_0 KV.
What actually moved it:
- ECC off +1.5 GB VRAM and ~9% decode. - q8_0 KV, not q4_0. On Pascal q4 KV falls apart with depth (30.6 vs 42.2 t/s at 32k). - Check your MTP head. The first upload of this model, and quants built from it, have an untrained MTP head. `blk.40.nextn.shared_head_norm` mean ~1.02 = untrained, ~2.93 = trained. Draft acceptance went from ~45% to 60-85%. - int8 dp4a for the KQ half of the flash-attention tile kernel. Pascal has no tensor cores and crippled fp16, but dp4a is fast. Prefill +26% at 32k, +41% at 100k. - MTP prefill window. The MTP head is a full-attention layer, so on long prompts it costs ~13% prefill. Letting it read only the last 8k of the prompt gets that back (121k prompt: 327 -> 373 t/s). Output doesn't change, the main model still verifies every draft.