Ling-3.0-tiny UBTree DFlash draft runs on stock llama.cpp, claims up to 61% code speedup
A Reddit user, /u/Barni275, says he extracted the pure DFlash portion of the newly published UBTree DFlash draft for Ling-3.0-tiny, dropping 7 ubtree_selector tensors, so it can run with stock mainline llama.cpp through the dflash drafter. In his own small tests, prose generation rose from 78.8 to 84.8 tok/s (+8%) and Python code generation from 79.0 to 127.2 tok/s (+61%), results he described as moderate and not independently verified. He made small patches to the llama.cpp GGUF converter, published the converted model and procedure on Hugging Face, and speculated that native UBTree support in llama.cpp would bring larger gains.
The post shows a workaround that lets an unsupported speculative-decoding draft run on mainline llama.cpp, though the reported speedups come from the author's own small-scale tests.
I use Ling-3.0-tiny on my LLM server for text processing. I convert a lot of PDF documents to markdown for downstream LLM processing, and I built my custom harness for it. It uses ONNX model to parse layout, extracts text, and then runs LLM processing stage to remove extra whitespaces, join lines and so on. This tiny model is quite capable for it, but it is slow for it's size.
Today I saw post about newly published UBTree DFlash for Ling-3.0-tiny, https://www.reddit.com/r/LocalLLaMA/s/yllgDx4pAk . I was very excited, but then realized that, unfortunately, no inference engine supports it.
I was very impatient, and extracted pure DFlash part from this UBTree (dropping 7 ubtree_selector.* tensors). This part can be run with stock mainline llama.cpp using dflash drafter:
llama-server \ -m Ling-3.0-tiny-Q4_K_M.gguf \ -md Ling-3.0-tiny-UBTree-dflash-bf16.gguf \ --spec-type draft-dflash --spec-draft-n-max 6 \ -fa on -c 32768 Results are moderate. I run only small tests, and get the following results.
| Workload | Baseline (no spec-draft) | + DFlash draft | Gain | Acceptance / mean len |
|---|---|---|---|---|
| Prose generation | 78.8 tok/s | 84.8 tok/s | +8% | 0.28 / 2.7 |
| Code generation (Python) | 79.0 tok/s | 127.2 tok/s | +61% | 0.53 / 4.2 |
To perform conversion, I made tiny patches for llama.cpp GGUF converter (described on HF model card). Everything other is stock.
If real UBTree support will be available in llama.cpp, it should give more speedup, for sure.
Converted model and conversion procedure are on HF: yury-kapoyko/Ling-3.0-tiny-UBTree-dflash-GGUF · Hugging Face
submitted by /u/Barni275 [link] [comments]