Ling-3.0-tiny DFlash 提速 61%
Reddit 用户 /u/Barni275 称，他从新发布的 Ling-3.0-tiny UBTree DFlash 草稿中提取出纯 DFlash 部分，去掉 7 个 ubtree_selector 张量，使其可通过 dflash drafter 在原版 llama.cpp 上运行。他自测的小规模结果显示，散文生成从 78.8 提升至 84.8 tok/s（+8%），Python 代码生成从 79.0 提升至 127.2 tok/s（+61%）；他称结果“中等”，且未经独立验证。他为 llama.cpp 的 GGUF 转换器做了小幅修改，并将转换后的模型与流程发布在 Hugging Face，同时推测若 llama.cpp 原生支持 UBTree 会带来更大提速。
该帖提供了一种变通方案，让缺乏引擎支持的投机解码草稿模型在原版 llama.cpp 上运行，但提速数据来自作者的小规模自测。
I use Ling-3.0-tiny on my LLM server for text processing. I convert a lot of PDF documents to markdown for downstream LLM processing, and I built my custom harness for it. It uses ONNX model to parse layout, extracts text, and then runs LLM processing stage to remove extra whitespaces, join lines and so on. This tiny model is quite capable for it, but it is slow for it's size.
Today I saw post about newly published UBTree DFlash for Ling-3.0-tiny, https://www.reddit.com/r/LocalLLaMA/s/yllgDx4pAk . I was very excited, but then realized that, unfortunately, no inference engine supports it.
I was very impatient, and extracted pure DFlash part from this UBTree (dropping 7 ubtree_selector.* tensors). This part can be run with stock mainline llama.cpp using dflash drafter:
llama-server \ -m Ling-3.0-tiny-Q4_K_M.gguf \ -md Ling-3.0-tiny-UBTree-dflash-bf16.gguf \ --spec-type draft-dflash --spec-draft-n-max 6 \ -fa on -c 32768 Results are moderate. I run only small tests, and get the following results.
| Workload | Baseline (no spec-draft) | + DFlash draft | Gain | Acceptance / mean len |
|---|---|---|---|---|
| Prose generation | 78.8 tok/s | 84.8 tok/s | +8% | 0.28 / 2.7 |
| Code generation (Python) | 79.0 tok/s | 127.2 tok/s | +61% | 0.53 / 4.2 |
To perform conversion, I made tiny patches for llama.cpp GGUF converter (described on HF model card). Everything other is stock.
If real UBTree support will be available in llama.cpp, it should give more speedup, for sure.
Converted model and conversion procedure are on HF: yury-kapoyko/Ling-3.0-tiny-UBTree-dflash-GGUF · Hugging Face
submitted by /u/Barni275 [link] [comments]