llama.cpp XMX matrix engine refactor nearly doubles B70 speculative decoding speeds
A Reddit user on r/LocalLLaMA reports that llama.cpp pull request #29864, which refactors Intel XMX matrix engines and is attributed to budiony, made a huge improvement to speculative decoding speeds on an Intel B70 GPU running SYCL. In before/after runs on Muse Glimmer Q6_K (30B) with DFlash2 across nine prompts, throughput rose from 38.0 to 73.0 tok/s on code_python and aggregate wall time fell from 46.86s to 26.68s. The poster notes the acceptance rate dropped from 0.7845 to 0.363 because cheaper token verification allowed n-max to be increased; the numbers are a single, unverified user report.
If the reported figures hold, llama.cpp users running Intel GPUs under SYCL could see substantially faster speculative decoding, though the benchmarks come from one user and have not been independently verified.
| This made a huge improvement to speculate decoding speeds on my B70 running SYCL. Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2. NOTE: Due to the significantly reduced cost of token verification, the n-max was increased which is why the acceptance rate is lower. Before: ``` code_python pred= 192 draft= 127 acc= 106 rate=0.835 tok/s=38.0 code_cpp pred= 192 draft= 175 acc= 113 rate=0.646 tok/s=36.2 explain_concept pred= 192 draft= 135 acc= 116 rate=0.859 tok/s=41.2 summarize pred= 188 draft= 170 acc= 142 rate=0.835 tok/s=50.1 qa_factual pred= 192 draft= 145 acc= 107 rate=0.738 tok/s=37.0 translation pred= 192 draft= 175 acc= 133 rate=0.760 tok/s=45.1 creative_short pred= 192 draft= 130 acc= 105 rate=0.808 tok/s=37.1 stepwise_math pred= 192 draft= 167 acc= 145 rate=0.868 tok/s=53.2 long_code_review pred= 192 draft= 159 acc= 118 rate=0.742 tok/s=38.4 Aggregate: { "n_requests": 9, "total_predicted": 1724, "total_draft": 1383, "total_draft_accepted": 1085, "aggregate_accept_rate": 0.7845, "wall_s_total": 46.86 } ``` After: ``` code_python pred= 192 draft= 324 acc= 143 rate=0.441 tok/s=73.0 code_cpp pred= 192 draft= 424 acc= 140 rate=0.330 tok/s=68.5 explain_concept pred= 192 draft= 330 acc= 141 rate=0.427 tok/s=70.7 summarize pred= 192 draft= 416 acc= 151 rate=0.363 tok/s=85.4 qa_factual pred= 192 draft= 399 acc= 133 rate=0.333 tok/s=60.3 translation pred= 192 draft= 356 acc= 152 rate=0.427 tok/s=87.7 creative_short pred= 192 draft= 379 acc= 125 rate=0.330 tok/s=53.6 stepwise_math pred= 192 draft= 462 acc= 148 rate=0.320 tok/s=77.6 long_code_review pred= 192 draft= 414 acc= 139 rate=0.336 tok/s=64.2 Aggregate: { "n_requests": 9, "total_predicted": 1728, "total_draft": 3504, "total_draft_accepted": 1272, "aggregate_accept_rate": 0.363, "wall_s_total": 26.68 } ``` submitted by /u/nickm_27 [link] [comments] |
#llama.cpp#ggml-org#Intel#Intel B70 GPU#SYCL#Muse Glimmer#DFlash2