Forge 0.2.1 Adds 192K-Context Qwen3.8-27B Support on M6
According to the developer’s report, ANEMLL Forge 0.2.1 can run Qwen3.8-27B on the Neural Engine of a 32GB M6 Mac mini with context windows up to 192K tokens. The release adds an INT8 K/V cache and a smaller drafter that shares the target model’s output head, reportedly saving 0.86GB of ANE memory without reducing speed. The developer reports content-dependent decode rates of 10–62 tokens per second, a 96K configuration using about 21.4GB, and an initial full 192K compilation time of roughly two hours.
ANEMLL Forge 0.2.1: Qwen3.8-27B on the M6 Neural Engine now runs up to 192K context (M6 32GB).
New: INT8 K/V cache (half of FP16) and a smaller drafter sharing the target's output head — 0.86 GB less ANE memory, same speed.
Decode depends on content: JSON 35–62 tok/s, code 18–32, prose 10–17. Prefill 260 tok/s at short context, 62 at 192K. 128K–192K use a smaller prefill batch (32 vs 64 rows) to stay within memory budget.
Heads up: full 192K first compile takes ~2h on M6 (one time, cached). `--ctx` exports only the entries you need, so compile is much faster: 16K is ~10 min from download to running server. IMO 96K is the sweet spot for size/speed on 32GB: 123 tok/s prefill, ~27 tok/s decode, ~21.4 GB total memory.
Release notes: https://github.com/Anemll/anemll-forge/blob/main/docs/RELEASE\_NOTES\_v0.2.1.md
Video (Tetris generated over ~150K tokens, no compaction): https://x.com/anemll/status/2111941341819408866
Paper: Three Bit MLP Vector Quantization in Apple Neural Engine — https://github.com/Anemll/anemll-forge/blob/main/docs/papers/anemll-forge-v02.pdf
Cheers, Anemll 🤖
#ANEMLL Forge 0.2.1#Qwen3.8-27B#Apple Neural Engine#M6 Mac mini