Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Strata

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 73

  1. Reddit test says slower Flash-Next tokens improved agent work

    In an r/LocalLLaMA post, one user reports that Flash-Next running in Strata produced tokens quickly with IQ3_XXS and experimental speed optimizations, but performed poorly on real agent tasks because it overthought and remained stuck in self-doubt. The author says IQ3_S without experimental optimizations was slower in tokens per second but completed real-world work faster, with more meaningful output, less context consumption and reduced compaction.

    r/LocalLLaMA · 🔥 0
  2. Reddit user reports Strata raises Qwen decode speed to 150 tok/s

    A Reddit user reported that switching from llama-server to Strata raised Qwen 3.8 Flash Next decode speed from 70–80 tok/s to 150 tok/s. The user listed the setup as “1 pro 6000,” two RTX 3090s, 256 GB DDR4 and a 160k context window. The same user reported 70 tok/s decode for DeepSeek V4.1 Flash at 2.25-bit, with an Unsloth 6-bit quant, and said they did not know what implementation method Strata used.

    r/LocalLLaMA · 🔥 0
  3. Strata 0.2.10 targets long-context inference on 8GB GPUs

    Open-source local AI tool Strata 0.2.10 adds memory-aware allocation that can reduce the GPU-resident KV cache and offload some cache data to system memory or disk. The developers say the release can launch Qwen3.8IQ3_S with a 128K context on an 8GB device and make DFlash2 the default speculative decoder for single-request workloads. The update also introduces the All-cards Split Engine for coordinated multi-GPU processing, the ISTA-DAS Coder programming model, and fixes for agent and interface functions. Developer tests reported 63–73 tokens per second under an 8GB memory limit and 12.7–14.8 tokens per second on an Intel Arc A750 8GB with a 128K context; the source also notes that offloading shifts pressure to system memory and recommends at least 32GB of RAM.

    AIbase AI新闻 · 🔥 0
  4. Strata answer_room reportedly caps thinking and inserts wrap-up text

    A r/LocalLLaMA user reports that Strata and PolyStrata enable answer_room by default when thinking is on, max_tokens is at least 1024, and no reasoning budget is specified. The reported behavior limits thinking to 0.75 times max_tokens, inserts a model-voiced wrap-up line, and continues the same response. The client receives no error, flag, or finish reason, and there is often no log. The user cited source-code locations, but the wider effects and whether the injected text helps or harms remain unverified.

    r/LocalLLaMA · 🔥 0
  5. ISTA releases 1-bit DeepSeek-V4.1-Flash-1017 GSQ-RCO Coder with pruned experts

    The ISTA Deep Algorithms and Systems Lab says it has released a Coder build of DeepSeek-V4.1-Flash-1017, a sparse mixture-of-experts model with 384 routed experts per layer (6 active per token) and 763.2B parameters, 551.9B of them in the transformer. The build combines GSQ (Gumbel-Softmax Quantization) with RCO (Riemannian Constrained Optimization) to prune half the routed experts, keeping 192 of 384 per layer and cutting transformer weights to 72.1 GB from 138.2 GB in the unpruned IQ2_XXS build, which the lab rounds to 1 bit per parameter. Reported results show LiveCodeBench v6 at 88.57 versus 89.71 for the base model and SWE-bench Verified at 77.00 versus 81.20, and the GGUF runs in llama.cpp b11870 or newer and in Strata.

    r/LocalLLaMA · 🔥 0
  6. Reddit user reports 20–30% throughput gain from disabling MTP

    A Reddit user says disabling MTP improved token throughput by 20–30% in tests involving GyroStrata and Strata. The tester used a GeForce RTX 3060 with 32GB of RAM and reports that the issue may arise when a model is large relative to available memory; these results remain anecdotal.

    r/LocalLLaMA · 🔥 0
  7. Strata patch speeds up DeepSeek V4.1 agent sessions

    A developer patched Strata's DeepSeek V4.1 engine to reuse cached context and prioritize experts already resident in VRAM or RAM during prompt processing. On an RX 9070 XT 16 GB system, the developer reports reducing agent-session startup from about 100 seconds to roughly 5 seconds while generating at approximately 7 tokens per second. The patch has only been tested on AMD HIP, and its approximate context splicing and slight answer changes lack formal evaluation.

    r/LocalLLaMA · 🔥 0
  8. Reddit user reports Speedy outperforming Avalon and Strata

    A Reddit user reported that the Speedy local-LLM inference engine outperformed Avalon on cold prefill, reading speed and total exchange time at 60K, 120K and 198K context lengths. In the same test on an RTX 4090 with 128 GB DDR5-3600 memory and a Ryzen 9 7950X, Speedy reportedly generated tokens about 45% faster than Strata and slightly faster than Avalon. The figures use ISTA IQ3_S quantization and were not independently verified.

    r/LocalLLaMA · 🔥 0
  9. tandem.cpp accelerates CPU-assisted MoE decoding on consumer GPUs

    The independent tandem.cpp project has released patches for llama.cpp that keep frequently used MoE experts on the GPU while the CPU handles the rest, targeting single-user decoding when a model does not fit in VRAM. The project reports that Qwen3.8-Flash-Next reached 97.3 decoding tokens per second with greedy sampling on an i5-12400F system with 64 GB DDR4 memory and an RTX 4090, compared with 29.8 tokens per second for llama.cpp. It also reports gains for gpt-oss-120b and other MoE models, although the implementation is tuned specifically for Qwen3.8.

    r/LocalLLaMA · 🔥 0
  10. Infernix updates Qwen support and adds a beta WebUI launcher

    The Infernix developer announced a release optimized for recent Qwen-architecture models and NVIDIA RTX 5090 systems on Windows and Linux. It adds Qwen3.8-Flash-Next variants, calibrated automatic speculative decoding, a reproducible lossless mode, and a beta WebUI launcher for model downloads, configuration, statistics, logs, and test chats. The developer also claims performance advantages over Strata and Basalt, but no independent benchmarks were supplied.

    r/LocalLLaMA · 🔥 0
  11. EPYC 7443 upgrade boosts local MoE inference in one system

    In a field report, an r/LocalLLaMA user replaced a Core Ultra 7 265KF and MSI Z890 with an EPYC 7443 and Supermicro H12SSL-NT while retaining the same GPU inventory. The upgrade also increased memory from 64 GB DDR5 to 256 GB DDR4 ECC RDIMM and widened PCIe links. After tuning BMC fan thresholds, RAM overclocking, IOMMU and BIOS settings, the author reported GLM-5.3-Flash and DeepSeek-V4.1-Flash decode speeds increasing by roughly fourfold, while Qwen3.8-27B dense-model performance was broadly unchanged. The author attributes the large MoE gains mainly to the models fitting in VRAM and RAM, with less data being read from NVMe storage. These are results from one user's system. One GPU was unavailable during the post-change MoE tests, so those runs used three GPUs rather than four, and Maya was updated from versions 1.0.34 or 1.1.20 to 1.1.42 during testing.

    r/LocalLLaMA · 🔥 0
  12. Reddit user reports Strata underperformed llama.cpp on dual R9700

    A Reddit user testing the Strata inference engine with Qwen Next Flash on two Radeon R9700 cards said they reached only Q3 quantization and encountered hangs when experts were offloaded to RAM. The user reported about 60 tokens per second on paper, but HumanEval+ results of 93.9% for 3-bit Flash-Next versus 95.1% for Qwen3.8-27B Q8 under llama.cpp, with nearly twice as many tokens needed. They also questioned whether favorable reports about model-specific inference engines reflect hype or edge cases.

    r/LocalLLaMA · 🔥 0
  13. Reddit poster questions llama.cpp amid a wave of bespoke inference engines

    A Reddit poster on r/LocalLLaMA argues that llama.cpp has fallen behind newer local-inference projects such as Strata, while acknowledging that the criticism may be overstated and that they still use it for 27B model runs. The poster also describes a crowded ecosystem of forked engines and says they plan to test Avalon after their current GPU benchmarks finish.

    r/LocalLLaMA · 🔥 0
  14. Community benchmark finds near-linear Qwen scaling to eight streams

    A Reddit user reported spending about $40 on vast.ai instances to benchmark RAM-heavy inference servers with Qwen 3.8 Flash Next in Unsloth Q8. On systems with two AMD EPYC 7642 processors and 251GB or 504GB of DDR4 memory, the poster reported nearly linear per-stream scaling through eight concurrent streams at a 124k context window. Raw benchmark data was posted in the Reddit comments, and concurrency beyond eight streams was not tested.

    r/LocalLLaMA · 🔥 0
  15. Betelgeuse reaches 24 tok/s for Qwen 3.8 Flash Next on dual-channel RAM

    In a self-published r/LocalLLaMA report, the Betelgeuse developer tested Qwen 3.8 Flash Next IQ4_XS on an RTX 3060 12GB with 16GB of system RAM. Betelgeuse reached 24.0 tok/s with dual-channel 2x8GB DDR4 and 14.0 tok/s with a single 16GB stick, while the developer reported byte-exact output and no dropped experts; stock llama.cpp and Strata were slower in the same tests.

    r/LocalLLaMA · 🔥 0
  16. Betelgeuse Runs Qwen 3.8 Flash Next at 22 tok/s on RTX 3060

    A Betelgeuse developer reports that the llama.cpp fork ran a 68GB GSQ-RCO-IQ2_XS build of Qwen 3.8 Flash Next on an RTX 3060 12GB with 16GB DDR4 RAM, sustaining 22.03 tok/s and bursting to 48 tok/s. The developer says Betelgeuse computed all 10 routed experts across 48 layers without pruning and produced byte-exact output against stock llama.cpp over 576 generated tokens. These results come from one developer and have not been independently reproduced.

    r/LocalLLaMA · 🔥 0
  17. Unsloth adds DeepSeek support in Strata v0.2.9

    Niko Veit announced that Unsloth now supports DeepSeek models. The post listed improvements on Strix, parallel request handling, NVIDIA DGX Spark and Intel B-series hardware, alongside a link to the Strata v0.2.9 release.

    UnslothAI (X) · 🔥 0
  18. QEngine 1.3.1 hits 504 tok/s in community Flash-Next test

    A r/LocalLLaMA user reports that QEngine 1.3.1 served Qwen3.8-Flash-Next at 406–504 tok/s in single-stream decoding on two RTX 5090 GPUs. The test also found QEngine roughly level with Strata on agentic prompt latency and faster at 10K-context prefill, while Strata led at 90K. The author cautions that repeated prompts and QEngine's adaptive expert cache put the decode result near best-case conditions, and the RAM-versus-VRAM comparison used only one boot and pass per mode.

    r/LocalLLaMA · 🔥 0
  19. Swift 1.5 Flash Next Heretic v2 targets reverse engineering

    A developer released an uncensored GGUF quantization of Swift 1.5 Flash Next using v2 heretic weights. The creator said the collaboration produced a quantization and importance-matrix recipe emphasizing coding, design and reverse engineering, including new reverse-engineering and direct-answer content. The creator also said the v2 weights preserve reasoning ability and output length and represent an improvement over v1. The release includes prebuilt Strata artifacts, and the creator warned that Strata may report a missing BF16 kernel because of the quantization specification.

    r/LocalLLaMA · 🔥 0
  20. Loading a Q8 ngram with Qwen 3.8 Flash Next in Strata

    A Reddit author describes side-loading a Q8 ngram into Strata with the --ple-gguf option or, in Docker, a STRATA_SET entry. The author reports that an unmodified Unsloth Q8 ngram paired with the IQ4_XS quantization produces an architecture-mismatch error. Adding the general.architecture metadata value “strata-ple” to the GGUF file allows Strata to load it. The author did not verify whether the Q8 ngram improves output quality, and says their memory-use and prefill/decode-speed checks were not thorough. They also report, based on what they had heard, that a BF16 ngram is not noticeably different while requiring about 50 GB more storage.

    r/LocalLLaMA · 🔥 0
  21. User Reports Nearly 400 Tokens/sec for Qwen-Flash-Next

    A Reddit user says they built an inference engine that runs Qwen-Flash-Next at nearly 400 tokens per second on three 32GB NVIDIA V100 GPUs in an older Dell R740. The engine was originally forked from Strata, with its internals rebuilt for the older Volta generation. The performance claim has not been independently corroborated and the post provides limited technical detail.

    r/LocalLLaMA · 🔥 0
  22. User reports running Qwen 3.8 Flash Next with Strata on 8 GB of VRAM

    A Reddit user reports running Qwen 3.8 Flash Next with Strata on a Linux system equipped with an RTX 5060 Laptop GPU with 8 GB of VRAM, 32 GB of RAM and an SSD. The post documents memory-allocation and tuning parameters, including offloading experts to SSD storage and distributing the KV cache between VRAM and RAM. With the Gyro S quantization, the user reported prompt-processing rates of 800–1,040 tokens per second and token-generation rates of 30–45 tokens per second, depending on configuration and context length. The user also said the model performed better in their benchmarks and real-world tests than the Qwen 3.6 35B A3B fine-tunes and Qwen 3.8 27B models they had tested. These measurements and comparisons reflect one user's system and have not been independently reproduced.

    r/LocalLLaMA · 🔥 0
  23. Hobbyist boosts Strata decoding for Qwen3.8-Flash-Next

    A hobbyist running IQ3_S Qwen3.8-Flash-Next on Strata 0.1.43 with an RTX 4090, 48 GB RAM and a Ryzen 9 3900X reported 2,076-token prompt processing and an average decode rate of 95–105 tokens per second at a 200,000-token context. A separate grid-search result rose from 97 to 142 tokens per second with --adapt-async 1, --kv-resident 20480 and STRATA_ADAPT_LAG=2. The author also found that --ple-io ram, a large explicit --expert-cache and --no-prefill-borrow reduced throughput by 90%, 72% and 46%, respectively.

    r/LocalLLaMA · 🔥 0
  24. Qwen3.8 Flash Next IQ3_S benchmarked on RTX 5080

    A Reddit user reported benchmarking Qwen3.8 Flash Next IQ3_S with Strata/Basalt on an RTX 5080 with 16GB VRAM, an Intel Core Ultra 9 285K, 62GB DDR5-6800, 262K context and an int8 KV cache. In 103 opencode agent-coding requests with contexts up to 130K, it averaged 5,011 tok/s for prompt processing and 153 tok/s for generation. A 15K–207K context sweep averaged 5,475 tok/s for prompt processing and 154.9 tok/s for generation, with a peak of 5,662 tok/s and 186.6 tok/s, respectively. The agent test recorded a generation peak of 262 tok/s, the report’s best single second, exceeding the title’s “up to 250 tok/s” claim.

    r/LocalLLaMA · 🔥 0
  25. ik_llama.cpp Fork Update Adds RTX 2060 Inference Results

    IceFog72 released an updated ik_llama.cpp fork and invited community feedback on r/LocalLLaMA. The developer self-reported about 45–48 tokens per second on short prompts, about 37 tokens per second at a 28K context length, and prefill speeds of roughly 430–700 tokens per second on an RTX 2060 with 6 GB of VRAM. The update retains the core idea of the earlier approach but has a slightly different implementation and still requires cleanup and broader testing.

    r/LocalLLaMA · 🔥 0
  26. Reddit user reports 10 tokens per second for DeepSeek V4.1 Flash

    Reddit user u/DeMischi reported running DeepSeek V4.1 Flash at 10 tokens per second with an 8K context using Strata. The setup included an RTX 5090, 64GB of DDR5 RAM, a Lexar 4TB NVMe drive and Windows 10. The user said the model ran successfully but that, at this speed, using the API was cheaper than paying for the electricity required for local inference.

    r/LocalLLaMA · 🔥 0
  27. capy-flash v0.7.4 expands local Qwen3.8 inference

    capy-flash v0.7.4 is a standalone local inference engine for Qwen3.8-27B and Qwen3.8-Flash-Next, designed around NVIDIA NVFP4 quantization and multi-GPU tensor parallelism. The author reports broader single-GPU support, persistent context caching, faster long-prompt processing, and compatibility with Claude Code, Codex CLI, and OpenCode; the post also includes local throughput benchmarks, but these results have not been independently verified.

    r/LocalLLaMA · 🔥 0
  28. AbliteratedOutTheBox Ported to Strata

    A developer has ported the AbliteratedOutTheBox inference-time abliteration engine, forked from llama.cpp, to Strata. The developer says it can use abliterated models without large amounts of VRAM or fine-tuning, but the report does not independently validate those benefits. The developer describes the Strata port as experimental and says full-model behavior and response quality still require validation.

    r/LocalLLaMA · 🔥 0
  29. Reddit analysis compares Gyro and EXL3 quantization

    A Reddit author’s code analysis says Gyro-S and Gyro-M belong to the same QTIP/Trellis-coded quantization family as EXL3, but use 15-bit states, 128-weight block encoding, and an explicit lookup-table codebook. The author found no published evidence that Gyro outperforms EXL3 in decode speed, perplexity, KLD, or top-1% performance. Direct interoperability between the formats has not been demonstrated, and Gyro’s quantization recipe is currently closed source.

    r/LocalLLaMA · 🔥 0
  30. ISTA-DASLab releases 1.75–2.25-bit DeepSeek-V4.1-Flash GGUFs

    ISTA-DASLab announced three GSQ-RCO GGUF builds of DeepSeek-V4.1-Flash-1017: IQ1_M at 1.75 bits per weight, IQ2_XXS at 2.00 bpw, and IQ2_XS at 2.25 bpw, with backbones ranging from 120.7GB to 155.2GB. The release post says IQ2_XXS matched the base model on the evaluated benchmarks at reasoning effort 50, while a community discussion questioned the saturated benchmarks and noted that no independent KLD or perplexity measurements were supplied. The models reportedly run through Strata on a single 12GB GPU but cannot yet run in llama.cpp because architecture support has not been merged.

    r/LocalLLaMA · 🔥 0

← PreviousNext →

Experience and discussion from the community

Share my Strata experienceAsk about Strata

Nobody has shared their experience with Strata yet.