llama.cpp
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 13
- llama.cpp on a jailbroken PS5: run any GGUF model natively
A GitHub project called PS5LM drew attention on Hacker News with a title claiming it lets a jailbroken PlayStation 5 run any GGUF-format model natively through llama.cpp. The only available information is the project title and the forum discussion link, with no details on inference speed, model sizes supported, or the required firmware or jailbreak version. It therefore reads as a hobbyist porting experiment whose real-world usability still needs more testing.
Hacker News · AI（100+ 分） · 🔥 20 - PrismML releases Ternary Bonsai 2 on-device models
PrismML on Oct. 15 announced the Ternary Bonsai 2 series of on-device models in 2B, 4B and 9B sizes, built with ternary weights that encode parameters as only -1, 0 or +1 and take up roughly 0.43GB, 0.95GB and 2GB of storage respectively, joining the previously released 27B version to cover lightweight to complex local workloads. The company says the models target local agents on phones, PCs and wearables, handling instruction following, everyday tasks, tool calling and image understanding; across 19 benchmarks spanning six capability areas, the 9B model used about 40% fewer tokens on standard instruction execution and everyday math tasks. PrismML claims up to 5x throughput and 3x energy efficiency versus equivalent full-precision models on hardware such as the RTX 4090 and M5 Pro, with the 9B reaching up to 34 tokens per second and the 2B up to 120 tokens per second on an iPhone 18 Pro Max; the models are built on Alibaba's Qwen 3.5 base, released under Apache 2.0, and runnable via llama.cpp on NVIDIA GPUs or MLX on Mac, iPhone and iPad.
IT之家 AI · 🔥 18 - nFox launches Blockhouse sandbox, plans to open-source it
AI governance startup nFOX Lab launched the nFOX Platform for managing self-hosted large language models and said it plans to release its Blockhouse security sandbox under the Apache 2.0 license. Blockhouse removes privileges by default and restores access through policies enforced at the Linux kernel level, routing communications through a monitored channel called Porthole and keeping runtime credentials and audit controls outside the model process. It supports the vLLM, SGLang and llama.cpp inference engines, while the platform is available now across cloud VMs, rented GPU hosts and on-premises Linux servers with Nvidia GPUs; Blockhouse is in beta for qualified enterprise evaluations. Founder and CEO Gordon Allott said the company is self-funded, has paying beta pilot customers but is not yet profitable, and intends to raise venture capital and later expand into agent protection.
SiliconANGLE AI · 🔥 4 - RTX Spark review: beats M5 Max in local AI, lags in gaming
Tom's Hardware tested Nvidia's RTX Spark platform in a top-trim Microsoft Surface Laptop Ultra carrying the full RTX Spark Superchip with 20 CPU cores, 6,144 CUDA cores and 128GB of LPDDR5X-9400 memory. Running Artificial Analysis's new end-to-end agentic benchmark AgentPerf-local with Qwen 3.8 27B and Qwen 3.6 35B A3B, the RTX Spark system finished about 25% faster than a 14-inch MacBook Pro with Apple's M5 Max, while AMD's Ryzen AI Max+ 395 (Strix Halo) needed roughly twice as long, about 40 minutes, on the heavier test. Gaming is a weaker point: roughly 300 GB/s of shared memory bandwidth, Arm CPU cores and the overhead of the Prism x86 emulator leave frame rates and input latency behind dedicated RTX 50-series gaming laptops, and the reviewer leans on DLSS Frame Generation to compensate. The outlet notes that launch embargoes limit this first round to platform-wide results, with battery life, build and display details to follow.
Tom’s Hardware · 🔥 3 - Llama for Windows released: one-click local model downloads
Llama for Windows, the Windows counterpart to the macOS Llama app, is a free, open-source Windows 11 application built on Microsoft's WinUI 3 that lives in the system notification area and opens its control panel from a taskbar icon. It connects directly to the Hugging Face model hub, shows a curated model list with parameter counts, disk usage and vision support, and downloads GGUF-quantized models driven by the llama.cpp ecosystem, all without admin rights, manual compilation or complex launch parameters. The app offers an Alt+Space floating chat window, a built-in web UI for loading and unloading local models, and can start an OpenAI-compatible local server at localhost:9931 that scripts and AI agents can call, with multiple apps able to share one model file. According to the project, prompts and inference stay on the device and no telemetry, analytics events or installation identifiers are collected, though network access is still needed to fetch models and updates; it supports Windows 11 x64 and ARM64 plus NVIDIA's RTX Spark platform, and does not list Windows 10 as supported.
cnBeta · 🔥 7 - Local-first AI assistant Jarvis trends on GitHub with offline voice chat
Jarvis is a privacy-focused desktop AI assistant that offers voice and text chat, personal memory, dictation and MCP tool use, and runs offline via Ollama or an OpenAI-compatible local model server on macOS, Windows and Linux. While listening it keeps a short rolling transcript of nearby speech, so when addressed by name mid-conversation it can answer using what was just discussed; its diary and knowledge graph are stored on the user's own machine. It is free for personal use with a separate commercial license, and the developer notes it is built primarily on macOS, so Windows and Linux behavior may differ and dictation is unavailable on macOS 26 and later.
GitHub Trending · Python · 🔥 3 - Houmo AI open-sources full software stack for its M50 edge chip
At an ecosystem partner conference on October 14, Houmo AI said it will fully open-source its AI software stack, covering four layers — model quantization and compilation, inference frameworks and operator libraries, domain kits and middleware, and deployment/operations tools — in three batches, with 15 projects due by January, including adaptations of llama.cpp and vLLM; postmo-modelzoo, postmo-quantflow and vllm-postmo are already on GitHub. The company says its M50 chip, less than a year into mass production, has been picked for more than 150 terminal products by over 100 customers, including Lenovo's 300g AI host P7 (up to 48GB LPDDR5/5X, a 122-billion-parameter model running offline at 50 tokens/s), Great Wall's N90 Pro laptop running a 35B model fully offline on Kylin V11, and UGREEN's HomeAgent HA100 Pro. Houmo also claims its SRAM compute-in-memory design cuts chip area by about 70% and power to roughly one-tenth, and that M50 is China's only high-compute edge AI chip compatible with mainstream operating systems and CPU architectures.
新智元 · 🔥 3 - Jailbroken PS5 runs 27B local AI models at 20 tokens/sec
GitHub developer cobanov previously ported llama.cpp to the PS5 as the community project PS5LM, but it ran as a CPU payload limited to roughly 6.25 GiB of RAM and mainly handled small 0.8B and 2B models. Reddit user u/AdIllustrious7246 then forked it into a native app that reaches the console's GPU through PS5_Vulkan, a port of the RADV graphics driver, raising usable memory to about 11.4 GiB and running Qwen 3.8 27B with low-bit quantization (such as IQ2_XXS) at roughly 20 tokens per second. The developer also reported about 61 tokens/sec for Gemma 4 E4B and 124 tokens/sec for Llama 3.2 3B, and added a dashboard showing live generation speed, GPU/CPU usage, power draw and temperatures, plus an OpenAI API-compatible interface for LAN access. All figures are the developer's own publicly posted results, require a jailbroken console and specific firmware (docs list 7.00 to 13.60), and are not an official Sony feature.
cnBeta · 🔥 7 - Jailbroken PS5 runs 27B LLM locally at 20.9 tok/s via homebrew llama.cpp port
A Reddit user turned an unofficial llama.cpp port, dubbed "PS5LM," into a native PS5 app, moving inference from the CPU to the GPU via Mesa's RADV driver over Vulkan and raising usable memory from 6GB to 11.4GB. Constrained by the console's 16GB of GDDR6, it could only run a roughly 2-bit quantized Qwen 27B model (6.8GB of weights, 64K context, 91% memory use) at 20.9 tok/s, while smaller models were faster: Gemma 4 E4B at 61 tok/s and Llama 3.2 3B at 124 tok/s, with total power draw of 86W and a 54°C temperature. The trade-offs are heavier quantization errors and weaker reasoning, plus a jailbreak that voids the warranty and risks online bans.
Wccftech · 🔥 0 - Liquid AI releases LFM2.5-24B-A2B for on-device agents
Liquid AI released LFM2.5-24B-A2B on Oct. 13, 2026, an open-weight reasoning-only model with 24B total and roughly 2B active parameters, built for on-device agents; base and post-trained checkpoints are on Hugging Face. It extends context from 32,768 to 131,072 tokens, doubles vocabulary to 128K, scales pretraining from 17T to 33T tokens, and adds RL training inside agent harnesses like OpenClaw, Hermes Agent, Pi and OpenCode. Liquid AI reports it beats or matches Qwen3.6-35B-A3B on five of eight agentic/tool-use benchmarks (LiveClaw 48.09 vs 38.19; AutomationBench 27.69 vs 8.72), while trailing on SWE-Bench Verified (51.00 vs 66.20) and knowledge tests; it also reports reward-hacking behavior during teacher specialization. Day-one support covers llama.cpp, MLX, vLLM, SGLang and ONNX, with 184 tokens/s on Apple M5 Max and API pricing of $0.10/$1.00 per 1M input/output tokens.
Liquid AI Blog · 🔥 0 - AWS details llama.cpp decision-model serving on SageMaker AI with SemIf
AWS published a walkthrough for serving decision models with llama.cpp on Amazon SageMaker AI using specialized Deep Learning Containers. The standard path deploys a GGUF model from Amazon S3 and maps InvokeEndpoint to llama-server's OpenAI-compatible chat completions; an optional SemIf child image (SemIf was formerly OpenJev) returns probabilities over caller-declared options by reading final-position logits instead of decoding an answer token. On 231 public JevBench tasks with Qwen3.5-4B Q4_K_M, a 32-thread CPU endpoint (ml.c6i.8xlarge) was correct on 179/231 (77.5%, p50 1.715s, p95 19.291s), while full GPU offload on ml.g6e.xlarge scored 180/231 (77.9%, p50 0.352s, p95 0.599s) — 4.9x faster at p50 and 32.2x at p95. AWS says the two images used different llama.cpp builds and disagreed on six hard tasks, so it calls the accuracy gap runtime variation rather than proof that GPU offload improves quality, and cites Jev, Kev and Strands Decider 2B as related decision-model work.
AWS Machine Learning Blog · 🔥 0 - Hugging Face Transformers tops GitHub Trending
Hugging Face's Transformers library appeared on GitHub Trending's daily list. The project positions itself as the model-definition framework for state-of-the-art text, vision, audio, video and multimodal models, for both inference and training, and its documentation says there are over 1M Transformers model checkpoints on the Hugging Face Hub. It says the library interoperates with training frameworks such as Axolotl, Unsloth, DeepSpeed, FSDP and PyTorch-Lightning, and with inference engines and modeling libraries including vLLM, SGLang, TGI, llama.cpp and mlx, and requires Python 3.10+ and PyTorch 2.6+.
GitHub Trending（每日） · 🔥 0 - JetBrains releases Mellum2.1 coding model
JetBrains launched Mellum2.1, keeping Mellum2's 12B mixture-of-experts architecture with 2.5B active parameters and the Apache 2.0 license, and focusing the upgrade on agentic coding. JetBrains says reinforcement learning was expanded from a short post-training step into the main training stage, letting the model explore codebases, edit files, identify root causes of failing tests and draft and verify fixes. The model is available on Hugging Face, with GGUF builds for llama.cpp, Ollama and LM Studio and a vLLM MTP speculative-decoding component promised later.
IT之家 AI · 🔥 0
Experience and discussion from the community
Share my llama.cpp experienceAsk about llama.cpp
Nobody has shared their experience with llama.cpp yet.