Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

vLLM

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 195

  1. Nscale Joins PyTorch Foundation as Platinum Member

    Nscale has joined the PyTorch Foundation as a Platinum member, gaining a seat on the organization’s governing board. The AI cloud provider said it will continue investing in Ray following its planned acquisition of Anyscale and contribute to open-source infrastructure for distributed AI workloads.

    PyTorch Blog · 🔥 0
  2. Community tester benchmarks Qwen models on four Intel Arc Pro B65 GPUs

    A community tester built a four-card Intel Arc Pro B65 system and published long-context sweeps for Qwen3.8-Flash-Next-W4A16 and Qwen3.8-27B. The tests covered multiple context lengths and concurrency levels; one four-card INT4 run reported Qwen3.8-27B throughput of 50.4 tokens per second at zero context and 33.8 tokens per second at 200K. The author also began system-RAM offload tests, which had only been performed with Flash-Next at publication.

    r/LocalLLaMA · 🔥 0
  3. Developer reports 190 tok/s for Qwen3.8-27B on two RTX 3090s

    A developer reports that the Geezuz dual-GPU CUDA kernel ran Qwen3.8-27B on two power-limited RTX 3090s at 190 tok/s for one request and an aggregate 622 tok/s across eight requests. The open-source project splits computation across two GPUs over PCIe, supports tensor-core quantization paths and MTP drafting, and includes an OpenAI-compatible server, but its speed and output-quality comparisons have not been independently verified.

    r/LocalLLaMA · 🔥 0
  4. llm-d 1.0 adds enterprise controls for self-hosted inference

    IBM and other contributors released version 1.0 of the open-source llm-d distributed AI inference framework, adding multi-tenant quality of service, zero-downtime upgrades and agent traffic controls for Kubernetes-based model serving. The release also integrates context-cache reuse across multi-turn requests, tiered cache storage, batching optimization and distributed deployment of large MoE models. IBM and Red Hat said a demonstration ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at five to 10 times lower cost than a hosted service.

    SiliconANGLE AI · 🔥 0
  5. DeepSeek open-sources DeepInfer inference framework

    DeepSeek has open-sourced DeepInfer, the inference framework used for large-scale DeepSeek V4.1 Flash services, along with the deepseek-recipe token-conversion component. The supplied reports describe support for distributed inference, speculative decoding, structured output, NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs. The public release does not include all infrastructure required for online services, including KVCache management and load balancing. DeepSeek says one cluster serves 155 trillion tokens per day and generates $38 per GPU-hour at off-peak pricing; the supplied reports provide no independent measurement of those figures.

    InfoQ 中文 AI&大模型 · 🔥 0
  6. Stop buttons only halt LLM output when the app actually closes the connection

    In a post on r/OpenAI, Sorami reported a single-lab test of self-hosted vLLM and SGLang engines plus two AI gateways: generation stopped immediately with no extra tokens only when the app actually hung up the connection, while an app that merely stopped displaying output but stayed connected kept the model writing in the background and consuming unseen GPU time. The post also says that when an app timed out and retried without closing the first connection, both requests kept running and answers delivered on time nearly halved. The authors recommend that stop buttons and retry logic close the connection rather than just hide output, and note they did not test OpenAI's own hosted service.

    r/OpenAI · 🔥 0
  7. Berth v0.3 parks local AI engines in system RAM to speed GPU switching

    Berth, a system service its developer describes in a r/LocalLLaMA post, watches the GPU and swaps and loads whichever engine the user wants, so that VRAM is dedicated to each running function without manual swaps or leftover processes holding memory. The developer has released v0.3, which parks engines in system RAM instead of unloading them, so switching back to a parked engine skips the reload from disk. They are looking for testers with 64, 96 or 128 GB of RAM and a 24 GB card, multi-GPU machines, Windows with an NVIDIA card, and AMD on Linux, noting that their own 62 GB workstation cannot measure parking at larger sizes because Strata alone pins 40 GB.

    r/LocalLLaMA · 🔥 0
  8. Google and Meta Open-Source TorchTPU for PyTorch on TPUs

    Google and Meta’s PyTorch team released TorchTPU as an open-source native backend for running PyTorch workloads on Google Cloud TPUs. It supports standard PyTorch APIs, eager-mode debugging and torch.compile optimization, with integrations spanning Hugging Face Transformers and Diffusers, vLLM, SGLang, llm-d, TorchTitan and LiteRT. Google says its training setup was validated on up to 8,000 Ironwood TPUs and achieved as much as 52% model FLOPS utilization with Qwen3-8B; these figures are vendor claims.

    Hugging Face · 🔥 0
  9. vLLM and SGLang launch native PyTorch backends for Google Cloud TPUs

    vLLM released vLLM TorchTPU, an out-of-tree platform plugin that lets vLLM run on Google Cloud TPUs, and Google said it also launched a native PyTorch backend for SGLang built with RadixArk. Both engines are built on Google's TorchTPU stack, reuse existing model definitions and keep upstream schedulers, KV cache managers and OpenAI-compatible servers, with TPU-specific Pallas kernels for attention, mixture-of-experts and quantized matmuls. Google says the engines are running in production with preview customers and cites SemiAnalysis InferenceX results including 2.01 cents per million tokens, while vLLM lists TPU v7x (Ironwood) as recommended hardware and v6e, v5e and v5p as experimental.

    vLLM Blog · 🔥 0
  10. Google open-sources TorchTPU, giving PyTorch a native TPU backend

    Google announced Tuesday that TorchTPU, its version of PyTorch for TPUs, is now available as an open-source project, adding a native torch.device("tpu") backend so existing PyTorch training scripts need only minimal device-initialization changes. Google and Meta co-developed TPU-native execution passes inside TorchTitan, and a TorchTitan TPU fork is published on GitHub; Google says it worked with Hugging Face, Inferact (vLLM) and RadixArk (SGLang) on tool compatibility. Google reports vendor benchmarks including about 57% MFU for Llama 3 405B from 128 to 1,024 TPU v7x chips and 32.42% MFU for Qwen3 235B on a single 1,024-chip slice, while Amazon released its own PyTorch variant, TorchNeuron, for Trainium and AWS a day earlier.

    Google Developers Blog · 🔥 0
  11. Developer ports Arch Linux to an RTX Spark laptop, exposing 116 GiB unified memory to CUDA

    An Arch Linux package maintainer documented on his personal blog porting Arch Linux ARM to an ASUS ProArt P16 — the RTX Spark laptop configuration with 128 GB RAM, 6,144 CUDA cores and an NVIDIA N1X GPU — and published his bootable setup and patches. In his own 7-Zip runs, single-thread performance was roughly a tie with published Windows results, while 20-thread compression was about 5% and decompression about 3% ahead, though the comparison machines used unspecified or different 7-Zip versions and dictionary sizes. In local inference he measured about 30.9 tokens/s on Qwen3.8-Flash-Next coding tasks, and said all 171 measured responses passed his checks with median response times growing under 1.6% over a ten-minute run.

    Hacker News 首页(100+ 分) · 🔥 0
  12. Qwen3.8-27B and Qwen3.8-Flash-Next local inference benchmark roundup

    An r/LocalLLaMA user, FeydRowan, compiled single-request decode speeds for Qwen3.8-27B and the 125B MoE Qwen3.8-Flash-Next on consumer hardware, marking rows as “verified” only when backed by a LocalMaxxing run or a paired run on the same machine. The author's own capy-flash rows were measured on 2× RTX 5070 Ti with 96 GB DDR5, while most other figures are as reported by each project; the post also compares tensor parallelism versus layer splitting, expert offload, quant formats and drafters, and calculates tokens per second per €1,000 using EU VAT-inclusive prices. It puts 2× R9700 on top for Flash-Next cost-efficiency and says 96 GB of DDR5 is about 40% of the bill for such builds, but the performance and price data are largely user-compiled and self-reported.

    r/LocalLLaMA · 🔥 0
  13. Community recipe runs Qwen 3.8 Flash Next on two CMP 170HX GPUs at 840 tok/s aggregate

    Reddit user /u/dead-fish on r/LocalLLaMA published an optimization recipe for running Qwen 3.8 Flash Next on two CMP 170HX mining cards, both connected over PCIe Gen2 x4 with no x16 mod and no TP2. The setup uses a W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding, with the n-gram table memory-mapped from NVMe and everything else fitting on the two GPUs. The author reports 840 tok/s aggregate decode for eight concurrent code requests, about 116 tok/s per request with 231 ms time to first token, 6,300–7,100 tok/s prefill on 4k–64k prompts and about 5,900 tok/s at 128k, while using the full 262,144-token window.

    r/LocalLLaMA · 🔥 0
  14. vLLM cuts Elastic EP scaling downtime to under 2 seconds

    vLLM published a technical post describing how asynchronous preparation and the NIXL EP backend let a running Elastic Expert Parallelism deployment add or remove GPUs with near-zero serving downtime. In its DeepSeek V4 Pro benchmark on GB200 GPUs, scaling from 16 to 20 GPUs previously paused serving for 100 seconds under the earlier Elastic EP approach and 197 seconds with a full restart, while the new flow cut the pause to 0.82 seconds; scaling back to 16 GPUs fell from 56 seconds to 1.96 seconds. The reconfiguration flow splits into background preparation — initializing joining workers, building the target communication groups, transferring weights over NVLink or RDMA, and warming up kernels and CUDA graphs — and a short commit phase that briefly pauses schedulers; vLLM says the changes are in the latest vLLM and NIXL releases.

    vLLM Blog · 🔥 0
  15. Corbenic's Galahad puts KV cache on server SSDs; its 14x claim draws scrutiny

    TechRadar reports that Belgian startup Corbenic AI's Galahad, in beta since October 1 2026, writes a model's KV cache to encrypted local NVMe storage and reloads it after GPU memory is cleared or the server restarts, so the model need not reread the same text. Founder Sietse Schelpe says it needs only Linux, an Nvidia GPU and a standard NVMe drive, with no special hardware. The report notes, however, that the headline 14x figure owes most to a retrieval component called Blaise, which hands the model roughly 668 tokens instead of the whole corpus; with the KV component Taliesin alone, vLLM took 15.56 seconds per question versus an 8.11-second baseline, and SGLang 22.02 versus 10.29. It also notes that LMCache, Nvidia Dynamo, WEKA and Phison already persist KV cache to storage, and that Corbenic's numbers come from the founder's own runs on rented cloud GPUs in self-published preprints, with raw logs not yet released.

    TechRadar · 🔥 0
  16. ASUS launches first Intel Arc Pro B70 card: 32GB ECC VRAM, 367 TOPS

    ASUS has introduced its first Intel discrete GPU, the UGen Intel Arc Pro B70 32GB GDDR6, aimed at AI inference and workstation workloads — marking the vendor's return to Intel's add-in card lineup after skipping several generations. The card uses the Xe2-based Big Battlemage GPU with 32 Xe cores, 256 XMX AI engines, 32GB of GDDR6 ECC memory, 608GB/s bandwidth, PCIe 5.0 x16 and a 310W typical board power; ASUS rates INT8 throughput at up to 367 TOPS. ASUS says it supports frameworks including vLLM, llama.cpp, Ollama, PyTorch, OpenVINO, oneAPI and Vulkan, and that up to four cards can be combined for 128GB of total memory to run larger models. ASUS has not published a price; IT之家 notes comparable Arc Pro B70 cards typically sell for about $1,299 to over $2,000.

    IT之家 AI · 🔥 0
  17. Fanshi and Reisi Zhilian launch open-source ModelSphere deployment platform

    On October 16, Fanshi Intelligent Technology Group (ticker 6682.HK) and Beijing Reisi Zhilian Technology announced ModelSphere, an all-in-one production-grade AI deployment platform released free to enterprise users under the Apache 2.0 license and already available on GitHub, combining compute management, model deployment and runtime optimization in one platform. According to the two companies, it offers a graphical console and preset deployments plus elastic autoscaling, prefill-decode disaggregation, three-tier KV cache, high-availability load balancing and automatic stream resumption, and integrates the EngineX inference engine to schedule domestic accelerators from Huawei Ascend, Iluvatar CoreX, Moore Threads and Hygon DCU in a unified or mixed pool. The vendors also claim its AutoTune scheduling raises peak system throughput by about 285% over a baseline, roughly 4x the original performance, and that it has been tuned for DeepSeek-V4/V4-Flash, Qwen, Kimi, MiniMax and GLM models; those performance figures come from vendor materials and no independent measurement was reported.

    InfoQ 中文 AI&大模型 · 🔥 0
  18. vLLM adds day-0 support for Reflection's 501B Beam model

    The vLLM blog says it supports Reflection's Beam from day 0: a 501B-parameter MoE with 23B active parameters per token, optimized on NVIDIA Blackwell, with reasoning, tool calling and structured outputs working out of the box and tensor, data or expert parallelism available. The team says coding agents autonomously optimized the model, making batch-1 decode 2.71× faster and prefill 1.87× faster than its first implementation; with Beam's DFlash2 drafter, a single request reaches 708 tokens/s on TP8 with an acceptance length of 3.7, while adaptive verification adds about 10% throughput at high concurrency. vLLM also worked with SkyRL on day-0 RL support for Beam, with LoRA, BF16 full fine-tuning and MXFP8 RL available directly.

    vLLM Blog · 🔥 0
  19. vLLM Uses Coding Agents to Optimize Kimi K3 Inference on GB300

    The vLLM team said it used agent-assisted development to build a unified CuTe DSL decode path for Kimi K3 on NVIDIA GB300 systems, covering attention, MoE and communication across 8 or 16 GPUs. The team reported more than 840 tokens/s at batch size 1 on 16 GPUs with DSpark, 1.5–2× higher decode throughput in several tested configurations and up to 1.8× higher P90 interactivity on AgentX. SemiAnalysis separately reported 423 tok/s/user on GB300 NVL72, a 1.68× speedup, while the figures depend on different configurations and measurement methods.

    vLLM Blog · 🔥 0
  20. vLLM: Kimi K3 on AMD MI355X tops NVIDIA HGX B300 throughput per GPU

    A vLLM blog post details the engineering behind serving Kimi K3 on an eight-GPU AMD Instinct MI355X system with vLLM, covering cache reuse of conversation state, eliminating duplicate MoE work across GPUs, fusing memory traffic, an AMD AITER assembly path that speeds long-context MLA decoding, and parallelized KDA prefill and speculative decoding. Citing a SemiAnalysis InferenceX snapshot from October 9, 2026, it reports about 7,567 total tokens/s/GPU at 50 tokens/s/user P90 interactivity versus 6,083 for NVIDIA HGX B300, a 24.4% advantage. Using modeled TCO of $1.50 versus $2.26 per GPU-hour, it claims 87.4% more tokens per dollar; the figures come from a vendor-adjacent source and modeled cost assumptions, with no independent replication reported.

    vLLM Blog · 🔥 0
  21. Reflection releases Apache 2.0 weights for Beam, a 501B open coding model

    Reflection AI released open weights for Beam, its first large language model, on Hugging Face on October 16 under the Apache 2.0 license, in BF16, FP8 and NVFP4 checkpoints. Beam is a sparse mixture-of-experts model with 501B total parameters and 23B active per token, a context window Reflection reports at 1 million tokens, aimed at coding and agentic tasks. Hugging Face, AWS and Baseten detailed launch-day support through transformers, vLLM, SGLang and managed endpoints, alongside post-training recipes and a drafter model for speculative decoding.

    Hugging Face · 🔥 0
  22. nFOX Lab launches AI model management platform, plans to open-source Blockhouse sandbox

    AI governance startup nFOX Lab has launched a platform for managing owned large language models and said it plans to open-source Blockhouse, a security sandbox for AI workloads, under the Apache 2.0 license. According to SiliconANGLE, Blockhouse removes privileges by default and restores access through policies enforced at the Linux kernel level, routes communications through a monitored channel called Porthole, and verifies model files with SHA-256 hashes before serving. nFOX says the platform is available now while Blockhouse is in beta for qualified enterprise evaluations; the startup is self-funded, has paying beta pilot customers and is not yet profitable.

    SiliconANGLE AI · 🔥 0
  23. Sipeed unveils 128GB MaixStation local-AI workstation

    Sipeed announced the MaixStation, a local-AI workstation built around Axera’s 16-core AX8860P and offered with 128GB of LPDDR5X across a 256-bit bus. Sipeed claims 273 GB/s of memory bandwidth, a 256-TOPS native-FP4 NPU, and an expected price of about $3,999 for the 128GB model. It says published Qwen tests outperform the Nvidia DGX Spark by roughly 10%, while another claimed run reports 25% faster prefill and 30% faster decode; these vendor results have not been independently validated, and the NPU toolchain and Kickstarter shipping remain pending.

    Tom’s Hardware · 🔥 0
  24. PentAGI: open-source autonomous penetration testing multi-agent system

    The project README describes PentAGI as an open-source, self-hosted multi-agent system for autonomous penetration testing, developed by VXControl, a security engineering company based in Dubai. It says a user describes a target and goal in plain language; specialized AI agents plan the work and run 20+ professional tools such as nmap, Metasploit and sqlmap inside an isolated Docker sandbox, remember what they learned and return a report. The README also says results and successful approaches are stored in PostgreSQL with pgvector, with an optional Neo4j-based Graphiti knowledge graph, and that the system supports 10+ cloud LLM providers, an OpenAI-compatible gateway and local models through Ollama or vLLM. These are the project's own claims.

    GitHub Trending(每日) · 🔥 0
  25. Nvidia releases 120B Nemotron 3.5 Super VL and two Bolt edge models

    Nvidia released Nemotron 3.5 Super VL, an open-weight multimodal mixture-of-experts model with 120 billion total and 12 billion active parameters, aimed at always-on agents and combining reasoning, instruction following and tool use with text, image, document and video understanding. Weights are available on Hugging Face in BF16 and NVFP4, organizations can post-train it with Nvidia NeMo on their own data and workflows, and both vLLM and SGLang announced day-0 serving support. Nvidia also introduced the Nemotron 3.5 Bolt line: the 74B-A9B 24GB, a 74-billion-parameter, 9-billion-active model compressed with 2-bit quantization to fit 24GB of memory, is available now, while a roughly 4GB Bolt E4B is planned for October 23. Nvidia claims up to 5x the throughput of comparable open models on agentic tasks, an 80.8% CharXiv chart-reasoning score (84% with Python code execution), and that Dynamo can serve up to 8x as many concurrent users with 85% lower time to first token under specified conditions; these figures are vendor-reported and not independently verified.

    vLLM Blog · 🔥 0
  26. vLLM adds day-0 support for Meta's Muse Spark Open, an 816B open-weight MoE

    vLLM said it worked with Meta Superintelligence Labs to bring day-0 inference support to Muse Spark Open, an open-weight sparse mixture-of-experts model built for long-horizon agentic coding workflows. The model has 816B total parameters with 42B active per token and a 1M-token context window, plus native image, video and audio perception. Meta released weights in BF16 and NVFP4, AMD published MXFP4 weights, and vLLM supports tensor and pipeline parallelism on Nvidia and AMD with initial TPU support. vLLM also shipped a DSpark speculative-decoding draft model and day-0 LoRA reinforcement-learning support. vLLM called it "the most capable US open-weights model to date," a claim not backed by independent benchmarks in the post.

    vLLM Blog · 🔥 0
  27. Intel launches Inference Microservices for Xeon LLM deployments

    Intel announced Intel Inference Microservices, a containerized microservice offering for LLM inference that aims to give enterprises tuned model serving on existing Xeon processors without days of manual hardware optimization. Intel says each microservice runs one model in its own container behind an OpenAI-compatible API, detects the host's processor generation, allocated cores, NUMA nodes and built-in accelerators at startup, and applies pre-validated configurations; the first versions use vLLM with BF16 and run on Docker or Kubernetes 1.24+, requiring 4th–6th Gen Xeon and 32GB of memory recommended. Model-specific images fail at startup if the host is unrecognized rather than running untuned, and Intel plans later additions including Helm charts, a Kubernetes Operator, more inference engines and precisions, plus extending the model to GPUs such as Crescent Island.

    IT之家 AI · 🔥 0
  28. Houmo open-sources AI software stack, outlines 2D and 3D SRAM-CIM chip roadmap

    At its first ecosystem partner conference in Beijing on October 14, Houmo AI said it is open-sourcing its full "Houmo AI ecosystem software" stack, spanning model quantization and compilation, inference frameworks and operator libraries, domain kits and middleware, and deployment/operations tools, with 15 open-source projects to go online by January; repositories such as postmo-modelzoo, postmo-quantflow, vllm-postmo and adapted llama.cpp and vLLM builds are already visible on GitHub. The company also detailed two paths built on SRAM compute-in-memory: 2D CIM for high capacity and compute (already shipping in M30 and M50) and 3D CIM for high-bandwidth dense models, with its first 3D CIM chip A20 in production and an M6X successor to M50 in development, claimed at 3–6x the M50's overall capability. Houmo says M50 has landed more than 150 product design wins and over 100 customers within a year of mass production, including Lenovo's AI host P7, Deepshard's Truffle 1, UGREEN's HomeAgent HA100 Pro, Liaogu's ClawHouse X1 and Great Wall's N90 Pro laptop; these performance and commercial figures are vendor claims.

    新智元 · 🔥 0
  29. Baidu Releases 3B Unlimited-OCR for Document Parsing

    Baidu released the open-weight, MIT-licensed Unlimited-OCR model, a 3B image-text-to-text system designed for one-shot, long-horizon document parsing across images, multi-page images and PDFs. Its model card lists Transformers inference, vLLM and SGLang deployment, ms-swift training, availability through Baidu Cloud and ModelScope, and a demo on Hugging Face Spaces. Baidu said the model aims to push DeepSeek-OCR further, but the supplied reports do not include independent comparative evaluations.

    百度文心 · Hugging Face · 🔥 0
  30. vLLM brings adaptive verification to draft models without a confidence head

    vLLM's blog says Adaptive Verification now supports all draft-model-based speculator types, not just DSpark drafters that ship with a confidence head. For drafters lacking one, such as EAGLE, MTP, DFlash and some DSpark checkpoints, vLLM uses a lightweight logistic regression that maps the log-odds of the drafter's most likely token to the target model's acceptance probability, with one shared slope plus one intercept per draft position (K+1 parameters) fitted online from the accept/reject decisions produced during normal verification. The feature landed in PR #52228 and is enabled with enable_adaptive_verification; on DeepSeek-V4-Flash with DSpark the estimator beat the trained confidence head on AUC and log-loss but had worse ECE, and the two were nearly tied on calibration once the first draft position was excluded. Stated limitations include requiring a target attention backend that supports variable-length verification, and smaller gains for MTP and EAGLE than for DFlash and DSpark.

    vLLM Blog · 🔥 0

← PreviousNext →

Experience and discussion from the community

Share my vLLM experienceAsk about vLLM

torchtitan: A PyTorch-native platform for pretraining and post-training generative AI models at scale.

What problem it solves Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development. Key features Composable model …

github.comhttps://github.com/pytorch/torchtitan

vllm-ascend: A community-maintained hardware plugin for running vLLM on Huawei Ascend NPUs.

What problem it solves vLLM users need a way to run the framework on Huawei Ascend NPU hardware through a supported integration path. The plugin decouples Ascend NPU integration from vLLM and enables supported model types to run on the platform. Key features Hardware-pluggable vLLM integration — The plugin provides an interface for integrating Ascend NPUs with vLLM and follows the community's hard…

github.comhttps://github.com/vllm-project/vllm-ascend

vllm: A fast and easy-to-use library for LLM inference and serving.

What problem it solves Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms. Key features Efficient memory and serving — vLLM manages atte…

github.comhttps://github.com/vllm-project/vllm

semantic-router: An open, programmable decision layer for models and compute.

What problem it solves Agent harnesses need to coordinate models, tools, task state, and changing compute requirements without manually managing every backend. They also need a way to apply location constraints and priorities for quality, latency, and cost to each call. Key features Policy-driven model routing — Select or combine models for each call according to a configured policy while leaving …

github.comhttps://github.com/vllm-project/semantic-router

openworker: A desktop app and harness SDK for building and running AI coworkers that work across everyday tools.

What problem it solves People need a way to delegate research, writing, code review, recurring tasks and work across connected tools to AI agents without giving up control over actions and data. They also need a foundation for embedding agent loops, tool calls, approvals, streaming and durable sessions into their own applications. Key features Everyday AI coworker — The desktop app supports resear…

github.comhttps://github.com/andrewyng/openworker

Hy-MT2: A family of fast-thinking multilingual translation models for complex real-world translation scenarios.

What problem it solves Real-world translation involves more than converting text between languages: tasks can require terminology, style, personalization, delimiter preservation, and structured-data constraints. Hy-MT2 is designed to support translation among 33 languages and follow translation instructions in multiple languages for complex real-world scenarios. Key features Instruction-driven tas…

github.comhttps://github.com/Tencent-Hunyuan/Hy-MT2

G0DM0D3: A fully open-source, privacy-transparent, multi-model chat interface for red teaming, cognition research, and liberated AI interaction.

What problem it solves Researchers, red-teamers, and system tinkerers need a chat interface that can work with multiple hosted and local models while exposing evaluation, perturbation, and sampling controls. They also need clear visibility into data flows, storage, telemetry, and deployment boundaries so they can make informed choices about privacy and self-hosting. Key features Multi-model races …

github.comhttps://github.com/elder-plinius/G0DM0D3

memorilla: A memory system that gathers information from an entire document collection into query-dependent memory tokens for a frozen LLM.

What problem it solves A frozen LLM can only use information that fits in its context window, making it difficult to answer questions from large document collections or long interaction histories. Memorilla addresses this by reading a collection and supplying the LLM with a fixed number of query-dependent memory tokens. Key features Query-dependent memory compression — The system reads a complete …

github.comhttps://github.com/snap-stanford/memorilla