Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

mesh-llm

mesh-llm• Pinned

mesh-llm: Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API.

What problem it solves

Running large language models can require resources that do not fit on a single machine. Mesh LLM addresses this by pooling GPUs and memory across nodes, routing requests to suitable peers, and splitting large models across machines.

Key features

  • OpenAI-compatible serving — Every node exposes the same `/v1` API, while requests are routed by the `model` field to a peer that can serve that model. Operators can also run a local model without joining a mesh.

    Source: README

  • Public and private meshes — Nodes can join a discovered public mesh or operate within a private mesh. Public meshes advertise through Nostr discovery, while private meshes use invite tokens.

    Source: README

  • Distributed model execution — Models too large for one machine can be divided into package-backed layer stages. A coordinator assigns contiguous layer ranges, starts downstream stages, and publishes the stage-0 route after the stages are ready.

    Source: README

  • Encrypted peer transport — Peer-to-peer inference traffic uses QUIC end-to-end encryption, including requests, responses, and split-model activations. Iroh relays forward encrypted packets without reading their payloads.

    Source: README

  • Experimental mixture of agents — The experimental `mesh` model fans a request out to available models and arbitrates their responses with deterministic logic. Tool calls continue through the same pipeline.

    Source: README

Quick start

Copied verbatim from the project's README.

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash
mesh-llm setup
mesh-llm serve --auto

Who it may suit

Inference: may suit developers and operators who want to serve models through a unified API while using resources from one or multiple machines.

Topics to discuss

  • How should private and public mesh discovery balance accessibility with operator control?

  • Which routing and split-execution behaviors should become stable first?

  • What deployment scenarios and failure patterns should the community document next?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/Mesh-LLM/mesh-llm#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/Mesh-LLM/mesh-llm

Comments

…