What problem it solves

Developers need to run AI models inside applications while handling model downloads, inference configuration, and chat sessions. The project presents local inference with data privacy and no inference costs as its deployment model.

Key features

• High-level application API — The engine exposes a simple API for creating an engine, loading a model, opening a chat session, and retrieving replies.

  Source: README

• Unified model configuration — A shared configuration approach is designed to make support for additional models consistent.

  Source: README

• Traceable computations — Computations can be traced to support correctness checks against source-of-truth implementations.

  Source: README

• Unified memory on Apple devices — The engine uses unified memory on Apple devices for its inference workload.

  Source: README

Quick start

Copied verbatim from the project's README.

uv add uzu==0.6.17

import asyncio

from uzu import ChatConfig, ChatMessage, ChatReplyConfig, Engine, EngineConfig


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4")
    if model is None:
        return

    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    session = await engine.chat(model, ChatConfig.create())

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]

    replies = await session.reply(messages, ChatReplyConfig.create())
    if not replies:
        return

    message = replies[-1].message
    print(f"Reasoning: {message.reasoning}")
    print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

Who it may suit

Inference: This project may suit application developers who want to integrate locally executed AI models through Rust, Python, Swift, or TypeScript bindings.

Topics to discuss

• Which model architectures and runtime configurations should receive priority for support?

• How should teams manage model memory requirements and session reuse in production applications?

• What patterns should the community use to verify engine computations against source-of-truth implementations?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/trymirai/uzu#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.