Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Berth v0.3 parks local AI engines in system RAM to speed GPU switching

AI summary

Berth, a system service its developer describes in a r/LocalLLaMA post, watches the GPU and swaps and loads whichever engine the user wants, so that VRAM is dedicated to each running function without manual swaps or leftover processes holding memory. The developer has released v0.3, which parks engines in system RAM instead of unloading them, so switching back to a parked engine skips the reload from disk. They are looking for testers with 64, 96 or 128 GB of RAM and a 24 GB card, multi-GPU machines, Windows with an NVIDIA card, and AMD on Linux, noting that their own 62 GB workstation cannot measure parking at larger sizes because Strata alone pins 40 GB.

In-depth insight

The developer says their own 62 GB workstation cannot measure parking at the sizes other users run, so the benefit of v0.3's RAM-parking approach remains unverified outside their setup.

#Berth#ComfyUI#OpenWebUI#llama-swap#vLLM#Strata

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.