Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Stop buttons only halt LLM output when the app actually closes the connection

AI summary

In a post on r/OpenAI, Sorami reported a single-lab test of self-hosted vLLM and SGLang engines plus two AI gateways: generation stopped immediately with no extra tokens only when the app actually hung up the connection, while an app that merely stopped displaying output but stayed connected kept the model writing in the background and consuming unseen GPU time. The post also says that when an app timed out and retried without closing the first connection, both requests kept running and answers delivered on time nearly halved. The authors recommend that stop buttons and retry logic close the connection rather than just hide output, and note they did not test OpenAI's own hosted service.

In-depth insight

The authors say apps that merely hide output can keep consuming GPU time nobody sees, and recommend stop buttons and retry logic actually close the connection.

We measured what happens when a user presses stop in an AI app.

The short version: the AI only stopped when the app actually hung up the connection. If the app just stopped showing the answer but stayed connected, the AI kept writing in the background, using GPU time nobody would see.

What we found on the setup we tested:

- When the app hung up, the AI stopped straight away. Nothing extra was generated.

- When the app stayed connected but stopped reading, the AI kept going.

- When the app timed out and retried without hanging up on the first attempt, both kept running. Answers delivered on time nearly halved.

#Sorami#vLLM#SGLang

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.