Modal launches Sticky Sessions to pin serverless requests to one container
Modal announced Sticky Sessions, a routing feature built on its Modal Servers that sends every request within a session to the same container. Developers enable it with the @modal.sessioned() decorator and start sessions through the SDK or HTTP API, with each session receiving a signed token so multiple clients can share the same state. Modal says sessions become the unit of autoscaling, and it cites voice agents, shared chat rooms, stateful UIs such as Gradio and Streamlit, and multiplayer games as target workloads.
Modal's post argues that stateful AI workloads such as streaming speech-to-text and voice agents need session affinity that its request-level routing did not provide, which the vendor says it now addresses.
At Modal, our serverless compute powers real-time inference, training, and batch jobs, scaling automatically with demand. Sandboxes offer a different model, containers whose lifecycles you explicitly control. A growing number of workloads require something in-between; autoscaling serverless compute that maintains continuity between requests. Consider a speech-to-text model with decoder state for an audio stream, a voice agent that holds a conversation, or a multiplayer game. For these workloads, users must come back to the same container on every reconnect for the entirety of their session.
We recently launched Sticky Sessions, built on top of our low-latency Modal Servers. Every session’s requests land on the same container. To use Sticky Sessions, just create a Modal Server with the @modal.sessioned() decorator and start the sessions with the SDK or the HTTP API:
@app.server()
@modal.sessioned()
class Server:
@modal.enter()
def start(self):
subprocess.Popen(["python", "-m", "http.server", "8000"])
@app.local_entrypoint()
def main():
session = Server.sessions.start(idle_timeout=600)
headers = {"Modal-Authorization": f"Bearer {session.token}"}
print(requests.get(Server.get_url(), headers=headers).text)#Modal#Modal Servers#Sticky Sessions#Gradio#Streamlit#Pipecat