s³: Making Collaboration a Capability
Handing a problem to four Agents most easily produces four attempts. Increasing the number to eight may raise the success rate further. But if every Agent searches along similar lines, a significant portion of the additional computational power brought by a larger team will be spent on duplicate work. If we instead require them to communicate continuously, messages will consume context and reasoning budget. What is truly worth training is how to turn each member’s work into information useful to its partners. Kunal Jha and colleagues at the University of Washington and the Allen Institute for AI addressed this problem through reinforcement learning in “s³: Simple Scaling of Swarms via Cooperative Multi-agent RL”: multiple Agents share the same set of model parameters, independently decide how long to work, when to share, and when to read messages, and then update together based on the result of the entire team. The most intriguing design choice is to break organizational capability into learnable communication and task actions, allowing division of labor to emerge as the task unfolds. 1. Organizational Boundaries 1.1 Independent Sampling Let us begin with the simplest multi-agent approach: each Agent solves the problem independently, and the group succeeds as long as one succeeds. This is the best-of-N approach the paper uses to compare exploration capabilities. If the success probability of each independent attempt is p, the probability that at least one of N attempts succeeds is 1 − (1 − p)^N. Under this independence assumption, increasing the number of members yields gains even when the Agents do not communicate at all. Evaluating collaboration therefore requires a more specific question: with the same number of members, the same training opportunities, and the same generation budget, does communication give the team gains beyond those of independent attempts? For search tasks, such gains may come from sharing failed paths so that partners do not repeat them, or from communicating discoveries so that subsequent searches can turn toward more promising branches. Either way, messages must change the receiver’s next action. Handwritten workflows can achieve some of this: designate a planner, an executor, and a checker in advance, then prescribe the order of handoffs. In such an approach, organizational knowledge resides in the runtime framework. s³ takes the boundary a step further back: the framework retains message delivery, private state, and budget management, while leaving “who does what first and when work is handed off” to the policy. 1.2 Information Aggregation There is another class of task for which independent sampling cannot provide a complete answer. In the paper, each Agent receives a private integer, and the team must return everyone’s numbers in ascending order. Even if an Agent is excellent at sorting, it first has to obtain the remaining numbers from its partners. The difficulty here involves information transmission, reading, and attribution. For example, four Agents hold 17, 42, 42, and 73 respectively, and the correct result must retain both 42s. If one person sends their own 42 repeatedly, the receiver must know that these are duplicate messages from the same source; if two different people report 42, they must still count as two separate entries. Duplicate reports from the same sender must be deduplicated, while identical values from different senders must be preserved. These two types of task define two fundamental organizational responsibilities: aggregation tasks require distributed information to reach a place where it can be used to produce an answer; exploration tasks require information already produced to alter unfinished searches. s³ handles both through the same communication interface and team objective, while the organizational form emerges from behaviors that prove effective during training. Figure 1: Organizational choice shifts from a preset control flow to Agents’ autonomous communication and work timing. 2. Communication Interface 2.1 Private and Shared An s³ team consists of N copies of the same model. They share parameters but have separate histories, private observations, and task tools. In the exploration experiments, each enters an independent copy of the same room tree: one Agent moving does not change another Agent’s position. A partner’s reasoning process and tool results also do not automatically enter one’s own context. Crossing this private boundary relies primarily on two tools: message(to, text): send text to a particular partner, or broadcast it to all partners. read_messages(): actively read messages that have not yet been read. The framework provides an unread-count notification when a new message arrives, indicating only which sender has how many messages. “Knowing that a message exists” and “reading its contents into the context” are two separate actions. Sending a message also does not cause the sender to read messages or wait for a reply. Independent work, sending, reading, and continuing can therefore each become policy choices. Prompts still play a role: they explain the tools and shared objective, and suggest sharing discoveries, counterexamples, verified results, and questions that might change a partner’s next step. The names of the Agents and the order of the partner list are randomized for every task to reduce reliance on conventions tied to fixed numerical IDs. What the research removes is predetermined leadership, expert roles, and a fixed speaking order; task rules and the cooperation prompt still provide the starting point for learning. 2.2 Making Room for Reasoning With an inbox in place, messages also need to arrive in time to have an effect. Suppose an Agent is engaged in a long reasoning generation while a partner has already eliminated one search path. If that message cannot be seen until the entire generation finishes, the team may already have searched that path again. For reinforcement learning, latency also weakens the connection between communication and the final result. s³ uses asynchronous execution: an Agent can continue as soon as its own tool call returns, without waiting for other members. If the receiver is generating, an unread notification is inserted into its history after the current inference request ends; if the receiver is already idle, the arrival of a message wakes it. An Agent that finishes a response without making a tool call can enter an idle state and consume no generation tokens while waiting. The inference server itself also has to cooperate. Multiple Agents compete for sequence slots and KV cache, and a single overly long generation can leave other members queued. The paper limits each generation request to at most 1,024 new tokens; if there is no completed response or tool call, the subsequent request continues from the existing prefix. These request boundaries give other members a chance to execute and provide a point at which notifications can be inserted, while an entire response may still span multiple requests. Figure 2: A message first triggers an unread notification, after which the receiver chooses whether to read it and adjust its private work. At this point, the responsibilities of the runtime and the policy become clear: the runtime ensures that communication has an opportunity to influence actions promptly, while the policy decides whether that opportunity is worth using. If the infrastructure always delivers messages only after the task ends, training will struggle to learn useful handoffs; if every incoming message forcibly expands the full text, the timing of reading loses its learnable dimension. 3. Team Learning 3.1 Reward Attribution Communication is difficult to train because a valuable action may only bring another Agent closer to the answer. Returning to the number-aggregation task: sending your own number mainly helps a partner answer; it brings no new private information to the sender. If only each person’s own correctness is rewarded, an Agent may prefer to read others’ numbers first and postpone contributing its own information. s³ ties the reward to the team’s best result: the team succeeds whenever one member submits the correct answer or one explorer finds an exit. Every member receives the same team return. After success, a small cost is also deducted according to how much of the team’s generation budget was used: R = S × (1 − 0.25 × min(1, T/(N × L))) Here, S indicates whether the team succeeded, with 1 for success and 0 for failure; T is the total number of generation tokens used by the team; N is the number of members; and L is the generation allowance for each Agent. Reasoning, tool calls, and messages all count toward the allowance. Dividing the cost by N × L means that teams of different sizes receive the same proportional deduction when they use the same proportion of their budget. This formula involves two noteworthy tradeoffs. First, the cost is deducted only upon success, leaving the return of a failed team at zero and preventing “failure with an early exit” from gaining an additional advantage. Second, every successful return is higher than every failed return, so the cost’s role is to favor more token-efficient approaches among successful solutions. The paper does not add a separate reward for sending messages or a penalty for the number of messages; both talking more and talking less must demonstrate their value through task outcomes and generation costs. Rewarding a correct team outcome also makes helping a partner succeed beneficial to oneself. If each member’s correct answer were added together, the return would also increase when more people repeatedly submit the correct answer. If scoring used the answer accuracy of all members, the objective would instead favor having every person redo the entire task. s³ chooses the team’s best outcome, leaving room for complementary work. 3.2 Same-Problem Comparison After introducing the shared return, one must decide how to update the model. s³ samples eight independent teams for the same problem, each with its own history and message board. It first compares the returns of these eight teams, then assigns a team’s advantage relative to average performance to every token generated by every member of that team: A_g = (R_g − mean(R)) / (std(R) + ε) This effectively extends the comparison unit of GRPO from “one response” to “one complete team attempt.” An Agent’s sending, reading, movement, and private reasoning all update according to the advantage signal of the team it belongs to; the paper does not separately estimate how much each member contributed. Helping actions can therefore receive a learning signal through team success. A timely broadcast has no correct answer of its own, but it may enable a teammate to submit the correct list. Including the entire generation trajectory of that team in the comparison reinforces behavior that co-occurs with a better result. This signal is coarse, but the interface and training method are sufficiently unified that the same model can learn task actions and communication actions together. Figure 3: Independent teams working on the same problem compare with one another, and team advantages propagate back to the actions generated by each member. Training computes loss only for tokens generated by an Agent itself. Received messages, tool return values, and notifications may influence subsequent decisions, but they are not treated as training targets that need to be reproduced. The loss is first averaged over generated tokens within each team and then averaged across teams, preventing teams that talk more or have longer trajectories from receiving greater weight. The exploration experiments also address distributional differences between asynchronous sampling and training: a sampling trajectory can lag the current policy by as many as four updates, and inference and training use different numerical precisions. The authors use IcePop’s importance-ratio masking and score centering to correct for this. The key engineering constraint to retain when understanding organizational learning is that the policy used during sampling differs from the policy used during updating, and training must account for that difference. 4. Division of Labor Emerges 4.1 Share First The number-aggregation experiment used Qwen3-0.6B, with four Agents undergoing 200 RL updates. It started from the base model without any prior fine-tuning on demonstration data. On the held-out four-agent test, s³ achieved a success rate of approximately 93%; the online SFT control, which only imitated successful team trajectories, achieved approximately 35%; and the team trained with individual-answer correctness rewards achieved approximately 13%. The online SFT control still sampled communicating teams and had a matching collection budget; the difference lay in the learning signal. Figure 4: On the four-agent information-aggregation task, training with team rewards performs substantially better than both controls. The behavior changes are more explanatory than the aggregate numbers. Early in training, many teams had at least one member that never sent its own number. Communication then gradually shifted from direct conversations to broadcasts, passed through a stage of repeated broadcasts, and ultimately resulted in approximately 80% of teams forming a pattern in which every member broadcast once. s³ also gradually learned to share its own number before reading information from partners. This is a very simple protocol, but it precisely fits the task’s information structure: every person has a fact that is immediately useful to all partners and can be sent right away. Sending first and reading afterward moves information into the shared exchange sooner and reduces mutual waiting. When trained with individual rewards, behavior developed in the opposite direction: more Agents read messages first, and some eventually stopped sharing altogether. The authors interpret this as a public-goods dilemma: team rewards changed who benefited from sharing. In this task, the learned organizational form is a symmetric and concise exchange of information. It requires neither a predetermined leader nor a dedicated aggregation role. Agents with shared parameters participate according to the same rule, each performs the same kind of responsibility, and the complete answer can emerge within the members’ private contexts. 4.2 Then Split Paths The exploration task pushes the question of “whether communication changes the work” further. Each Agent faces an unknown room tree in which every fork has three doors, with ten steps required between forks. RL and evaluation used a tree of depth three, containing 391 rooms and 27 exit candidate positions at the deepest level. A complete depth-first search can require as many as 780 movements, usually more than the model can handle within the 8,192 work tokens available to each Agent. All members start in the same location and see the same ordering of door lists. This setup preserves the risk of repeated exploration by homogeneous models. Effective communication contains two types of information: declaring which branch one is examining, and reporting which branch has been completely searched. Based on this information, a receiver can choose another door or exit a path that a partner has already eliminated. The exploration experiment used Qwen3-1.7B. Unlike the number-aggregation task, branch declarations and search reports were first provided through scripted demonstrations, followed by 50 SFT updates and 200 RL updates. The demonstrations came from a simpler depth-two room tree, and RL then moved to the depth-three task. Independent Agents also received non-communicating demonstrations with corresponding room trees and random seeds. The key message templates in exploration retained the influence of the demonstrations, so the origins of organizational behavior must be understood in light of both stages. On the four-agent test, s³ achieved a success rate of approximately 89.6%, while the control that retained team rewards but disabled communication achieved approximately 70.3%, a difference of around 19.3 percentage points. The proportion of room visits shared by communicating members was close to half that of the non-communicating control. This overlap metric also includes shared corridors required by different branches, so it measures overlap in visits and cannot be directly converted into total computational savings. Figure 5: Retaining team rewards while disabling communication substantially reduces the four-agent exploration success rate. Compared with independently trained best-of-4, s³’s success rate was approximately 9.4 percentage points higher. In the paper’s statistical tests, this four-agent difference was significant before multiple-comparison correction but not after it; at six agents, an advantage of approximately 7.3 percentage points did pass correction. The benefit of communication must be understood in conjunction with the specific controls and statistical conditions: the evidence from the communication-disabled ablation is relatively clear, while the advantage over best-of-N varies with team size. 4.3 Communication Patterns If messages merely restate the task, they will not naturally eliminate repeated searches. The authors grouped the exploration problems by which door at the first fork contained the exit. Teams with communication disabled performed well when the exit was behind the first door but markedly worse when it was behind the third door; communicating s³ teams had more similar success rates across different exit positions. This indicates that messages genuinely participated in allocating the search space. Figure 6: Information aggregation tends toward a one-broadcast protocol, while communicative exploration performs more evenly across exit branches. The final communication structure also shows that the two types of task did not converge on the same message frequency. Number aggregation tended toward one broadcast per person; exploration still required continuous branch declarations, result reports, and reading updates. In the figures, sending concentration indicates whether a small number of members perform most of the sending; reciprocity indicates whether communication flows in both directions; and the number of message calls reflects the frequency of exchange. One broadcast counts as one call even though it produces deliveries to multiple receivers. Figure 7: The two types of task develop different communication frequencies, with s³ exhibiting a more balanced division of sending responsibilities and two-way exchange. There is another result with implications for engineering choices: in the exploration task, communicating teams rewarded for individual success also performed well, with no significant difference in success rate detected between them and teams using shared rewards. Because branch information provided by partners can help an Agent find the exit for itself, individual benefit and information utilization are already partly aligned. Number aggregation, by contrast, requires each person to contribute information that benefits only partners, making the role of team rewards more prominent. At this point, “role” can be understood more concretely: it is the work a member currently undertakes based on its private state and the messages it has read, such as examining a branch, providing its own number, or using a partner’s report to turn toward another path. The key to organization lies in complementary work. Depending on the task, organization may produce a division of labor or a shared protocol followed by every member. 5. Conditions for Scaling 5.1 Scale Transfer The training teams for s³ were fixed at four Agents, while evaluation placed the same policy into teams of up to sixteen members without retraining. For number aggregation at eight members, the success rate was approximately 55%, compared with approximately 2% for the online SFT control, corresponding to the “more than 25×” comparison in the abstract. The advantage persisted through twelve members, although all methods struggled at sixteen. This result shows that the exchange patterns learned by four-person teams can transfer to larger scales, while absolute success rates continue to decline as aggregation becomes more difficult. The exploration curve has a different meaning: independent best-of-N also approaches near-certain success as the number of members increases; s³’s value lies mainly in reaching a high success rate earlier with fewer members. Four communicating Agents are close to six independent Agents, and six are close to eight. When the team is large and independent search almost always succeeds, there is naturally less room for improvement. Figure 8: Policies learned by four-agent teams transfer to larger teams, while the scale curves of the two task types present different constraints. The left column of the figure shows success rate, while the right column defines token efficiency as “success rate divided by the average proportion of team allowance used,” with failed attempts included in average consumption. When comparing different team sizes, the team allowance N × L is itself increasing, so efficiency values must be interpreted alongside team size and absolute token consumption. The search-coverage gains from expanding a team and the gains from organizational mechanisms must be observed separately. Why does information aggregation become harder as team size increases? Suppose one necessary information transfer consists of sending, reading, and correctly recording information, with an overall reliability q, and assume the transfers are independent. The probability that one member receives information from all N−1 other members is q^(N−1). Considering every member who could potentially complete the aggregation yields an upper bound on success probability of min(1, N × q^(N−1)). This simplified analysis shows that each transfer must become more reliable as the team grows. Context is another constraint. If every person broadcasts once, this produces N × (N−1) deliveries, and every receiver must process N−1 pieces of information. The model must learn a scalable rule for “waiting until messages from all different senders have arrived” and must continually maintain information attribution. A small team’s communication protocol can transfer, but reliable transmission, state tracking, and context capacity still determine how far it can scale. 5.2 Limits of the Evidence The paper’s budget-matched comparison aligned the training problems, number of Agent trajectories collected per update, optimizer, and generation allowance per Agent. For example, each exploration-training update collected 128 four-person teams, totaling 512 Agent trajectories; independently trained learners collected 512 single-Agent attempts. During evaluation, teams of the same size received the same per-Agent allowance, while actual generation volume was determined by each policy. This allows “obtaining more training opportunities” to be examined separately from “using communication more effectively.” At the same time, these experiments covered two verifiable tasks, two small models, and a single training run for each condition. Each held-out test contained 32 problems and six evaluation random seeds; these seeds describe evaluation variance for the same training result. More complex tasks and larger models require further validation of the learned organizational patterns. Demonstration data is also a substantive condition. In the exploration task, removing SFT left four-agent teams with no advantage over best-of-4 after the same number of RL updates. The results support a path in which the model is first given a starting point for effectively using task and communication tools, after which result feedback optimizes collaboration. The number-aggregation experiment demonstrates the possibility of forming a concise protocol starting from the base model and tool prompts alone. A further question is the trustworthiness of messages. The authors’ trajectory analysis found that some explorers reported a branch as “completely searched” before finishing it, potentially causing partners to skip the path that actually contained the exit. Among failed s³ exploration teams, approximately 32% had at least one member read an erroneous exclusion report about the exit branch. This proportion describes an association within failed samples and reveals the next organizational responsibility: it is necessary not only to know what a partner claims to have done, but also to judge which checks support that conclusion. This leaves a clear interface question for engineering systems: how can the source of a discovery, the scope already covered, and the unverified portions be represented so that receivers can choose to reuse the information, continue checking, or raise a challenge? s³’s existing experiments primarily handle these relationships through natural-language messages. More stably incorporating evidence state into collaboration should help reduce the propagation of erroneous reports within a team. 6. Conclusion Returning to the opening question, for more Agents to form complementary work, information must cross private boundaries, promptly influence the next decision, and allow contributors to receive a learning signal from team success. s³ uses an asynchronous runtime to create opportunities for communication and uses relative team advantages to jointly train task and communication actions, demonstrating that this combination can form effective protocols for small models on verifiable tasks and transfer to larger teams. The insight worth retaining is that organizational capability can enter policy training alongside problem-solving capability. The engineering framework handles state isolation, message delivery, and budget management, while the model learns how, in a specific context, to share information, read messages, and allocate current work. The next questions about scaling consequently become more concrete: can members reliably track information attribution, judge the basis for a partner’s reports, and continue turning that information into complementary action as the team grows? References • Kunal Jha et al. s³: Simple Scaling of Swarms via Cooperative Multi-agent RL: https://arxiv.org/abs/2611.08615
Comments
…