Recursive Evolution•Florian•BOXBOX: Evolving the Execution Framework During a Task
Performance optimization usually starts with a clear goal: completing the same work faster. What really takes time is identifying bottlenecks, designing experiments, comparing candidates, and then determining whether the differences come from the code or from measurement noise. After handing this process to an Agent, there is another kind of cost: repeatedly organizing logs, waiting for long-running commands, and reconstructing conclusions that have already been reached. The longer the task, the more worthwhile it becomes to optimize the execution method itself. Researchers from Nanjing University, Zhejiang University, Birkbeck, University of London, and Microsoft proposed BOXBOX, which incorporates this problem into the Agent architecture: the model remains fixed, while the Agent modifies its own Harness based on experience with the current task, then continues execution with the existing context. Here, the Harness is the executable framework surrounding the model, responsible for organizing inputs, providing tools, handling feedback, and managing state. The key to understanding BOXBOX is to distinguish what can change from what must remain continuous throughout the change process. 1. Execution Bottlenecks 1.1 A Fixed Loop Let us first look at a minimal Agent: the model chooses a command based on the history, the Harness executes that command in the terminal and sends the output back to the model, and the model then decides what to do next. The experiments in BOXBOX begin with a terminal-based Harness such as Terminus-KIRA. It is sufficiently general to build projects, run workloads, and collect results, while also leaving many mechanical steps for the model to arrange one turn at a time. For example, a build command has not finished, but the tool has already returned. The model reads through the context and issues a “continue waiting” command. After receiving the timing log, it then writes commands to extract the results and compare versions. Each step can be completed, yet it may repeatedly consume reasoning calls. The project code determines the runtime cost of a workload, while the Harness determines the interaction cost the Agent incurs in finding better code. The two form two optimization loops within the same long-running task. Figure 1: Project optimization changes the code being tested, while Harness improvement changes the execution method used to find the optimization. Increasing the runtime gives the Agent more opportunities to experiment; encapsulating repetitive steps that have already become stable as tools enables it to repeat less and compare better within the same amount of time. The question then becomes: who discovers these repetitions, and who decides when modifying the execution framework is worthwhile? 1.2 Experience Enters the Picture One approach is to optimize the Harness before the task begins. Researchers collect trajectories from training tasks, analyze failures, modify the framework, and then use selected versions for new tasks. BOXBOX treats this approach as Harness Evolution, or HE, for comparison. Its advantage is that experience can be accumulated in advance, but the challenge is that problems encountered during training may differ from the current task’s actual constraints. The paper provides a concrete failure. To prevent invalid submissions, HE added checks: candidate versions must have a Git commit, repeated timing results, and evidence of correctness. In one test task, the Agent placed these artifacts in the editable repository, but the Harness checked Git status in the parent working directory. As a result, submission requests continued to be blocked; the trajectory recorded 473 blocked requests over approximately 3 hours and 46 minutes. Patching the evidence files could not fix the problem that the framework had selected the wrong directory. This example separates two levels: while the Agent is working on problems within the project, it may also be blocked by its own execution rules. The current trajectory is both task progress and material for diagnosing the Harness. BOXBOX therefore enables the task Agent to request a framework review itself, bringing “continue optimizing the project” and “first improve the execution method” into the same budget decision. 2. Stability and Variability 2.1 The Kernel Boundary Once modifications to the executable framework are allowed, a more fundamental question must be addressed: who takes over the task after the modified program is ready? If the new framework cannot start or encounters an error while importing state, recovery must remain outside it. BOXBOX separates the system into an editable Harness and an independent Kernel. The Harness is the Agent’s implementation, and the Runtime is the version of that implementation currently running; the Kernel manages control transfer during review, candidate validation, hot reload, and recovery. The Agent can modify prompt rules, tools, execution loops, and state management, while the Kernel remains outside this editing path. Figure 2: The Agent requests an inbound review, and the Kernel validates the new framework and continues the original task. The specific entry point is the boxbox() tool. Calling it does not require the Agent to prepare a patch in advance; instead, it transfers control to the Kernel. The entire transition can be divided into four steps: Save the scene: Preserve the task conversation, review history, and records of reviewed trajectories; the original terminal connection and project workspace continue to exist. Review separately: Start a review session using the current Harness, read the task trajectory and Git history, and modify only the Harness, with project code serving as read-only material. Validate and switch: Inspect the committed candidate, load the new Runtime, and confirm that it supports task execution and context migration; import using a state copy, then hand over the original terminal session on success. Continue or recover: If the update succeeds, continue with the new framework; if there were no modifications or activation failed, preserve or restore the old Runtime and context. The “scene” saved here has two parts: state held by the framework can be exported as a snapshot, while the task environment remains in place through the existing connection. A normal hot reload pauses only the Agent’s task decision loop; existing processes in the container can continue running. This separation makes switching frameworks and restarting the project environment two distinct operations. Runtime failures follow a different recovery path. A new Runtime remains in a pending-confirmation state until it reaches the next fault-free Agent-initiated review. If a failure occurred beforehand, the Kernel can restore the last confirmed version and its state, then launch a review based on the failure record. Failure recovery establishes a new terminal, so shell-local state is lost, while project files, framework history, and execution records are preserved. What a normal switch and a failure recovery each preserve directly determines where the task can resume. 2.2 The Recursive Loop At this point, the Harness can already be replaced during a long-running task. The “recursion” in BOXBOX comes from the new Harness’s combined influence over two types of subsequent activity: it controls both task execution and later review sessions. After the first review improves the tools, the Agent accumulates experience under the new tools; the next review then reads that experience and determines whether further modification is needed. Self-improvement therefore forms an “execute—observe—revise—reexecute” loop. The model parameters remain unchanged; changes enter the organization of model inputs, the available actions, and the processes by which actions are executed and feedback is returned. The recursion occurs at the execution-framework layer. This loop also has pacing constraints. The paper’s initial Harness required task activity between two reviews and set a 30-minute cooldown after each review. Once those conditions were met, the Agent decided whether to request the next review. A review could preserve the status quo, provide targeted feedback, or add reusable capabilities. The justification for modification is its expected contribution to the remaining task, while preserving the existing framework is also a valid decision. 3. Turning Experience into Tools 3.1 A Measurement Assistant The framework can be modified, but what specifically should be changed? The CCC case in the paper provides an easily understood evolutionary path. The task was to accelerate the compilation of SQLite using this C compiler. The Agent had to repeatedly run the compilation, analyze the time spent in each stage, and compare code changes. The first review found that the Agent had been manually processing timing output, so an assistant was added to handle repeated runs, record results, and compare candidates in a unified way. After it had been used for a while and the Agent still had to manually extract per-stage timing, the second review added stage summaries to the same assistant. The tool’s scope expanded gradually as actual use exposed repetitive work. The more important change occurred in the measurement method. One code modification appeared to reduce compilation time by 9.4% when measured in separate batches; after switching to alternating runs of the unmodified version and the candidate version, their median runtimes were almost identical. The third review incorporated this paired comparison into the assistant. It was subsequently used to eliminate multiple modifications that lacked stable gains before the search continued for effective optimizations within the compiler. Figure 3: CCC’s measurement assistant gradually adds repeated runs, stage summaries, and paired comparisons. The performance leap shown in the figure came from modifying the project code; the review improved the experimental process supporting those modifications. The value of experience lies in changing how the next round generates and filters evidence. Once paired comparison becomes a reusable tool, the measurement method can continue to contribute to subsequent candidates without requiring the model to arrange it again each time. 3.2 Waiting and Context Other forms of experience are better suited to the execution logic. When optimizing Rust Clippy, long-running build and benchmark commands returned early, causing the model to issue waiting calls repeatedly. One Harness update extended the internal waiting window for ordinary commands to 180 seconds, while remaining subject to the existing block timeout constraint. The key logic shown in the paper is: AUTO_COMMAND_WAIT_SEC = 180 wait_budget = min( BLOCK_TIMEOUT_SEC, max(command.duration_sec, AUTO_COMMAND_WAIT_SEC), ) The tool checks for a completion marker within the window and returns once the command completes; if the window expires, it returns a progress snapshot. The tool description is updated accordingly, telling the Agent that it only needs to continue waiting when a command exceeds the window. Status checks that previously required repeated action from the model thus become deterministic polling inside the tool. The FluidSynth case advances in the same direction: first, the waiting capability of the existing command executor was extended, and then benchmark_matrix was added to run batches of labeled experiments and report runtimes and exit statuses. A single call can combine a build, multiple renders, and output checks, allowing the model’s judgment to focus on experimental results. Context management faces similar constraints. During one optimization of Trixi, the review record showed that later requests contained approximately 145,000 input tokens, and the existing summarization mechanism did not sufficiently limit the growth of history. The updated execution loop added an entry point for proactive compression based on message count, reusing the existing summarizer; if summarization succeeds, it prepares information for continuation, while a failure leaves the original history intact. These modifications span tool implementations, tool descriptions, and execution loops. Predefining the editable scope to a single component would make some changes difficult to implement fully. At the same time, continuity requires that key decisions and experimental conclusions can be carried forward, even though the specific representation of state may be adjusted as the task requires. Hot reload preserves the scene, while proactive compression reduces the cost of subsequent processing; the two address different constraints. 4. The Scale of the Benefits 4.1 The Evaluation Loop These cases explain how modifications can help, but performance gains still require consistent comparison conditions. For this purpose, the paper constructed PerfRace: each task provides a real software project, a fixed workload, and an external Judge, with the Agent searching for bottlenecks from an end-to-end workload perspective. Tasks do not have a predefined target patch; the optimization space is determined jointly by the project and the workload. PerfRace contains 100 tasks spanning seven domains, including developer tools, data systems, and scientific computing. Projects were first filtered, after which Agents and humans worked in pairs to construct and review tasks, checking whether the workloads served a practical purpose, whether repeated measurements were reliable, and whether implementations that did no work or skipped work could fool the scoring. In the main experiment, 40 tasks were used for HE training, 10 for validation, and the remaining 50 served as the common test set for all methods. Figure 4: PerfRace tracks performance improvements that preserve correctness using fixed workloads and isolated evaluation. Upon submission, the Judge measures the project snapshot in an independent shadow container, using a separate copy of the workload and additional hidden correctness test cases. The validation code and expected outputs are not present in the Agent’s workspace. Correct submissions are scored using “the unmodified project’s metric ÷ the candidate’s metric”; for time-based tasks, this is the usual speedup ratio. Incorrect submissions receive zero, and the unmodified project starts at 1×. The paper tracks the best correct speedup ratio achieved as of a given point in time. A failed experiment does not erase the best result already obtained, but it consumes available time. The four-hour budget includes task execution, Harness reviews, and Judge validation, but excludes environment initialization. Thus, modifying tools, running experiments, and requesting feedback all carry a time cost, allowing the researchers to determine whether framework changes are worthwhile. 4.2 Time and Cost All four methods begin from the same initial Harness, and both task execution and Harness optimization use GPT-5.6 Luna with reasoning effort set to high. Each method was run independently three times on every test task, with each run lasting four hours. Frozen keeps the framework fixed throughout; HE uses a framework selected offline; Harness Scaling, or HS, uses an external controller to modify the framework after one task attempt ends, then starts a new session, preserving project-code progress while resetting the execution context. For the three runs, the paper first takes the geometric mean of the speedup ratios within each task and then takes the harmonic mean across the 50 tasks to obtain Speedup@4h. The precision in the main results table is retained below; model costs are calculated using the token usage and pricing recorded in the paper, averaged over each test run and including Harness optimization calls during testing. Method Speedup@4h Model Cost Speedup/$ Frozen 1.442× $23.22 0.2142 HE 1.281× $24.56 0.1243 HS 1.539× $12.98 0.2872 BOXBOX 1.702× $16.43 0.4886 Figure 5: Under the same task time and cumulative model cost, BOXBOX achieves a higher aggregate speedup. The latter half of the results is the most explanatory. Frozen’s aggregate speedup increased from 1.410× at two hours to 1.442× at four hours; HS increased from 1.431× to 1.539×; BOXBOX increased from 1.530× to 1.702×. For this set of tasks, online, task-specific framework improvement can bring additional gains while execution continues. The cost comparison requires context. BOXBOX’s average model cost was approximately 29.3% lower than Frozen’s, while it cost more than HS and achieved a higher speedup. HS’s session resets can reduce the input cost accumulated from history, but later attempts may need to reconstruct previous judgments and experimental conclusions. BOXBOX preserves context, trading that benefit for the ability to continue carrying its history so it can optimize directly from existing results. The paper also reports Speedup/$: first calculate the ratio of the average speedup to the average cost for each task, then take the arithmetic mean across tasks. Under this metric, BOXBOX scores 0.4886 and HS scores 0.2872. It emphasizes the relationship between task-level gains and cost, and cannot be reproduced by directly dividing the aggregate speedup in the main table by the global average cost. The main-table cost for HE does not yet include offline training and validation; after including the existing records, the amortized cost is at least $33.73 per test run. The shaded regions in the figure are 95% intervals obtained by resampling the three runs while holding the fixed set of 50 tasks constant. Therefore, these results describe performance with this model, this starting Harness, and these workloads; the benefits across models and a broader range of tasks require further experimentation. 4.3 Triggers and Scope The main results compare the complete approach, while the ablation study further separates two design choices. The researchers randomly selected 14 test tasks. When reviews were scheduled every 30 minutes while full editing remained allowed, the speedup was 1.508×; when Agent-triggered reviews were retained but the editing scope was restricted, it was 1.349×; the complete BOXBOX on the same subset achieved 1.584×. These figures come from 14 tasks and should be compared within that subset. The constraint of scheduled reviews is that they may interrupt the continuous process of forming hypotheses, implementing changes, and validating results, and they may consume calls even when the framework does not need modification. Agent-initiated triggers leave the timing of modifications to the executor that is experiencing those difficulties. Full editing addresses a different class of problems: modifying command waiting behavior requires changes to both the tool implementation and its description, while modifying the conditions that trigger summarization requires entering the execution loop. Figure 6: Reviews consume a small amount of task time, while modifications are distributed across layers such as prompts, tools, and execution logic. Across 150 BOXBOX test runs, the median number of reviews was five, and 92% of runs contained three to six reviews, including both initiated reviews and failure-recovery reviews. Review time accounted for a median 2.1% of the four-hour budget, while review cost accounted for a median 13.8% of total model cost. The researchers also analyzed 620 applied, non-empty framework updates covering categories such as prompt rules, skills, tools, execution logic, and failure handling; a single update may involve multiple categories. These observations and cases point to a shared design tradeoff: hand work that has already recurred and whose need for evidence has become stable to tools, while leaving experimental choices and interpretation of results to the model. Determining when encapsulation is worthwhile and how far it should go requires judgment based on subsequent use, and the current task trajectory is the direct material for that judgment. 5. Conclusion Returning to the opening question: in a long-running task, an Agent can do more than improve the project being tested; it can also improve the execution method that produces those modifications. BOXBOX turns this feedback path into a system mechanism: the task Agent decides when to request a review, uses current experience to modify the Harness, and an independent Kernel preserves state, validates the switch, and continues the task. Its most noteworthy architectural principle is to separate the mutable execution policy from the mechanisms responsible for maintaining task continuity. The results on PerfRace support the value of this approach for four-hour performance optimization: measurement assistants reduce mechanical organization, internal waiting reduces unnecessary calls, and state continuation allows subsequent exploration to inherit existing conclusions. The Kernel’s activation checks primarily handle runtime operation and state migration; actual performance gains are still realized through task experiments. The paper also positions the system as a prototype in an isolated research environment; production use would require independent permissions, sandboxing, security reviews, and monitoring to constrain modifications to executable code. The question that now deserves serious attention is which changes formed during the current task can be transferred to other tasks, and which are suited only to the workload at hand. BOXBOX has placed “how to do this better” within the Agent’s action space, but how long these experiences remain useful and how broadly they apply will determine how far recursive Harness improvement can go. References • BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization: https://arxiv.org/abs/2611.09218
Comments
…