Claude Weight Edits Improve Silencio-2 Voice Controls
Developer N1kuson reports using Claude Opus 5.5 to diagnose and directly edit weights in Silencio-2, a small finetune of Kyutai's Pocket TTS with inline emotion and sound tags. The edit used no retraining or gradient steps and improved held-out whisper reliability from 5/24 to 22/24 attempts, while emotion separability rose from 0.45 to 0.71. The developer also documented glitches, slurred whispers, and metric limitations in the self-reported experiment.
The experiment demonstrates a reported use of an LLM agent for interpretable, direct weight editing of a speech model, offering a possible alternative when additional tagged training data is scarce.
I'm building Silencio-2, a small finetune of Kyutai's Pocket TTS. It adds inline tags for emotions (<happy>, <sad>, <angry>, <fear>, <surprised>, <whisper>) and for sounds like [laugh]. I simply didn't have enough tagged data to make them stable. <happy> worked, the others were hit or miss, and <whisper> gave an actual whisper in about 1 of 5 takes.
So instead of collecting more data, I tried the approach from this post on weight-space programming: an LLM agent inspects the model's weights, finds the mechanism behind the failure, edits a small number of weights directly, and keeps an edit only if it improves a held-out set. Here the agent was Claude (Opus 5.5). It wrote the probes, ran the experiments and built the edit. I steered and listened to the results.
What we found
- The tag embeddings barely moved from the words they were initialized from: 60–80% cosine similarity to "sad", "whisper" and so on. Their norm doesn't matter either, because LayerNorm sits right before attention.
- An attention probe showed that audio frames attend to
<happy>100–3000× more than to an average text token. For the other tags it's only 1–10×. The model seems to know what the tags mean, it just barely reads them. - Causal check: I boosted attention to
<whisper>at inference time, directly in the activations (×30, all heads). That gave a whisper in 62% of takes with clean words. ×100 breaks speech. Boosting only the 3 "reader" heads did almost nothing.