Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Claude Weight Edits Improve Silencio-2 Voice Controls

AI summary

Developer N1kuson reports using Claude Opus 5.5 to diagnose and directly edit weights in Silencio-2, a small finetune of Kyutai's Pocket TTS with inline emotion and sound tags. The edit used no retraining or gradient steps and improved held-out whisper reliability from 5/24 to 22/24 attempts, while emotion separability rose from 0.45 to 0.71. The developer also documented glitches, slurred whispers, and metric limitations in the self-reported experiment.

In-depth insight

The experiment demonstrates a reported use of an LLM agent for interpretable, direct weight editing of a speech model, offering a possible alternative when additional tagged training data is scarce.

I'm building Silencio-2, a small finetune of Kyutai's Pocket TTS. It adds inline tags for emotions (<happy>, <sad>, <angry>, <fear>, <surprised>, <whisper>) and for sounds like [laugh]. I simply didn't have enough tagged data to make them stable. <happy> worked, the others were hit or miss, and <whisper> gave an actual whisper in about 1 of 5 takes.

So instead of collecting more data, I tried the approach from this post on weight-space programming: an LLM agent inspects the model's weights, finds the mechanism behind the failure, edits a small number of weights directly, and keeps an edit only if it improves a held-out set. Here the agent was Claude (Opus 5.5). It wrote the probes, ran the experiments and built the edit. I steered and listened to the results.

What we found

  • The tag embeddings barely moved from the words they were initialized from: 60–80% cosine similarity to "sad", "whisper" and so on. Their norm doesn't matter either, because LayerNorm sits right before attention.
  • An attention probe showed that audio frames attend to <happy> 100–3000× more than to an average text token. For the other tags it's only 1–10×. The model seems to know what the tags mean, it just barely reads them.
  • Causal check: I boosted attention to <whisper> at inference time, directly in the activations (×30, all heads). That gave a whisper in 62% of takes with clean words. ×100 breaks speech. Boosting only the 3 "reader" heads did almost nothing.

#Silencio-2#Kyutai Pocket TTS#Claude Opus 5.5

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.