CueCaps Studio在浏览器内生成视频字幕
AI 总结
开发者u/Pantheon3D推出CueCaps Studio,通过transformers.js调用onnx-community/whisper-base_timestamped转录视频,并用约1100万参数的BERT-mini模型标注字幕。该工具在真实GPU上使用WebGPU和FP32,其他环境使用WASM与8位解码器,软件WebGPU适配器也回退至WASM。开发者称学习型标注器在留出数据上优于手写规则,但目前仅支持英语,嘈杂音频仍需云端转录选项。
深度见解
这一项目展示了将语音识别与蒸馏版字幕标注模型组合成紧凑浏览器端流程的实践,为不想把每次渲染都上传到字幕服务的创作者提供了具体方案。
| i make Blender tutorials as shorts and got tired of uploading every render to a caption service, so i made a caption tool that runs the models in the browser instead transcription is onnx-community/whisper-base_timestamped through transformers.js. real GPUs get WebGPU in fp32, everything else gets WASM with the decoder in q8. software WebGPU adapters turned out to be slower than WASM so those get WASM too the part i had the most fun with is the caption tagger: which words get emphasis, where lines break, and fixing missing punctuation and capitals. it started as hand written rules, now it’s a BERT-mini (about 11M params, 8 labels per token, ~12mb) running in q8 on WASM. the training data came from LFM3-Omni-2B on llama.cpp on my own pc, about 190 scripts a minute on a 4070 ti super with 128 parallel slots. it only got to replace the rules because it beat them on held out data whisper base isn’t as accurate as the bigger ones so there’s a cloud option for noisy audio, and the tagger is english only for now has anyone gotten whisper small or turbo running in the browser at a usable speed? that’s what i want to try next :) https://cuecaps.com/studio?ref=reddit if you want to see it run (i’m the creator) also thank you for reading, it means a lot to me submitted by /u/Pantheon3D [link] [comments] |
#CueCaps Studio#onnx-community/whisper-base_timestamped#transformers.js#BERT-mini#LFM3-Omni-2B#llama.cpp#WebGPU#WASM