Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

CueCaps Studio Runs Whisper and BERT Captioning in Browser

AI summary

Creator u/Pantheon3D introduced CueCaps Studio, a browser-based tool that transcribes videos with onnx-community/whisper-base_timestamped through transformers.js and tags captions with an approximately 11M-parameter BERT-mini model. It uses WebGPU with FP32 on real GPUs and WASM with an 8-bit decoder elsewhere, while software WebGPU adapters fall back to WASM. The creator says the learned tagger, which handles emphasis, line breaks, punctuation, and capitalization, outperformed hand-written rules on held-out data, but the tool remains English-only and offers cloud transcription for noisy audio.

In-depth insight

The release demonstrates a compact browser-local pipeline combining speech recognition with a distilled caption-tagging model, providing a concrete option for creators who want to avoid uploading every render to a caption service.

i make Blender tutorials as shorts and got tired of uploading every render to a caption service, so i made a caption tool that runs the models in the browser instead transcription is onnx-community/whisper-base_timestamped through transformers.js. real GPUs get WebGPU in fp32, everything else gets WASM with the decoder in q8. software WebGPU adapters turned out to be slower than WASM so those get WASM too the part i had the most fun with is the caption tagger: which words get emphasis, where lines break, and fixing missing punctuation and capitals. it started as hand written rules, now it’s a BERT-mini (about 11M params, 8 labels per token, ~12mb) running in q8 on WASM. the training data came from LFM3-Omni-2B on llama.cpp on my own pc, about 190 scripts a minute on a 4070 ti super with 128 parallel slots. it only got to replace the rules because it beat them on held out data whisper base isn’t as accurate as the bigger ones so there’s a cloud option for noisy audio, and the tagger is english only for now has anyone gotten whisper small or turbo running in the browser at a usable speed? that’s what i want to try next :) https://cuecaps.com/studio?ref=reddit if you want to see it run (i’m the creator) also thank you for reading, it means a lot to me submitted by /u/Pantheon3D [link] [comments]

#CueCaps Studio#onnx-community/whisper-base_timestamped#transformers.js#BERT-mini#LFM3-Omni-2B#llama.cpp#WebGPU#WASM

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.