Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Stanford and Together AI propose parallel coarse-to-fine text generation

AI summary

Stanford University and Together AI researchers proposed Next-Concept Modeling (NCM), an approach that differs from autoregressive next-token prediction and diffusion-based generation. NCM first generates a coarse concept representation, then predicts finer levels of discrete text codes in parallel before decoding the result. In the paper’s experiments, NCM generated text 10 to 31 times faster than diffusion language-model baselines, with comparable or better quality on the tested tasks. It also outperformed autoregressive and diffusion baselines on difficult Sudoku puzzles with 30 clues, with an 86.1% complete-solution rate compared with 81.3% for FMLM+. The evaluated generator was about 100 million parameters, with experiments extending to 725 million parameters. Performance at larger scales, on exact-text tasks and on long-form generation still requires validation.

In-depth insight

The reported results establish NCM as a distinct generation approach with substantial speed gains over diffusion baselines in the tested settings. Its broader usefulness remains dependent on validation at larger scales and on exact-text and long-form tasks.

如今的大模型再聪明,输出内容也是一个字一个字往外蹦。有没有一种可能:真正的 AGI 不是这样?

近日,斯坦福大学与 Together AI 的研究者提出了一种既不同于自回归,也不同于扩散的第三种文本生成范式。他们把 NeurIPS 2024 最佳论文 VAR 在图像上的「下一尺度预测」配方搬进文本,让模型先并行写出几句话的「概念大纲」,再逐层并行填充细节,直到还原成完整文字。

于是新模型就叫 Next-Concept Modeling(NCM,下一概念建模),对主流 Next-Token Modeling(下一 token 建模) 来了个正面叫板。

在若干实验中,NCM 以相近或更好的质量,实现了相对扩散语言模型 10—31 倍的生成速度。

#Stanford University#Together AI#Next-Concept Modeling (NCM)#Visual Autoregressive Modeling (VAR)

Original · 4 min readRead the original
Group chat

No comments yet. Start the conversation.