Apple publishes tri-modal masked diffusion model study
Apple's machine learning research team presented a NeurIPS paper introducing what it describes as the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text and audio-text data. The work systematically studies multimodal scaling laws, modality mixing ratios, noise schedules and batch-size effects, and proposes an SDE-based reparameterization that decouples the physical batch size chosen for compute reasons from the logical batch size used to balance gradient variance. The authors also pretrain a preliminary 3B-parameter model on 6.4T tokens that shows strong results in text generation, text-to-image and text-to-speech tasks, describing the effort as the largest-scale systematic open study of multimodal discrete diffusion models to date.
Discrete diffusion is a notable alternative to autoregressive language modeling, and this open study spells out multimodal training and scaling details that others can build on.