SQAM: Q-learning with scalar adjoint matching for flow policies
An arXiv preprint proposes Q-Learning with Scalar Adjoint Matching (SQAM). The authors observe that the batch-averaged velocity Jacobian of pretrained flow policies concentrates on its diagonal, and use this to derive a closed-form scalar adjoint that scales the value gradient at the final action by flow time, removing per-step vector–Jacobian products. SQAM pairs this with a value penalty at policy-generated actions. It beats the strongest baseline in each of the four hardest OGBench domains by 18 to 35 percentage points, and when fine-tuning a vision-language-action policy on a real bimanual robot it improves over supervised fine-tuning on all three tasks.
If the simplification holds, it could cut the compute cost of fine-tuning flow policies and robot policies with off-policy reinforcement learning.
Computer Science > Machine Learning
arXiv:2610.10437 (cs)
[Submitted on 7 Oct 2026]
Title:Q-Learning with Scalar Adjoint Matching
Authors:Yonghoon Dong, Minsung Yoon, Jaehyuk Kim, Jungwoo Park, Changyeon Kim, Jinwoo Shin
View a PDF of the paper titled Q-Learning with Scalar Adjoint Matching, by Yonghoon Dong and 5 other authors