JuiceFS Adds Fine-Grained Data Warming for Lance Datasets
JuiceFS has begun supporting fine-grained data warming for Lance datasets, according to pull requests #7398 and #7399. The report describes Lance as organizing and selecting data, while JuiceFS uses a train list and local NVMe caching to prewarm required files near GPU nodes; uncached data can still be fetched from object storage. It also describes an unnamed leading autonomous-driving company using Lance, JuiceFS, and object storage, while noting that publicly documented production practices and cases for the combination remain limited.
The integration addresses two data-access issues described in the report: Lance can reduce unnecessary reads by selecting relevant fields, while JuiceFS can reduce waiting for data by placing required files near compute nodes. The combination is presented as most relevant when both read amplification and data-access latency are significant I/O constraints.
随着多模态 AI 和模型训练规模扩大，AI 数据链路需要处理的数据越来越多，类型也越来越复杂。训练任务可能只需要一部分字段，却要从大规模数据集中筛选和读取；数据分布在不同云和数据中心时，计算节点还可能因为访问延迟而等待。这带来了两个问题：怎样减少不必要的数据读取，以及怎样让需要的数据及时到达计算节点。
01 Lance 和 JuiceFS：分别解决什么问题？
Lance 是一个面向多模态 AI 的开源列式湖仓格式（lance.org），提供列式存储、多模态 Schema、向量索引和版本管理等能力，帮助用户组织图片、视频、文本和 Embedding 等数据。它关注的是：任务需要哪些数据，怎样避免读取无关内容。
以自动驾驶数据为例，一条样本可能包含大量标注属性，但一次训练只会用到其中一部分。采用列式存储后，任务可以只读取需要的字段，减少无效数据读取，也让标注属性的更新和迭代更加灵活。