Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

EMR Serverless StarRocks Adds Paimon Global Index Support

AI summary

Alibaba Cloud EMR Serverless StarRocks now supports Paimon Global Index for locating data through Row IDs before reading relevant rows. The platform offers BTree indexes for scalar filtering, Vector indexes for semantic retrieval, and a combined path that applies business filters before vector search. Its two-stage execution architecture searches index shards in parallel, then reads candidate data through Paimon Native Reader.

In-depth insight

The capability enables structured filtering and semantic retrieval in one SQL query, including RAG document recall and product search, while avoiding unnecessary reads of unrelated lake data.

在湖仓中,查询需求并不总是大范围的统计分析。有时,我们只想根据订单 ID、设备 ID 找到少量记录;有时,则需要从海量商品、文档或图片向量中,召回最相似的 Top-K 结果。

以商品搜索为例,一次查询可能既要限定租户、品类和价格范围,又要找出与用户输入语义最接近的商品。这要求查询既能高效过滤业务条件,也能完成向量检索。

分区裁剪、Min/Max 统计等优化,可以帮助查询跳过不相关的文件。但对于这类只需要返回少量结果的查询,剩余文件中仍可能包含大量不符合条件的数据。能否进一步缩小读取范围,成为提升查询效率的关键。

Paimon Global Index 提供了更细粒度的数据定位方式:先通过索引找到目标数据的 Row ID,再定位并读取对应数据。基于这一能力,EMR Serverless StarRocks 支持通过 BTree 索引进行标量过滤,通过 Vector 索引进行向量检索,也支持将两者结合,先限定业务范围,再执行语义检索。

#Alibaba Cloud EMR Serverless StarRocks#Paimon#Stella#Paimon Native Reader#BTree Global Index#Vector Global Index

Original · 13 min readRead the original
Group chat

No comments yet. Start the conversation.