Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

lance

lance: An open lakehouse format for multimodal AI.

What problem it solves

AI and machine learning workflows require vector search, fast random access, multimodal data storage, data evolution, and hybrid search combining vectors, full-text, and SQL predicates. Lance brings these capabilities directly into the lakehouse format.

Key features

  • Expressive hybrid search — Combine vector similarity search, full-text search with BM25, and SQL analytics on the same dataset using accelerated secondary indices.

    Source: README

  • Native multimodal data support — Store images, videos, audio, text, and embeddings in one format with blob encoding and lazy loading.

    Source: README

  • Data evolution — Add columns with backfilled values without full table rewrites, supporting feature engineering workflows.

    Source: README

  • Zero-copy versioning — Automatic versioning provides ACID transactions, time travel, tags, and branches without requiring additional infrastructure.

    Source: README

  • Broad ecosystem integrations — The project integrates with Arrow, Pandas, Polars, DuckDB, Spark, Ray, Trino, Flink, and open catalogs.

    Source: README, python, java

Quick start

Copied verbatim from the project's README.

pip install pylance
import lance

import pandas as pd
import pyarrow as pa
import pyarrow.dataset

df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')

parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")
dataset = lance.dataset("/tmp/test.lance")
assert isinstance(dataset, pa.dataset.Dataset)

Who it may suit

Inference: may suit teams building search engines, feature stores, large-scale training pipelines, and systems that store or query multimodal AI data.

Topics to discuss

  • Which multimodal AI workloads most need unified storage for vectors, text, media, and embeddings?

  • How should hybrid search across vector, full-text, and SQL predicates be evaluated?

  • Which compatibility and versioning practices should production users adopt when SDK releases move quickly?


Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/lance-format/lance#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/lance-format/lance

Comments

…