lance: An open lakehouse format for multimodal AI.
What problem it solves
AI and machine learning workflows require vector search, fast random access, multimodal data storage, data evolution, and hybrid search combining vectors, full-text, and SQL predicates. Lance brings these capabilities directly into the lakehouse format.
Key features
Expressive hybrid search — Combine vector similarity search, full-text search with BM25, and SQL analytics on the same dataset using accelerated secondary indices.
Source: README
Native multimodal data support — Store images, videos, audio, text, and embeddings in one format with blob encoding and lazy loading.
Source: README
Data evolution — Add columns with backfilled values without full table rewrites, supporting feature engineering workflows.
Source: README
Zero-copy versioning — Automatic versioning provides ACID transactions, time travel, tags, and branches without requiring additional infrastructure.
Source: README
Broad ecosystem integrations — The project integrates with Arrow, Pandas, Polars, DuckDB, Spark, Ray, Trino, Flink, and open catalogs.
Source: README, python, java
Quick start
Copied verbatim from the project's README.
pip install pylanceimport lance
import pandas as pd
import pyarrow as pa
import pyarrow.dataset
df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')
parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")dataset = lance.dataset("/tmp/test.lance")
assert isinstance(dataset, pa.dataset.Dataset)Who it may suit
Inference: may suit teams building search engines, feature stores, large-scale training pipelines, and systems that store or query multimodal AI data.
Topics to discuss
Which multimodal AI workloads most need unified storage for vectors, text, media, and embeddings?
How should hybrid search across vector, full-text, and SQL predicates be evaluated?
Which compatibility and versioning practices should production users adopt when SDK releases move quickly?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/lance-format/lance#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…