What problem it solves

Data analysis workflows often require expressive query composition, flexible execution models, and interoperability with columnar data. Polars addresses these needs by allowing the DataFrame API and SQL to share the same optimizer and engine, with support for large workloads and custom extensions.

Key features

• Unified DataFrame and SQL Queries — Queries can be written with expressions, SQL, or a mixture of both. DataFrames and LazyFrames in scope can be queried by name, allowing SQL results to be filtered with expressions.

  Source: README

• Optimized Parallel Execution — Lazy queries are optimized before execution and run in parallel across available CPU cores. The engine is implemented in Rust with vectorized execution.

  Source: README

• Out-of-Core Processing — Queries that exceed the available memory budget can spill intermediate data to disk and continue processing outside the core memory limit.

  Source: README, crates/polars-mem-engine

• Extensibility and Accelerators — Developers can extend the engine with custom I/O and expression plugins. NVIDIA GPU acceleration is also available as an optional execution path.

  Source: README

• Columnar Interoperability — The engine uses the Apache Arrow Columnar Format, enabling zero-copy data sharing with compatible columnar data.

  Source: README, crates/polars-arrow

Quick start

Copied verbatim from the project's README.

pip install polars

import polars as pl

df = (
    pl.scan_parquet("orders.parquet")
    .filter(pl.col("status") == "shipped")
    .group_by("customer_id")
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.len().alias("n_orders"),
    )
    .sort("total", descending=True)
    .collect()
)

Who it may suit

This project may suit Python and Rust developers who need expressive analytical queries across DataFrame, LazyFrame, and SQL interfaces. It may also suit teams working with Apache Arrow-compatible data or extending query behavior through plugins.

Topics to discuss

• Which query style should be prioritized in future examples and documentation?

• What workflows would benefit most from plugins and custom expression support?

• How should users balance local execution, GPU acceleration, and distributed cluster deployments?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/pola-rs/polars#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.