TL;DR
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Polars says version 2.0 makes its streaming engine the default for LazyFrame collection and enables initial spill-to-disk support, aiming to reduce memory pressure on supported queries. The release also expands SQL support and adds a Map data type; its performance comparisons are vendor-run benchmarks, and some operations may no longer preserve row order by default.
Polars has released version 2.0, making its streaming engine the default when users collect a LazyFrame and enabling initial spill-to-disk support for supported operations. The changes are intended to help queries use less memory and run across larger workloads, but users may need to account for altered row-order behavior in some operations.
Under the new default, calling collect on a LazyFrame uses the streaming engine. Polars says the engine can improve memory use and performance on many queries, but it does not guarantee observable row order by default for certain operations, including joins, group-bys and unpivots. Users who need to preserve order can set maintain_order=True where supported.
Version 2.0 also enables out-of-core processing by default. Polars says spilling begins at about 80% of available RAM, with a default disk budget of 64GB. The initial set of supported operations includes sorts, window functions and many expressions; joins and group-bys are not yet supported for spill-to-disk, though the project says it plans to add them.
Other release highlights include expanded SQL support, optimizer and engine changes such as join reordering and improved common-subplan elimination, and a new Map dtype for dictionary-like key-value data. The release report also describes stricter handling of data types and explicitness, intended to provide faster feedback during development.
How the New Defaults Affect Queries
The default switch changes how existing lazy-query code executes, so users upgrading may see different memory use and performance without changing their query calls. It also means row order cannot be assumed for the named operations unless users request that behavior. Teams whose downstream steps depend on a particular order should test upgrade results and set the order option where needed.
Spilling can let some workloads continue when their working data exceeds available memory, using disk as additional capacity. The current limits matter: joins and group-bys are not covered yet, and the 64GB disk budget and approximate RAM threshold are stated defaults, not universal capacity guarantees. The SQL and optimizer work may also make Polars a more viable option for SQL-oriented analytics, but the release’s performance evidence comes from benchmarks run and reported by Polars itself.
high performance SSD for data processing
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why Polars Made This a Major Release
Polars presents version 2.0 as a release centered on execution behavior and workload coverage, rather than solely on a long list of new features. The project says the streaming engine’s row-order behavior was a reason for the major version bump: the default can differ from expectations users may have formed around earlier execution behavior.
For its SQL performance comparison, Polars tested queries derived from TPC-H and TPC-DS against DuckDB 1.5.6, a DuckDB 2.0 alpha build and DataFusion 54.0.0. The tests used two AWS machine configurations, ran each query five times in a hot setting and selected the fastest run. Polars reported that its default configuration was fastest in all but one benchmark, while also noting that scaling to 192 threads created overhead for small queries. These are the publisher’s results under its disclosed setup, not an independent evaluation or a guarantee for other data and hardware.
“Calling collect on a LazyFrame will now default to the streaming engine.”
— Polars release report
large capacity external hard drive 64GB
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of Spill and Benchmark Results
The release material does not give an exact publication date, nor does the supplied text provide detailed results for every benchmark query. Polars says it shared a repository so others can reproduce the comparison, but the reported results have not been independently established in the provided source. DataFusion also timed out on TPC-DS query 72, once on query 67, and ran out of memory on TPC-H query 18 on the smaller machine; those queries were excluded from the results for all engines.
It is not yet clear when spill-to-disk support will extend to joins and group-bys, or how the default memory threshold and disk budget will perform across different systems. The release report also identifies a small-query overhead when Polars scales to 192 threads and says a fix is hoped for in a later release, without specifying a version or date.
RAM monitoring and management tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Upgrade Checks and Planned Coverage
Users adopting version 2.0 should check whether joins, group-bys or unpivots in their workflows depend on row order, and use maintain_order=True where necessary. They can also test memory-heavy queries against the operations currently supported for spilling and review the configured disk budget against their environment.
Polars says it plans to add out-of-core support for joins and group-bys and hopes to address the 192-thread overhead in a later release. The report gives no delivery dates for either item. Replication of the published benchmarks may provide additional evidence about how the SQL and performance changes behave on other hardware and workloads.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
LazyFrame collection uses the streaming engine by default. The release also enables initial spill-to-disk support for selected operations.
Will Polars 2.0 preserve row order?
Not by default for some operations, including joins, group-bys and unpivots, according to Polars. Users who require observable order can set maintain_order=True where supported.
Which operations can spill to disk?
The release report lists sorts, window functions and many expressions. It says out-of-core support for joins and group-bys is planned but not yet available.
Are the performance claims independently verified?
The cited TPC-H and TPC-DS results were produced and reported by Polars. The project shared a benchmark repository for replication, but the supplied material does not include an independent verification.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
