Release 0.299

Breaking Changes

  • Fix partition filter cache metrics association. BREAKING: Metric names changed from partitionnamescache* to partitionfiltercache*. Users monitoring these JMX metrics must update their dashboards, alerts, and scripts to use the new metric names. The old metrics tracked the partition filter cache (filtered partition queries), not the partition names cache, as the name suggested. #27960

  • Add explicit strategy parameter to rewrite_data_files procedure with binpack and sort strategies. Breaking change: Default behavior changed from sort to binpack for faster rewrites. Existing queries that rely on data ordering must add strategy => ‘sort’ to maintain previous behavior. Queries using sorted_by must now explicitly specify strategy => ‘sort’. #28092

  • Replace the lance.root-url configuration property with lance.root. This is a breaking change: catalogs that set lance.root-url must be updated to lance.root before upgrading. See Configuration Properties. #27481

  • Upgrade minimum Elasticsearch version from 6 to 9 (breaking) in response to CVE-2024-52980. #25320

Highlights

  • Add multi-arch (amd64 + arm64) support for presto-native and presto-native-dependency release Docker images, with the arm64 build using a portable -march=armv8-a+crc+crypto baseline for broad ARM CPU compatibility. #28046

  • Add an Arrow federation connector to run federated queries. #26404

  • Fix timestamp operations to match the SQL specification. The value of a TIMESTAMP type is not affected by the session time zone. #24571

  • Add comprehensive cache metrics for all metastore caches. See Metastore Cache Metrics. #27960

  • Add support for AWS Glue Table and Column Statistics. #27112

  • Upgrade to Hive 4.0.1. #24571

Details

General Changes

  • Fix BATCH-mode RPC dispatch dropping and under-batching rows by gathering RPC input to a single driver in the local plan. #28230

  • Fix incorrect epoch-second and nanosecond decomposition for pre-1970 (negative) timestamps in TimestampType; getEpochSecond and getNanos now use floor division instead of truncation toward zero. #27935

  • Fix the optimize_row_in_predicate optimization so it applies to constant-folded ROW(...) IN (...) predicates, enabling per-column predicate derivation and partition pruning that previously did not occur. #27942

  • Fix timestamp operations to match the SQL specification. The value of a TIMESTAMP type is not affected by the session time zone. #24571

  • Fix incorrect results when reading from a partially materialized view, where partitions recomputed from the base tables could also be read from the view and counted twice. #27944

  • Fix a query failure (Field not found) when multiple remote function calls wrapped in TRY() appear in the same SELECT. #28232

  • Improve outer joins on skewed NULL join keys by spreading the null keys across partitions in their native type. This is controlled by the existing randomize_outer_join_null_key session property. #28153

  • Improve the optimize_cascading_filters_and_projections optimization to avoid duplicating multiply-referenced non-trivial expressions when coalescing cascading projections. #28216

  • Add JOIN support to the materialized view query optimizer. Queries that join a base table covered by a materialized view with another table can now be rewritten to scan the materialized view in place of the base table, subject to safety guards (matching GROUP BY, no aggregates over non-swapped tables, supported join types). #27733

  • Add scanRawInputDataSizeInBytes to task statistics, reporting the raw input data size read by table scan operators. #28222

  • Add scanRawInputDataSize to basic query statistics. #28222

  • Add native_exchange_materialization_enabled session property (Presto on Spark native codepath only) to control whether Velox native workers use MaterializedOutput/MaterializedExchange operators. When set to true, enables materialized exchange; when false (default), falls back to PartitionAndSerialize + ShuffleWrite. #27881

  • Add a driver-side metadata sidecar that registers native-only Velox functions into the Java planner at driver bootstrap. See built-in-sidecar-functions-enabled and related properties. #27698

  • Add column-level lineage for derived expression and aggregation columns in materialized view definitions, and emit a warning when a materialized view column has no base-table lineage. #28176

  • Add configuration and session properties for legacy ST_Equals behavior. #27015

  • Add configuration property server.startup-complete-required-for-active to report a node as not ready (/v1/info starting and /v1/info/state) until server startup has fully completed. Defaults to false. #28193

  • Add createTimestampType(int precision) factory supporting TIMESTAMP precisions p=0–12, with instance interning and semantic helpers toEpochMillis, toEpochMicros, and fromEpochComponents. Part of parameterized TIMESTAMP(p) support in #27934. #27935

  • Add indirect column lineage (JOIN, WHERE/HAVING, GROUP BY, ORDER BY, window, and conditional expressions from #27695) to the OpenLineage event listener’s columnLineage dataset facet, exposing it through InputField.transformations. Direct-lineage emission is unchanged. See OpenLineage Event Listener. #27994

  • Add indirect column lineage tracking (JOIN, FILTER, GROUP BY, ORDER BY, CONDITIONAL) to query analysis, building on the direct column lineage added in #25913. Indirect relationships are exposed to event listeners with a new relationship-metadata field on OutputColumnMetadata; existing direct-lineage consumers are unaffected. See OpenLineage Event Listener. #27695

  • Add multi-arch (amd64 + arm64) support for presto-native and presto-native-dependency release Docker images, with the arm64 build using a portable -march=armv8-a+crc+crypto baseline for broad ARM CPU compatibility. #28046

  • Add session property grouped_execution_when_capable (default disabled) that, together with grouped_execution, runs grouped execution for any bucketed grouped-execution-capable fragment even when no downstream operator makes grouping individually beneficial (for example a bucketed scan feeding a shuffle, or a bucketed-to-bucketed table write). This avoids re-partitioning already-bucketed data and bounds per-lifespan memory to a single bucket. #28097

  • Add session property pull_constant_projection_above_exchange (disabled by default) that pulls constant projection assignments above remote exchanges, avoiding serialization and shuffling of constant values across the network. #27499

  • Add session property pull_row_local_chain_above_exchange_strategy (default DISABLED) that pulls a chain of row-local operators (UNNEST and deterministic projections) above a repartitioning remote exchange so the exchange shuffles the smaller pre-expansion input, reducing network shuffle. #28079

  • Add support for ANSI SQL syntax in trim function. #28190

  • Add support for additional predicates on WHEN clauses in MERGE (WHEN MATCHED AND <condition>, WHEN NOT MATCHED AND <condition>). #27855

  • Add support for an any-typed variadic tail in JSON-file-based function definitions. #28207

  • Add support for hyphenated struct field names in nested ROW type columns. #27470

  • Add support for publishing release Docker images to GitHub Container Registry (GHCR) via the REGISTRY repository variable. #28010

  • Add the optimize_cascading_filters_and_projections session property (config property optimizer.optimize-cascading-filters-and-projections, default disabled). When enabled, the optimizer coalesces cascading projections by fully inlining deterministic child expressions and merges adjacent filter/projection nodes, co-locating shared subexpressions within a single operator so the native (Velox) backend’s common-subexpression elimination can deduplicate them. #28016

  • Add the optimize_join_fan_out session property (config property optimizer.optimize-join-fan-out, default disabled). When enabled, the optimizer collapses a fan-out equi-join whose preserved side is an aggregation grouped by a strict superset of the join keys: it packs the non-key columns with array_agg(row(...)) so the join becomes unique on the join key, then re-expands them with a local UNNEST above the join, reducing the rows shuffled through the distributed join. #27970

  • Add the materialized view query optimizer to the inner query of CREATE TABLE AS and INSERT statements, not just bare SELECT. #27917

  • Add validation to reject non-deterministic and session-time functions in CREATE MATERIALIZED VIEW definitions. #28220

  • Add optimizer rule parallelize_chained_aggregation (default: false) that inserts a local round-robin exchange to parallelize the outer PARTIAL in chained aggregations. #27884

  • Add the AUTOMATIC value for the rpc_streaming_mode session property, the rpc_batch_min_rows session property, and a pluggable RpcExecutionPolicy so deployments can resolve per-row versus batch RPC dispatch from the estimated input stats. #27984

  • Update ST_Equals function for empty geometries to return true regardless of geometry types. #27015

  • Update default value of deprecated.legacy-timestamp to false. #24571

  • Update the driver-side metadata sidecar registration of worker functions into the Airlift bootstrap. #27699

Prestissimo (Native Execution) Changes

  • Fix runtime type-mismatch crashes at exchange operators in Prestissimo when aggregation variable names sort differently from their Java allocation order. #27903

  • Fix LIKE, regexp, and json_extract queries applied to the result of a remote function (for example meta.ai.* outputs), which previously failed native query plan conversion. #28118

  • Add an Arrow federation connector to run federated queries. #26404

  • Add registration for Presto-specific cuDF functions when cuDF is enabled in Presto native. #28093

  • Add support for setting gflags through config.properties using the gflag. prefix. Property names use hyphens in place of underscores, such as gflag.velox-memory-num-shared-leaf-pools=64. Command-line flags take precedence over config values. See Presto C++ Configuration Properties for the full list of supported gflag properties. #28127

  • Add session properties to tune the adaptive RPC rate limiter and congestion window (native_rpc_ratelimiter_adaptive_enabled, native_rpc_ratelimiter_min_limit, native_rpc_ratelimiter_decrease_factor, native_rpc_ratelimiter_max_limit, native_rpc_congestion_max_window). #28115

  • Add a two-phase memory reclaim for MaterializedOutputBuffer, integrated with the Velox memory arbitrator. #27875

  • Update native_exchange_materialization_enabled session property default to true to enable MaterializedOutput and MaterializedExchange operators in Velox by default. #27980

Security Changes

Cassandra Connector Changes

  • Upgrade to Cassandra Java Driver 4.x. #27029

Delta Lake Connector Changes

  • Add support for reading Variant data as JSON. #27552

Hive Connector Changes

  • Fix Parquet RLE and PLAIN dictionary decoding for decimals backed by byte arrays. #28086

  • Fix partition filter cache metrics association. BREAKING: Metric names changed from partitionnamescache* to partitionfiltercache*. Users monitoring these JMX metrics must update their dashboards, alerts, and scripts to use the new metric names. The old metrics tracked the partition filter cache (filtered partition queries), not the partition names cache, as the name suggested. #27960

  • Fix to allow creating partitioned tables using AVRO format and avro_schema_url property. #27490

  • Add Azure filesystem impl registration for ABFSS and WASB/S schemes. #28054

  • Add comprehensive cache metrics for all metastore caches. See Metastore Cache Metrics. #27960

  • Add support for AWS Glue Table and Column Statistics. #27112

  • Add support for reading and writing per-column type attributes in DWRF files. #27940

  • Upgrade to Hive 4.0.1. #24571

Iceberg Connector Changes

  • Fix DROP TABLE for Hive-backed Iceberg tables to properly delete all data and metadata files on S3 using Iceberg’s CatalogUtil instead of relying on Hive metastore directory deletion. #27938

  • Fix timestamp-to-micros conversion and legacy-timezone adjustment in IcebergPageSink for pre-epoch timestamps, using the corrected TimestampType epoch helpers. #27935

  • Improve Iceberg table statistics computation by using snapshot-level total record counts instead of re-scanning manifests. #28248

  • Add Iceberg version validation to prevent silent data loss. #27655

  • Add configuration property iceberg.commit-number-retries to specify the default number of commit retries for newly created tables. #28055

  • Add explicit strategy parameter to rewrite_data_files procedure with binpack and sort strategies. Breaking change: Default behavior changed from sort to binpack for faster rewrites. Existing queries that rely on data ordering must add strategy => ‘sort’ to maintain previous behavior. Queries using sorted_by must now explicitly specify strategy => ‘sort’. #28092

  • Add optional delete_data_on_drop parameter to the register_table procedure to control whether underlying Iceberg data is deleted when the registered table is dropped. #27938

  • Add proxy support for Iceberg REST catalogs. See REST catalog. #28217

  • Add support for Basic Auth against the REST catalog server. See REST catalog. #28103

  • Add support for iceberg write-default. #27912

  • Add support to enable TLS for REST catalog communication. See REST catalog. #28103

  • Add read support for row lineage columns as per Iceberg V3 spec. #27743

  • Add support for Iceberg V3 field-id protocol in Presto Native. #28116

  • Add support for Iceberg V3 deletion vectors and UPDATE / MERGE statements in Presto Native. #28058

Lance Connector Changes

  • Fix catalog-wide table listing returning no tables for multi-level Lance catalogs (lance.single-level-ns=false). This affected information_schema.tables scans. #28268

  • Add LIMIT pushdown for Lance table scans to reduce rows read by Lance for simple limit queries. #28049

  • Add pluggable namespace support to the Lance connector using the lance-namespace API, enabling directory, REST, and custom namespace implementations. #27481

  • Add the lance.parent configuration property to select a parent namespace prefix for namespaces with three or more levels. See Configuration Properties. #27481

  • Replace the lance.root-url configuration property with lance.root. This is a breaking change: catalogs that set lance.root-url must be updated to lance.root before upgrading. See Configuration Properties. #27481

Oracle Connector Changes

  • Add support for REAL datatype insert. #28059

SPI Changes

  • Add OutputColumnMetadata.getColumnLineage() returning a unified Set<ColumnLineageEntry> that covers both DIRECT and INDIRECT lineage from #27695, with direct entries carrying IDENTITY, TRANSFORMATION, or AGGREGATION subtypes derived from the SELECT-list expression. getSourceColumns() and getIndirectSourceColumns() are retained as derived views and marked @Deprecated; existing event listeners and JSON consumers are unaffected. #27995

  • Add getScanRawInputBytes to QueryStatistics. Note: this adds a required constructor argument; plugins that construct QueryStatistics directly must be updated. #28222

  • Add a ConnectorMetadata.beginRefreshMaterializedView` overload that carries the materialized view refresh predicate (refresh scope) to connectors. #27949

  • Add getMaterializedViewRewrittenQuery to QueryCompletedEvent, exposing the query produced when a materialized view rewrite is applied. #27919

Credits

Aditi Pandit, Ajay Kharat, Allen Shen, Amit Dutta, Andrii Rosa, Apurva Kumar, Auden Woolfson, Ayasaz, Bryan Cutler, Chandrakant Vankayalapati, ChenXing Yang, Christian Zentgraf, Deepak Majeti, Deepak Mehra, Denis Krivenko, Denodo Research Labs, Dilli Babu Godari, Dong Wang, Henry Dikeman, Hongtao Yang, Jack Luo, Jalpreet Singh Nanda, Jianjian Xie, Joe Abraham, Kevin Tang, Madhavan, Maria Basmanova, Matt Gara, Miguel Blanco Godón, Natasha Sehgal, Neerad Somanchi, Nidhin Varghese, Nishitha K Bhaskaran, Nivin C S, Patrick Sullivan, Pramod Satya, Pratik Joseph Dabre, Reetika Agrawal, Sayari Mukherjee, Shahim Sharafudeen, Shakyan Kushwaha, Shreya, Shrinidhi Joshi, Sreeni Viswanadha, Steve Burnett, Timothy Meehan, Tirumala Saiteja Goruganthu, Vyacheslav Andreykiv, Yihong Wang, Ying, Zac, Zac Blanco, bcam-meta, bibith4, deepthibose01, dependabot[bot], feilong-liu, jkhaliqi, mohsaka, sumi-mathew, zhichenxu-meta