We’ve been hard at work on our catalog-managed vision for Delta Lake, a new architecture first introduced in Delta Lake 4.1.0 that uses a catalog-centric model for managing tables and ratifying commits. Along this journey, we’ve focused on the core capabilities required to expand the Delta ecosystem by simplifying integrations for the various engines supporting the Delta Lake protocol - as you’ve seen with Delta Kernel and more recently with the UC Delta API, our new API specification for working with Delta Lake tables.

The Unity Catalog specification is available for any catalog provider to adopt and opens the doors for different engines, like Apache Flink, Apache Spark, and Apache DataFusion, to talk to a unified API from their catalog of choice, while integrating with catalog-managed Delta Lake tables across a truly open ecosystem.

This mission to simplify ecosystem support continues with the release of Delta Lake 4.4.0.

What’s New in Delta Lake 4.4.0

There are many notable changes for this release across Delta Spark, Delta Kernel, Delta UniForm, Delta Sharing, as well as Delta Flink, all aimed at simplifying the way you work with your Delta tables and expanding the ecosystem support for catalog-managed tables as well as the new UC Delta APIs.

Delta-Spark now defaults to Spark 4.2

This release brings Apache Spark 4.2 support to Delta, and provides continued support for Apache Spark 4.1.0, as well as Apache Spark 4.0.1 across Delta Spark, Delta Connect, and Delta Sharing.

Apache Spark 4.2 is itself a substantial release that brings native GEOMETRY/GEOGRAPHY types, support for the SQL CHANGES clause for Auto CDC support in declarative SCD Type 1 pipelines, Apache Arrow-optimized Python UDFs (enabled by default), as well as Data Source V2 transaction-management and schema-evolution improvements.

export CATALOG_NAME=unity
export UC_SERVER_URL=http://localhost:8080
export UC_TOKEN=
$SPARK_HOME/bin/pyspark \
  --packages io.delta:delta-spark_4.2_2.13:4.4.0,io.unitycatalog:unitycatalog-spark_4.2_2.13:0.6.0 \
  --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
  --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
  --conf "spark.sql.catalog.${CATALOG_NAME}=io.unitycatalog.spark.UCSingleCatalog" \
  --conf "spark.sql.catalog.${CATALOG_NAME}.uri=${UC_SERVER_URL}" \
  --conf "spark.sql.catalog.${CATALOG_NAME}.token=${UC_TOKEN}" \
  --conf "spark.sql.defaultCatalog=${CATALOG_NAME}"

Note: Delta Lake 4.2 defaults to Spark 4.2, but supports Spark 4.1.0, and 4.0.1. See the release notes for the version matrix.

Identity & Generated Columns in SQL DDL

Delta Lake 4.4.0 teaches Delta Spark to understand Spark SQL’s identity-column syntax directly in CREATE TABLE. You can now declare a surrogate key with GENERATED ALWAYS AS IDENTITY (or GENERATED BY DEFAULT AS IDENTITY) and let Delta assign monotonically increasing values on write. No more bolting on monotonically_increasing_id() or hand-rolling your own sequence. This simplifies the creation and maintainability of any sequential data.

Let’s put it to work in the playground. For the sake of the example, let’s say we’re running an animal rescue, what I’m calling the sanctuary, and we want to now add a new inventory table for donated supplies. We give it an auto-assigned id, and — since we’ll want to group supplies by the rescue’s species (dog, cat, other) — we’ll partition by animal_category. The delta.feature.catalogManaged property opts the table into catalog-managed commits.

CREATE TABLE IF NOT EXISTS unity.sanctuary.inventory (
  id              BIGINT  GENERATED ALWAYS AS IDENTITY,
  name            STRING  NOT NULL,
  product_type    STRING  NOT NULL,
  animal_category STRING  NOT NULL,
  quantity        INT     NOT NULL
)
USING DELTA
TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported')
PARTITIONED BY (animal_category)

Identity columns are only half of the story. Delta 4.4.0 also preserves generated-column expressions when a table is created through Unity Catalog, computing (or validating) those values on every write. So a computed column round-trips cleanly through a catalog-managed table:

# ~100 shelter-supply products; the identity `id` is assigned by Delta on write
products = generate_rescue_products(total=100)
products_df = rescue_products_to_dataframe(products, spark)

(
    products_df.write
        .format("delta")
        .mode("append")
        .saveAsTable("unity.sanctuary.inventory")
)

Reading back, every row now carries a stable, monotonically increasing key (your names and quantities will differ — the generator is random):

spark.sql("SELECT * FROM unity.sanctuary.inventory ORDER BY id ASC").show(5)

+--+---------------+------------+---------------+--------+
|id|           name|product_type|animal_category|quantity|
+--+---------------+------------+---------------+--------+
| 1| Fleece Blanket|     blanket|            dog|      27|
| 2|     Dry Kibble|        food|            cat|      12|
| 3|   Squeaky Ball|         toy|            dog|      44|
| 4|Clumping Litter|      litter|            cat|       8|
| 5|     Bath Towel|       towel|          other|      19|
+--+---------------+------------+---------------+--------+

What’s even cooler here, say you want the caller to be able to supply their own id and only auto-fill when it’s omitted? You can now use the GENERATED BY DEFAULT AS IDENTITY for just that. And if you need a specific starting point or stride, both forms accept custom seeds — GENERATED ALWAYS AS IDENTITY (START WITH 100 INCREMENT BY 10).

Identity columns are only half of the story. Delta 4.4.0 also preserves generated-column expressions when a table is created through Unity Catalog, computing (or validating) those values on every write. So a computed column round-trips cleanly through a catalog-managed table:

CREATE TABLE unity.sanctuary.donations (
  amount          DOUBLE NOT NULL,
  tax_rate        DOUBLE NOT NULL,
  amount_with_tax DOUBLE GENERATED ALWAYS AS (amount * (1 + tax_rate))
)
USING DELTA
TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported')

Inspecting Table Partitions

Because inventory is partitioned by animal_category, we can reach for Spark SQL’s standard SHOW PARTITIONS command, which is now supported on Delta tables, to see exactly which partitions exist:

spark.sql("SHOW PARTITIONS unity.sanctuary.inventory").show()

+---------------+
|animal_category|
+---------------+
|            cat|
|            dog|
|          other|
+---------------+

Point it at a table that isn’t partitioned, like the pets table from our catalog-managed walkthrough, and Delta tells you precisely why there’s nothing to show:

spark.sql("SHOW PARTITIONS unity.sanctuary.pets")

AnalysisException
- [DELTA_SHOW_PARTITION_IN_NON_PARTITIONED_TABLE] SHOW PARTITIONS is not allowed on a table that is not partitioned: unity.sanctuary.pets

VOID Column Support

On Apache Spark 4.1 and later, Delta 4.4.0 preserves VOID (NullType) columns instead of failing or silently dropping them, and this includes operations like time-travel and path-based reads. The Delta protocol itself was updated in this release to define how readers and writers treat VOID, so an all-null placeholder column (a field you’ve scaffolded but haven’t populated yet) is retained rather than quietly discarded:

spark.sql("SELECT id, name, NULL AS not_yet_tracked FROM unity.sanctuary.inventory").printSchema()

root
 |-- id: long (nullable = true)
 |-- name: string (nullable = false)
 |-- not_yet_tracked: void (nullable = true)

Metric Views over Catalog-Managed Delta Tables

Moving to Apache Spark 4.2 unlocks more than Delta features. Pair Delta 4.4.0 with the Unity Catalog 0.6.0 connector and you can layer one of UC’s newest governed objects — the metric view — directly on top of your catalog-managed Delta tables.

A metric view defines reusable dimensions (grouping columns) and measures (named aggregations) over a source table or query, expressed in YAML. Consumers read the measures through the measure(...) function and GROUP BY the dimensions, so the aggregation logic lives in one governed place in Unity Catalog rather than being copy-pasted, and probably subtly diverging across dashboards, notebooks, and pipelines.

Say we have a catalog-managed Delta table ecomm.consumer.orders that contains e-commerce orders. We can define a metric view that scopes to the trailing seven days and exposes a few sales measures by region:

CREATE VIEW ecomm.consumer.orders_last_7_day_sales
WITH METRICS
LANGUAGE YAML
AS $$
version: "0.1"
source: ecomm.consumer.orders
filter: created_at >= current_timestamp() - INTERVAL 7 DAYS AND status NOT IN ('CANCELLED', 'RETURNED')
dimensions:
  - name: region
    expr: region
measures:
  - name: last_7_day_sales
    expr: sum(amount)
  - name: order_count
    expr: count(1)
  - name: avg_order_amount
    expr: avg(amount)
$$

Because the filter uses current_timestamp(), the seven-day window slides forward on every query — consumers always get a live trailing window without ever touching a WHERE clause. To read it, wrap each measure in measure(...) and GROUP BY the dimension:

SELECT
  region,
  measure(last_7_day_sales) AS last_7_day_sales,
  measure(order_count)      AS orders,
  measure(avg_order_amount) AS avg_order_amount
FROM ecomm.consumer.orders_last_7_day_sales
GROUP BY region
ORDER BY last_7_day_sales DESC

+------+----------------+------+----------------+
|region|last_7_day_sales|orders|avg_order_amount|
+------+----------------+------+----------------+
|    US|         4210.55|    17|          247.68|
|    GB|         3180.20|    12|          265.02|
|    DE|         2955.10|    11|          268.65|
+------+----------------+------+----------------+

Selecting a measure column directly (SELECT last_7_day_sales ...) isn’t supported — it must go through measure(...), which is what lets Spark rewrite the query into the aggregations declared in the YAML.

A metric view is a first-class governed UC object: it shows up on both the view surface (SHOW VIEWS) and the table surface (SHOW TABLES, alongside its source table), and DESCRIBE EXTENDED reports its dimension/measure columns and the METRIC_VIEW type. Because it’s just another UC object, you can inspect it from the bundled CLI too:

docker exec -it unitycatalog \
  bash bin/uc table get --full_name ecomm.consumer.orders_last_7_day_sales

Metric views need Apache Spark 4.2+ (the CREATE VIEW ... WITH METRICS DDL landed in 4.2) and the UC 0.6.0 Spark 4.2 connector (io.unitycatalog:unitycatalog-spark_4.2_2.13:0.6.0).

Delta UniForm & Delta Sharing

Delta UniForm keeps Apache Iceberg metadata in sync with Delta commits so Iceberg readers can query Delta tables without duplicating data. In 4.4.0, UniForm metadata can be initialized atomically with table creation.

Turning it on is just a pair of table properties when you go to execute CREATE TABLE. Thanks to atomic initialization the Iceberg metadata is written as part of creating the table rather than lazily on the first commit:

CREATE TABLE unity.sanctuary.adoptions (
  adoption_id STRING    NOT NULL,
  pet_name    STRING    NOT NULL,
  adopter     STRING    NOT NULL,
  adopted_at  TIMESTAMP NOT NULL
)
USING DELTA
TBLPROPERTIES (
  'delta.enableIcebergCompatV2'          = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
)

delta.enableIcebergCompatV2 puts the table in an Iceberg-compatible layout (it also switches column mapping to name mode), while delta.universalFormat.enabledFormats = 'iceberg' tells Delta to keep an Iceberg metadata view in sync. From here on, every write to the Delta table also refreshes the Iceberg snapshot, so an Iceberg-native engine can read unity.sanctuary.adoptions with no copy or extra ETL step.

One caveat: in 4.4, delta-iceberg_2.13 targets Spark 4.1 and is not compatible with Spark 4.2 — so to run this example in the playground, resolve the Spark 4.1 build (io.delta:delta-iceberg_2.13 against delta-spark_4.1_2.13:4.4.0) rather than the 4.2 default. But with the 1.12 release of Apache Iceberg, 4.2 support should be on the table, so you’ll need to wait a little longer for our 4.5 release (or when Spark 4.2. support arrives for Iceberg).

Delta Flink Continues to Evolve

The Kernel-based delta-flink connector takes a meaningful step forward. This release supports Apache Flink 2.0.2, 2.1.3, 2.2.1, and 2.3.0, and now publishes version-specific artifacts (delta-flink_2.0 … delta-flink_2.3) so you can pick the one matching your runtime.

The most visible new capability is primary-key upserts from Flink SQL. Declare a primary key and open the table in upsert mode, and Delta interprets Flink’s changelog stream for you:

CREATE TABLE orders (
  order_id STRING,
  amount   DOUBLE,
  PRIMARY KEY (order_id) NOT ENFORCED
) WITH ('write.mode' = 'upsert')

INSERT is treated as a new key, UPDATE_AFTER replaces an existing key, DELETE removes it, and UPDATE_BEFORE is ignored (a primary key is required in upsert mode). Under the hood the default upsert strategy uses merge-on-read and deletion vectors retire old row versions while new rows are appended, which avoids full rewrites.

Flink now also supports the UC Delta API for catalog-managed table loads, existence checks, commits, and storage-credential vending.

The following is an example of writing into a catalog-managed table using Delta Flink.

SET 'table.dml-sync' = 'true';
SET 'pipeline.name'  = 'dfp-03-uc-catalog';

CREATE CATALOG uc WITH (
  'type'     = 'unitycatalog',
  'endpoint' = 'http://host.docker.internal:8080',
  'token'    = 'not-used-auth-disabled'
);

CREATE TEMPORARY TABLE src (
  id    BIGINT,
  name  STRING
) WITH (
  'connector'       = 'datagen',
  'number-of-rows'  = '1000',
  'rows-per-second' = '1000',
  'fields.id.kind'  = 'sequence',
  'fields.id.start' = '1',
  'fields.id.end'   = '1000',
  'fields.name.length' = '8'
);

INSERT INTO uc.`unity`.`flink_playground`.`clickstream`
SELECT id, name FROM src;

Having been working with streaming data for the past decade, the next update is really cool. The 4.4.0 release adds support for reading the keyed changelog from Kafka. In the following example, we use RedPanda to upsert our Delta table by primary key. This is made possible due to the introduction of the primary-key upserts from Flink SQL. When combined, you can now apply true streaming upserts.

SET 'pipeline.name' = 'dfp-06-upsert-kafka-cdc';

CREATE TEMPORARY TABLE user_events_kafka (
  user_id  BIGINT,
  status   STRING,
  amount   DOUBLE,
  PRIMARY KEY (user_id) NOT ENFORCED
) WITH (
  'connector'                     = 'upsert-kafka',
  'topic'                         = 'ecomm.v1.clickstream',
  'properties.bootstrap.servers'  = 'redpanda:29092',
  'properties.group.id'           = 'dfp-06',
  'key.format'                    = 'json',
  'value.format'                  = 'json'
);

CREATE TEMPORARY TABLE user_state (
  user_id  BIGINT,
  status   STRING,
  amount   DOUBLE,
  PRIMARY KEY (user_id) NOT ENFORCED
) WITH (
  'connector'   = 'delta',
  'table_path'  = 'file:///opt/flink/data/06_upsert_kafka_cdc',
  'write.mode'  = 'upsert',
  'uid'         = 'dfp-06'
);

INSERT INTO user_state SELECT user_id, status, amount FROM user_events_kafka;

Try the Delta Flink connector yourself.

Wrapping Things Up

Delta Lake 4.4.0 pushes the catalog-managed vision forward in two directions at once. On the Delta Spark side, it brings a batch of everyday quality-of-life features to Apache Spark 4.2 — identity and generated columns in DDL,fVOID columns, and Unity Catalog metric views on top of your Delta tables. On the ecosystem side, it wires Delta Kernel and Delta Flink into the UC Delta API, so more of the open lakehouse can speak to catalog-managed tables through a single, open specification.

Try it yourself

Every Spark example above runs in the unitycatalog-playground, a Dockerized marimo Spark environment wired to a local Unity Catalog. To follow along:

git clone https://github.com/open-lakehouse/unitycatalog-playground.git
cd unitycatalog-playground
just uc=local start   # build + start the local UC + marimo stack, then print the notebook URL

Open the printed http://localhost:2718?access_token=... URL and work through the delta-new-in-4.4.0.py notebook (identity columns, SHOW PARTITIONS) and the metric-views.py notebook (metric views over catalog-managed Delta) cell by cell.

For the full list of changes — across Delta Spark, Kernel, UniForm, Sharing, and Flink — see the Delta Lake 4.4.0 release notes.

We’d love your feedback on these features, and welcome any questions or comments. Come find us on the Delta Users Slack or the Unity Catalog Slack.