Skip to content

[iceberg] Avoid partition evolution when partition fieldId is not 0#7186

Open
muttcg wants to merge 7 commits intoapache:masterfrom
muttcg:fix/issue_7068
Open

[iceberg] Avoid partition evolution when partition fieldId is not 0#7186
muttcg wants to merge 7 commits intoapache:masterfrom
muttcg:fix/issue_7068

Conversation

@muttcg
Copy link
Contributor

@muttcg muttcg commented Feb 2, 2026

Purpose

Linked issue: close #7068

Handles fieldId incompatibility between Paimon (starts at 0) and Iceberg (starts at 1).

Iceberg REST schema creation shifts all fieldIds by +1, causing field disorder. While schemas can be updated post-creation to start at fieldId 0, creating an empty schema first triggers partition evolution issues that break some query engines, making them believe the partition has evolved and that queries must account for this.

Strategy based on partition field position in the schema:

  • Position 0: Creates empty schema first, because partition evolution unavoidable
  • Position > 0: Creates dummy schema with offset fields and gap filling to preserve the partition spec

Tests

Added new unit test IcebergRestMetadataCommitterTest.testPartitionedPrimaryKeyTableWithNonZeroFieldId
Made tests with Polaris REST catalog and StarRocks

Docs

Added section to rest-catalog.md explaining changes

@muttcg muttcg marked this pull request as draft February 3, 2026 08:47
@muttcg muttcg marked this pull request as ready for review February 3, 2026 13:21
@muttcg muttcg marked this pull request as draft February 3, 2026 13:58
@muttcg muttcg marked this pull request as ready for review February 3, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iceberg REST export creates spec-id=0 (empty) + spec-id=1 partitioned, default-spec-id stays 0

1 participant