[CALCITE-7393] Support RelDataTypeDigest #4761

zhuwenzhuang · 2026-01-22T10:45:20Z

Use RelDataTypeDigest to reduce composite type's digest memory and digest relative operation's latency.
test case: HepPlannerTest.testLargeTypeDigest
This is the first part of latency/memory optimization of large plan.

xiedeyantu · 2026-01-22T13:33:39Z

Thank you for your contribution! I have a few suggestions:
First, please ensure consistency between the Jira title, commit message, and PR title.
Second, please do not arbitrarily change or delete public/protected methods or variables in existing classes (they can be marked as deprecated).
Third, any errors in the CI need to be checked again.

mihaibudiu · 2026-01-22T18:50:06Z

core/src/main/java/org/apache/calcite/rel/type/RelDataTypeDigest.java

+/**
+ * Digest of a RelDataType.
+ */
+public interface RelDataTypeDigest {


This could be two interfaces, in case Digest is useful somewhere else.

I noticed that the hashCode caching in RelDigest and RexCall already addresses some of the hashCode caching/comparison latency issues. Could you please clarify what you mean by “be two interfaces”?

interface HasDigestString { String getDigestString() } interface RelDataTypeDigest extends HasDigestString { RelDataType getType(); }

Other classes may implement HasDigestString.

mihaibudiu · 2026-01-22T18:50:38Z

core/src/main/java/org/apache/calcite/jdbc/JavaRecordType.java

  }

-  @Override public boolean equals(@Nullable Object obj) {
+  @Override public boolean deepEquals(Object obj) {


I think you'll need to leave the old implementations around too, they may break users who expect them to exist.

If we allow user to implement equals/hashCode for nested type, cache for hashCode and pointer compare for deepEquals will not work for them.
This situation is similiar to https://issues.apache.org/jira/browse/CALCITE-3786

mihaibudiu · 2026-01-22T18:52:14Z

core/src/main/java/org/apache/calcite/rel/type/RelDataTypeImpl.java


  protected final @Nullable List<RelDataTypeField> fieldList;
-  protected @Nullable String digest;
+  protected RelDataTypeDigest digest;


Changing protected fields is a breaking change, which has to be reflected in the semantic versioning. This would require a 2.0 release.

On the other hand, I don't know what else you can do to make the digest lazy. Maybe other people can weigh on this dilemma. This is why you should probably file a JIRA issue just for this topic and discuss the design there.

Maybe we can allow user to set String digest to define their own digest. otherwise, we use the RelDataTypeDigest

My suggestion is that some protected/public variables and methods must still be guaranteed to work properly. For example, this
"String digest" might eventually call
"toString" from
"RelDataTypeDigest" to get the value? Because these variables are exposed to users, and someone might currently be using them, we need to provide a compatibility fix. If you think reassigning the original variable would go against your design intention (e.g., it might increase memory usage instead of reducing it), then you need to declare that this is a breaking change.

mihaibudiu · 2026-01-27T17:47:56Z

I don't know how to review this; the change is reasonable in itself, but I don't know if this is an acceptable breaking change. I would appreciate comments from people who have dealt with similar breaking changes in the past.

xiedeyantu

I think this change has a fairly wide impact. I'm wondering if it's possible to control the new/old logic by introducing configuration options. To be honest, I can't judge at this point how the current changes will affect future compatibility. Although your changes may be good, the community probably values compatibility more.

zhuwenzhuang · 2026-02-04T10:40:00Z

@xiedeyantu
Thanks, I think use CalciteSystemProperties to control compatibility is a good idea.
The patch is updated base on 'calcite.disable.generate.type.digest.string', and a benchmark TypeDigestBenchmark added.

Use structured innerDigest instead of string digest for composite/UDT types to reduce memory and improve hashCode/equals latency. Controlled by `calcite.disable.generate.type.digest.string` (default: false). Legacy string digest is still used in hashCode/equals if explicitly set, ensuring backward compatibility. TestCase: TypeDigestBenchmark

sonarqubecloud · 2026-02-04T12:01:19Z

Quality Gate passed

Issues
13 New issues
0 Accepted issues

Measures
0 Security Hotspots
28.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mihaibudiu · 2026-02-06T19:12:36Z

In general you should avoid force-pushing until a review is finished, to make the reviewer's job easier by making it clear what is new.

mihaibudiu

This looks pretty good. Let's see if anyone else has comments.

mihaibudiu · 2026-02-06T19:13:46Z

core/src/main/java/org/apache/calcite/rel/type/RelDataTypeImpl.java


  protected final @Nullable List<RelDataTypeField> fieldList;
-  protected @Nullable String digest;
+  protected @Deprecated @Nullable String digest;


Please add also a @deprecated JavaDoc comment explaining what users are supposed to do instead.

mihaibudiu · 2026-02-06T21:56:31Z

core/src/main/java/org/apache/calcite/config/CalciteSystemProperty.java

+
+  /** Whether to disable generate type digest string.
+   *
+   * <p> Disable generate type digest string for every type can


Please be more explicit here: this is about the RelDataType.digest field.
"Type" can mean many things.

and add deprecated comment

zhuwenzhuang force-pushed the CALCITE-6907 branch from 604cf22 to 04538b2 Compare January 22, 2026 11:34

zhuwenzhuang changed the title ~~[CALCITE-6907] support RelDataTypeDigest~~ [CALCITE-6907] Support RelDataTypeDigest Jan 22, 2026

mihaibudiu reviewed Jan 22, 2026

View reviewed changes

zhuwenzhuang force-pushed the CALCITE-6907 branch from 04538b2 to a03c48a Compare January 23, 2026 11:36

zhuwenzhuang changed the title ~~[CALCITE-6907] Support RelDataTypeDigest~~ [CALCITE-7393] Support RelDataTypeDigest Jan 23, 2026

zhuwenzhuang force-pushed the CALCITE-6907 branch 4 times, most recently from 1f17a42 to 955f537 Compare January 26, 2026 01:49

zhuwenzhuang marked this pull request as draft January 26, 2026 01:55

zhuwenzhuang closed this Jan 26, 2026

zhuwenzhuang force-pushed the CALCITE-6907 branch from 955f537 to f33c6a2 Compare January 26, 2026 01:57

zhuwenzhuang reopened this Jan 26, 2026

zhuwenzhuang force-pushed the CALCITE-6907 branch 2 times, most recently from ac7bf74 to 078f2e4 Compare January 26, 2026 03:09

zhuwenzhuang marked this pull request as ready for review January 26, 2026 03:48

zhuwenzhuang requested review from mihaibudiu and xiedeyantu January 26, 2026 03:49

xiedeyantu reviewed Feb 4, 2026

View reviewed changes

zhuwenzhuang force-pushed the CALCITE-6907 branch from 078f2e4 to 7416868 Compare February 4, 2026 10:30

zhuwenzhuang force-pushed the CALCITE-6907 branch from 7416868 to 38fa020 Compare February 4, 2026 11:28

mihaibudiu approved these changes Feb 6, 2026

View reviewed changes

zhuwenzhuang force-pushed the CALCITE-6907 branch 2 times, most recently from 7c30976 to 9ca8243 Compare February 10, 2026 07:13

zhuwenzhuang force-pushed the CALCITE-6907 branch from 9ca8243 to a38a2d2 Compare February 10, 2026 09:38

[CALCITE-7393] Change flag name

892c828

and add deprecated comment

zhuwenzhuang force-pushed the CALCITE-6907 branch from a38a2d2 to 892c828 Compare February 10, 2026 12:00

[CALCITE-7393] Support RelDataTypeDigest #4761

Are you sure you want to change the base?

[CALCITE-7393] Support RelDataTypeDigest #4761

Uh oh!

Conversation

zhuwenzhuang commented Jan 22, 2026

Uh oh!

xiedeyantu commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented Jan 27, 2026

Uh oh!

xiedeyantu left a comment

Choose a reason for hiding this comment

Uh oh!

zhuwenzhuang commented Feb 4, 2026

Uh oh!

sonarqubecloud bot commented Feb 4, 2026

Quality Gate passed

Uh oh!

mihaibudiu commented Feb 6, 2026

Uh oh!

mihaibudiu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants