GH-825: Add UUID canonical extension type by jhrotko · Pull Request #903 · apache/arrow-java · GitHub

What's Changed

Add Uuid Extension type canonical implementation in Java. The proposed implementation was validated in Dremio engine

Closes #825

See previous PR. It is not sufficient to move the existing files, the "uuid" type used in tests was never meant to implement the canonical type. #824

@lidavidm could you take another look? Let me know if anything else is missing

Thank you for adding the canonical type.

there are some integration tests failling:

  ################# FAILURES #################
  FAILED TEST: custom_metadata Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-json-integration-test -arrow /tmp/tmprky9nvsk/020d3b5c_generated_custom_metadata.json_as_file -json /tmp/arrow-integration-ypsmstn1/generated_custom_metadata.json -mode VALIDATE
  With output:
  --------------
  arrow-json: JSON schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  Arrow schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]
  arrow-json: schemas did not match
  
  --------------
  
  
  FAILED TEST: flight_sql Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=43857 -scenario flight_sql
  With output:
  --------------
  gc 1 @3.115s 0%: 0.078+0.59+0.008 ms clock, 0.31+0.37/0.34/0.60+0.032 ms cpu, 3->3->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
  panic: expected: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
  
  goroutine 1 [running]:
  main.main()
  	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
  
  --------------
  
  
  FAILED TEST: flight_sql:extension Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=35551 -scenario flight_sql:extension
  With output:
  --------------
  panic: expected: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
  
  goroutine 1 [running]:
  main.main()
  	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
  
  --------------
  
  
  FAILED TEST: flight_sql Go producing,  Java consuming
  <class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 40707 -scenario flight_sql
  With output:
  --------------
  WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
  WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
  SLF4J(W): No SLF4J providers were found.
  SLF4J(W): Defaulting to no-operation (NOP) logger implementation
  SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
  Unknown error
  expected: '%!s(bool=true)', got: '%!s(bool=false)'
  
  --------------
  
  
  FAILED TEST: flight_sql:extension Go producing,  Java consuming
  <class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 32823 -scenario flight_sql:extension
  With output:
  --------------
  WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
  WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
  SLF4J(W): No SLF4J providers were found.
  SLF4J(W): Defaulting to no-operation (NOP) logger implementation
  SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
  Unknown error
  expected: '%!s(bool=true)', got: '%!s(bool=false)'
  
  --------------
  
  
  FAILED TEST: custom_metadata Java producing,  Go consuming
  <class 'RuntimeError'>: Go C Data Integration call failed: Schemas are different:
  - Json Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  - Imported Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]

It seems the metadata elements when it's a map the order is not maintained. I this a known issue? @lidavidm

It's apache/arrow-go#571

@lidavidm it was merged, should this be re-run?

@lidavidm anything else missing?

Thank you for addressing the comments.

This has conflicts

Isn't all of this redundant? If we have the static initializer, then ensureRegistered can just be a no-op. Or if we're going to use the atomic boolean, then we don't need the static initializer.

this is only used in tests similarly to Opaque extension type

There is no static initializer there. There is one here.

Thank you! I added the static implementation following your suggestion

Can you rebase please ?

I think these failures are related to upstream and won't go away on rebase.

I will check as I try to stabilize our CI. It fails way too often and it's an issue in PRs.

The PR looks good to me. I will merge as soon as CI is green.

CI failures are not relevant to this PR. I'm going ahead and merging.

## What's Changed The current UUID vector implementation creates new buffer slices when reading values through holders, which has several drawbacks: - Memory overhead: Each slice creates a new ArrowBuf object - Performance impact: Buffer slicing is slower than direct buffer indexing - Inconsistency: Other fixed-width types (like Decimal) use buffer indexing with a `start` offset field ### Proposed Changes 1. Add `start` field to UUID holders to track buffer offsets: - `UuidHolder`: Add `public int start = 0;` - `NullableUuidHolder`: Add `public int start = 0;` 2. Update `UuidVector` to use buffer indexing 3. Update readers and writers ### Related Work - Original UUID extension type implementation: GH-825 (#903) Closes #948

github-actions Bot added the breaking-change label Nov 3, 2025

This comment has been minimized.

Sign in to view

jhrotko force-pushed the GH-825 branch from 595d2ae to 4e8de67 Compare November 3, 2025 14:53

lidavidm requested changes Nov 3, 2025

View reviewed changes

jhrotko changed the title GH-825: Move UuidType to arrow.vector from test GH-825: Implement Uuid canonical type Nov 7, 2025

jhrotko changed the title GH-825: Implement Uuid canonical type GH-825: Add a canonical extension type UUID Nov 7, 2025

jhrotko force-pushed the GH-825 branch from a298920 to ebca624 Compare November 7, 2025 13:58

jhrotko marked this pull request as ready for review November 7, 2025 13:58

jhrotko requested review from jbonofre, laurentgo and wgtmac as code owners November 7, 2025 13:58

jhrotko changed the title GH-825: Add a canonical extension type UUID GH-825: Add UUID canonical extension type Nov 7, 2025

bodduv reviewed Nov 9, 2025

View reviewed changes

Comment thread vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java

Comment thread vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java

lidavidm reviewed Nov 10, 2025

View reviewed changes

lidavidm added enhancement PRs that add or improve features. and removed breaking-change labels Nov 11, 2025

github-actions Bot added this to the 18.4.0 milestone Nov 11, 2025

bodduv approved these changes Nov 14, 2025

View reviewed changes

jhrotko requested a review from lidavidm November 14, 2025 14:13

lidavidm reviewed Nov 17, 2025

View reviewed changes

jhrotko force-pushed the GH-825 branch from b04af47 to da42227 Compare November 17, 2025 13:02

jhrotko requested a review from lidavidm November 17, 2025 13:56

lidavidm reviewed Nov 19, 2025

View reviewed changes

jhrotko requested a review from lidavidm November 25, 2025 10:02

lidavidm approved these changes Nov 26, 2025

View reviewed changes

jbonofre approved these changes Nov 26, 2025

View reviewed changes

jhrotko force-pushed the GH-825 branch from 9b7c583 to 4195376 Compare November 26, 2025 09:18

jhrotko requested review from jbonofre and lidavidm November 26, 2025 10:38

jhrotko added 5 commits November 27, 2025 19:20

apacheGH-825 : Move UuidType to arrow.vector from test

6e2cbe5

Add canonical implementation

06a4bb2

Address comments

3cf2be7

Remove FixedSizeExtensionType

c6d1eb2

remove ensureRegistered method from UuidType

f86d83d

jhrotko force-pushed the GH-825 branch from 4195376 to f86d83d Compare November 27, 2025 19:20

jbonofre merged commit 82baf18 into apache:main Nov 28, 2025
22 of 25 checks passed

jhrotko deleted the GH-825 branch November 28, 2025 12:30

jhrotko mentioned this pull request Jan 7, 2026

GH-948: Use buffer indexing for UUID vector #949

Merged

+                /** Registers the UuidType in the extension type registry. */
+                private static final AtomicBoolean registered = new AtomicBoolean(false);
+                static {
+                  ExtensionTypeRegistry.register(INSTANCE);
+                }
+                /** Register the extension type so it can be used globally. */
+                public static void ensureRegistered() {
+                  if (!registered.getAndSet(true)) {
+                    // The values don't matter, we just need an instance
+                    ExtensionTypeRegistry.register(UuidType.INSTANCE);
+                  }
+                }

GH-825: Add UUID canonical extension type by jhrotko · Pull Request #903 · apache/arrow-java · GitHub

Uh oh!

GH-825: Add UUID canonical extension type - #903

GH-825: Add UUID canonical extension type#903jbonofre merged 5 commits intoapache:mainapache/arrow-java:mainfrom jhrotko:GH-825jhrotko/arrow-java:GH-825Copy head branch name to clipboard

Conversation

jhrotko commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's Changed

Uh oh!

This comment has been minimized.

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

jhrotko commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

lidavidm commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 12, 2025

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

lidavidm Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jhrotko Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

lidavidm Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jhrotko Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jbonofre commented Nov 26, 2025

Uh oh!

lidavidm commented Nov 26, 2025

Uh oh!

jbonofre commented Nov 26, 2025

Uh oh!

jbonofre commented Nov 28, 2025

Uh oh!

jbonofre commented Nov 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

GH-825: Add UUID canonical extension type#903
jbonofre merged 5 commits into
apache:mainfrom
jhrotko:GH-825

jhrotko commented Nov 3, 2025 •
edited
Loading

jhrotko commented Nov 7, 2025 •
edited
Loading