FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[AURON #2510] Add non-UTC timestamp coverage for native datediff by Sigma-Ma · Pull Request #2520 · apache/auron · GitHub

/ auron Public

[AURON #2510] Add non-UTC timestamp coverage for native datediff - #2520

Open
Sigma-Ma wants to merge 3 commits into
apache:masterfrom
Sigma-Ma:Auron-2510-add-non-utc-timestamp-coverage-for-native-datediff
ConversationCommits (3)ChecksFiles changed
Open

[AURON #2510] Add non-UTC timestamp coverage for native datediff#2520
Sigma-Ma wants to merge 3 commits into
apache:masterfrom
Sigma-Ma:Auron-2510-add-non-utc-timestamp-coverage-for-native-datediff

Conversation

Sigma-Ma commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2510

Rationale for this change

The datediff coverage added in #2502 only uses DATE inputs. TIMESTAMP inputs also need coverage for conversion to dates in the session time zone.

What changes are included in this PR?

Add a regression test using TIMESTAMP columns in America/Los_Angeles. It covers local and UTC date boundaries, positive and negative differences, and null inputs.

The test checks explicit expected results, compares with Spark, and verifies native operators.

Are there any user-facing changes?

No. This change only adds regression coverage.

How was this patch tested?

On Spark 3.5.8 with Scala 2.12 and JDK 8:

./build/mvn -B -ntp -Ppre -Pspark-3.5 -Pscala-2.12 -pl spark-extension-shims-spark -DskipBuildNative -DskipTests=false '-Dsuites=org.apache.auron.AuronFunctionSuite datediff' test
./build/mvn -B -ntp -Ppre -Pspark-3.5 -Pscala-2.12 -pl spark-extension-shims-spark -DskipBuildNative -DskipTests=false -Dsuites=org.apache.auron.AuronFunctionSuite test
./dev/reformat --check

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: OpenAI Codex (GPT-6)

github-actions Bot added the spark label Sep 10, 2026
slfan1989 self-assigned this Sep 10, 2026
slfan1989 requested changes Sep 10, 2026
View reviewed changes

slfan1989 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Thanks for adding this coverage.

The test cases clearly distinguish session-local date boundaries from UTC boundaries and also cover signed and null results.

However, the latest CI is consistently failing for Spark 4.0, 4.1, and 4.2, while the Spark 3.x jobs pass.

Could you please investigate the Spark 4.x failures and either fix the compatibility issue or confirm that they are unrelated before merging?

Sigma-Ma commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for adding this coverage.

The test cases clearly distinguish session-local date boundaries from UTC boundaries and also cover signed and null results.

However, the latest CI is consistently failing for Spark 4.0, 4.1, and 4.2, while the Spark 3.x jobs pass.

Could you please investigate the Spark 4.x failures and either fix the compatibility issue or confirm that they are unrelated before merging?

Thanks for pointing this out. checkSparkAnswerAndOperator already collects the Auron DataFrame, and the subsequent checkAnswer(df, ...) collected the same AQE plan again. Spark 4.x hits an internal assertion during that second execution.

I updated the explicit expected-result check to use a fresh DataFrame, while keeping the Spark comparison and native operator assertion. The focused test passes locally with Spark 4.0 and Scala 2.13. Spark 4.1 and 4.2 will be verified by CI.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

slfan1989 slfan1989 requested changes

Requested changes must be addressed to merge this pull request.

Assignees

slfan1989

Labels

spark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add non-UTC timestamp coverage for native datediff

2 participants

Back | FazBrowse Home | New Git URL