| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
For the JNI Ubuntu, I see this message from archery: × python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
Python < 3.10 is not supported
[end of output]
Checking for an update. |
Sorry, something went wrong.
|
https://github.com/apache/arrow-java/blob/main/compose.yaml#L102 I think we need to update our .env to pick up a newer tag like amd64-python-3.13-wheel-manylinux-2014-vcpkg-f7423ee180c4b7f40d43402c2feb3859161ef625 |
Sorry, something went wrong.
|
Yup agree. It's what I gonna try. |
Sorry, something went wrong.
|
The error with cffi not being available for free-threaded means it's trying to use free-threaded wheels (an python) which is unexpected from what I understand: RuntimeError: CFFI does not support the free-threaded build of CPython 3.13. Upgrade to free-threaded 3.14 or newer to use CFFI with the free-threaded build. |
Sorry, something went wrong.
|
We need apache/arrow#47749 and diff --git a/ci/scripts/jni_macos_build.sh b/ci/scripts/jni_macos_build.sh
index f7543b6f..7f927d19 100755
--- a/ci/scripts/jni_macos_build.sh
+++ b/ci/scripts/jni_macos_build.sh
@@ -142,6 +142,7 @@ github_actions_group_begin "Checking shared dependencies for libraries"
pushd "${dist_dir}"
archery linking check-dependencies \
--allow CoreFoundation \
+ --allow Network \
--allow Security \
--allow libSystem \
--allow libarrow_cdata_jni \for macOS. |
Sorry, something went wrong.
|
We need apache/arrow#47746 and diff --git a/.env b/.env
index d3e1c1d6..a7783537 100644
--- a/.env
+++ b/.env
@@ -40,7 +40,7 @@ ARCH_SHORT=amd64
# Default repository to pull and push images from
REPO=ghcr.io/apache/arrow-java-dev
-ARROW_REPO=apache/arrow-dev
+ARROW_REPO=ghcr.io/apache/arrow-dev
# The setup attempts to generate coredumps by default, in order to disable the
# coredump generation set it to 0
@@ -53,5 +53,4 @@ MAVEN=3.9.9
# Versions for various dependencies used to build artifacts
# Keep in sync with apache/arrow
ARROW_REPO_ROOT=./arrow
-PYTHON=3.9
-VCPKG="f7423ee180c4b7f40d43402c2feb3859161ef625" # 2024.06.15 Release
+VCPKG="4334d8b4c8916018600212ab4dd4bbdc343065d1" # 2025.09.17 Release
diff --git a/ci/docker/vcpkg-jni.dockerfile b/ci/docker/vcpkg-jni.dockerfile
index 55fa35e0..d6bd322a 100644
--- a/ci/docker/vcpkg-jni.dockerfile
+++ b/ci/docker/vcpkg-jni.dockerfile
@@ -18,24 +18,10 @@
ARG base
FROM ${base}
-# Install the libraries required by Gandiva to run
-# Use enable llvm[enable-rtti] in the vcpkg.json to avoid link problems in Gandiva
-RUN vcpkg install \
- --clean-after-build \
- --x-install-root=${VCPKG_ROOT}/installed \
- --x-manifest-root=/arrow/ci/vcpkg \
- --x-feature=dev \
- --x-feature=flight \
- --x-feature=gcs \
- --x-feature=json \
- --x-feature=parquet \
- --x-feature=gandiva \
- --x-feature=s3
-
# Install Java
# We need Java for JNI headers, but we don't invoke Maven in this build.
ARG java=11
-RUN yum install -y java-$java-openjdk-devel && yum clean all
+RUN dnf install -y java-$java-openjdk-devel && dnf clean all
# For ci/scripts/{cpp,java}_*.sh
ENV ARROW_HOME=/tmp/local \
diff --git a/compose.yaml b/compose.yaml
index b125c3c9..f5082a22 100644
--- a/compose.yaml
+++ b/compose.yaml
@@ -99,7 +99,7 @@ services:
cache_from:
- ${REPO}:${ARCH}-vcpkg-jni-${VCPKG}
args:
- base: ${ARROW_REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2014-vcpkg-${VCPKG}
+ base: ${ARROW_REPO}:${ARCH}-cpp-jni-${VCPKG}
volumes:
- .:/arrow-java:delegated
- ${ARROW_REPO_ROOT}:/arrow:delegatedfor Linux. |
Sorry, something went wrong.
|
I'm fixing the base image here (need to adapt). |
Sorry, something went wrong.
|
apache/arrow#47749 was merged and JNI jobs on macOS passed. |
Sorry, something went wrong.
|
@kou macos works now thanks. But not Ubuntu yet. I'm investigating. |
Sorry, something went wrong.
It's not merged yet. It's a blocker for Linux. |
Sorry, something went wrong.
and identify fixes needed to have CI happy
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Sorry, something went wrong.
|
apache/arrow#47915 can fix the symbol error but it seems that Java process is crashed: 2025-10-23T06:05:57.7339179Z [ERROR] See /home/runner/work/arrow-java/arrow-java/dataset/target/surefire-reports for the individual test results.
2025-10-23T06:05:57.7340624Z [ERROR] See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
2025-10-23T06:05:57.7342010Z [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
2025-10-23T06:05:57.7362032Z [ERROR] Command was /bin/sh -c cd '/home/runner/work/arrow-java/arrow-java/dataset' && '/usr/lib/jvm/temurin-17-jdk-amd64/bin/java' '--add-reads=org.apache.arrow.dataset=com.fasterxml.jackson.databind' '--add-opens=java.base/java.nio=org.apache.arrow.dataset,org.apache.arrow.memory.core,ALL-UNNAMED' '-jar' '/home/runner/work/arrow-java/arrow-java/dataset/target/surefire/surefirebooter-20251023060545407_171.jar' '/home/runner/work/arrow-java/arrow-java/dataset/target/surefire' '2025-10-23T05-55-19_252-jvmRun1' 'surefire-20251023060545407_167tmp' 'surefire_33-20251023060545407_169tmp'
2025-10-23T06:05:57.7366551Z [ERROR] Error occurred in starting fork, check output in log
2025-10-23T06:05:57.7367142Z [ERROR] Process Exit Code: 134
2025-10-23T06:05:57.7367590Z [ERROR] Crashed tests:
2025-10-23T06:05:57.7368213Z [ERROR] org.apache.arrow.dataset.substrait.TestAceroSubstraitConsumer
2025-10-23T06:05:57.7369823Z [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
2025-10-23T06:05:57.7374432Z [ERROR] Command was /bin/sh -c cd '/home/runner/work/arrow-java/arrow-java/dataset' && '/usr/lib/jvm/temurin-17-jdk-amd64/bin/java' '--add-reads=org.apache.arrow.dataset=com.fasterxml.jackson.databind' '--add-opens=java.base/java.nio=org.apache.arrow.dataset,org.apache.arrow.memory.core,ALL-UNNAMED' '-jar' '/home/runner/work/arrow-java/arrow-java/dataset/target/surefire/surefirebooter-20251023060545407_171.jar' '/home/runner/work/arrow-java/arrow-java/dataset/target/surefire' '2025-10-23T05-55-19_252-jvmRun1' 'surefire-20251023060545407_167tmp' 'surefire_33-20251023060545407_169tmp'
2025-10-23T06:05:57.7378464Z [ERROR] Error occurred in starting fork, check output in log
2025-10-23T06:05:57.7379052Z [ERROR] Process Exit Code: 134
2025-10-23T06:05:57.7379495Z [ERROR] Crashed tests:
2025-10-23T06:05:57.7380123Z [ERROR] org.apache.arrow.dataset.substrait.TestAceroSubstraitConsumer
2025-10-23T06:05:57.7381305Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:456)
2025-10-23T06:05:57.7382800Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:358)
2025-10-23T06:05:57.7384130Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:296)
2025-10-23T06:05:57.7385567Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
2025-10-23T06:05:57.7386899Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1337)
2025-10-23T06:05:57.7388551Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1135)
2025-10-23T06:05:57.7390302Z [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:969)
2025-10-23T06:05:57.7391570Z [ERROR] at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
2025-10-23T06:05:57.7392808Z [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2025-10-23T06:05:57.7393916Z [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:569)
2025-10-23T06:05:57.7394713Z [ERROR] at com.gradle.maven.cache.extension.c.n.a(SourceFile:53)
2025-10-23T06:05:57.7395641Z [ERROR] at com.gradle.maven.cache.extension.c.d.a(SourceFile:27)
2025-10-23T06:05:57.7396382Z [ERROR] at com.gradle.maven.cache.extension.c.q.a(SourceFile:23)
2025-10-23T06:05:57.7397145Z [ERROR] at com.gradle.maven.cache.extension.c.j.a(SourceFile:28)
2025-10-23T06:05:57.7397904Z [ERROR] at com.gradle.maven.cache.extension.c.p.a(SourceFile:27)
2025-10-23T06:05:57.7398647Z [ERROR] at com.gradle.maven.cache.extension.c.b.c(SourceFile:118)
2025-10-23T06:05:57.7399405Z [ERROR] at com.gradle.maven.cache.extension.c.b.a(SourceFile:62)
2025-10-23T06:05:57.7400154Z [ERROR] at com.gradle.maven.cache.extension.c.g.a(SourceFile:27)
2025-10-23T06:05:57.7400901Z [ERROR] at com.gradle.maven.cache.extension.c.a.a(SourceFile:46)
2025-10-23T06:05:57.7401607Z [ERROR] at com.gradle.maven.cache.extension.c.o.a(SourceFile:18)
2025-10-23T06:05:57.7402319Z [ERROR] at com.gradle.maven.cache.extension.c.a.a(SourceFile:46)
2025-10-23T06:05:57.7403012Z [ERROR] at com.gradle.maven.cache.extension.c.c.a(SourceFile:26)
2025-10-23T06:05:57.7403767Z [ERROR] at com.gradle.maven.cache.extension.c.h$1.run(SourceFile:35)
2025-10-23T06:05:57.7404749Z [ERROR] at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(SourceFile:30)
2025-10-23T06:05:57.7406286Z [ERROR] at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(SourceFile:27)
2025-10-23T06:05:57.7407496Z [ERROR] at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(SourceFile:67)
2025-10-23T06:05:57.7408598Z [ERROR] at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(SourceFile:60)
2025-10-23T06:05:57.7409694Z [ERROR] at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(SourceFile:167)
2025-10-23T06:05:57.7410874Z [ERROR] at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(SourceFile:60)
2025-10-23T06:05:57.7411981Z [ERROR] at org.gradle.internal.operations.DefaultBuildOperationRunner.run(SourceFile:48)
2025-10-23T06:05:57.7412947Z [ERROR] at com.gradle.maven.cache.extension.c.h.a(SourceFile:31)
2025-10-23T06:05:57.7413694Z [ERROR] at com.gradle.maven.cache.extension.c.m.a(SourceFile:80)
2025-10-23T06:05:57.7414547Z [ERROR] at com.gradle.maven.cache.extension.g.b.lambda$createProxy$0(SourceFile:76)
2025-10-23T06:05:57.7430662Z [ERROR] at jdk.proxy31/jdk.proxy31.$Proxy139.execute(Unknown Source)
2025-10-23T06:05:57.7431546Z [ERROR] at com.gradle.maven.scan.extension.internal.d.b.executeMojo(SourceFile:116)
2025-10-23T06:05:57.7432622Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
2025-10-23T06:05:57.7433731Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
2025-10-23T06:05:57.7434767Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
2025-10-23T06:05:57.7442486Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
2025-10-23T06:05:57.7443657Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
2025-10-23T06:05:57.7444785Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
2025-10-23T06:05:57.7454565Z [ERROR] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
2025-10-23T06:05:57.7456172Z [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
2025-10-23T06:05:57.7457742Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
2025-10-23T06:05:57.7459393Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
2025-10-23T06:05:57.7461035Z [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
2025-10-23T06:05:57.7462605Z [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
2025-10-23T06:05:57.7463701Z [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
2025-10-23T06:05:57.7464635Z [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
2025-10-23T06:05:57.7465624Z [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
2025-10-23T06:05:57.7466415Z [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906)
2025-10-23T06:05:57.7467207Z [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)
2025-10-23T06:05:57.7467980Z [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)
2025-10-23T06:05:57.7468918Z [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2025-10-23T06:05:57.7470166Z [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2025-10-23T06:05:57.7471634Z [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2025-10-23T06:05:57.7472815Z [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:569)
2025-10-23T06:05:57.7473810Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:255)
2025-10-23T06:05:57.7475177Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:201)
2025-10-23T06:05:57.7476314Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:361)
2025-10-23T06:05:57.7477468Z [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:314)
2025-10-23T06:05:57.7479073Z [ERROR] Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
2025-10-23T06:05:57.7483583Z [ERROR] Command was /bin/sh -c cd '/home/runner/work/arrow-java/arrow-java/dataset' && '/usr/lib/jvm/temurin-17-jdk-amd64/bin/java' '--add-reads=org.apache.arrow.dataset=com.fasterxml.jackson.databind' '--add-opens=java.base/java.nio=org.apache.arrow.dataset,org.apache.arrow.memory.core,ALL-UNNAMED' '-jar' '/home/runner/work/arrow-java/arrow-java/dataset/target/surefire/surefirebooter-20251023060545407_171.jar' '/home/runner/work/arrow-java/arrow-java/dataset/target/surefire' '2025-10-23T05-55-19_252-jvmRun1' 'surefire-20251023060545407_167tmp' 'surefire_33-20251023060545407_169tmp'
2025-10-23T06:05:57.7487688Z [ERROR] Error occurred in starting fork, check output in log
2025-10-23T06:05:57.7488315Z [ERROR] Process Exit Code: 134
2025-10-23T06:05:57.7488748Z [ERROR] Crashed tests:
2025-10-23T06:05:57.7489383Z [ERROR] org.apache.arrow.dataset.substrait.TestAceroSubstraitConsumer
2025-10-23T06:05:57.7490456Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
2025-10-23T06:05:57.7491742Z [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.lambda$null$3(ForkStarter.java:350)
2025-10-23T06:05:57.7492881Z [ERROR] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2025-10-23T06:05:57.7493932Z [ERROR] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2025-10-23T06:05:57.7495315Z [ERROR] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2025-10-23T06:05:57.7496267Z [ERROR] at java.base/java.lang.Thread.run(Thread.java:840)
2025-10-23T06:05:57.7496821Z [ERROR]
2025-10-23T06:05:57.7497152Z [ERROR] -> [Help 1]
BTW, can we reduce log from Maven? We can't use GitHub Actions UI to see logs because there are too many logs. (We need to use raw log instead...) |
Sorry, something went wrong.
|
Hmm, we've been running into this for a while now but haven't managed to actually track it down. |
Sorry, something went wrong.
|
We could disable the test. (Or just disable all tests...the JNI bindings are not very well tested in the first place.) |
Sorry, something went wrong.
…47915) ### Rationale for this change We need to use `orc_SOURCE` not `ORC_SOURCE` for Apache ORC and `glog_SOURCE` not `GLOG_SOURCE` for Google logging. ### What changes are included in this PR? Fix referred variable names. ### Are these changes tested? Yes in apache/arrow-java#865 . ### Are there any user-facing changes? Yes. * GitHub Issue: #47914 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
|
I can replicate the crash locally. I also see this: java.lang.RuntimeException: No function registered with name: make_struct Not sure what's happening here. Linking issue? |
Sorry, something went wrong.
|
I can see the same in CI. I thought compute functions were automatically registered? Or did something about that change over time |
Sorry, something went wrong.
|
I don't think I have the time to investigate this. I'm at least willing to disable the tests and declare that dataset needs maintainers in order to continue... |
Sorry, something went wrong.
|
Ah, we need to call arrow::compute::Initialize() explicitly with recent Apache Arrow C++. |
Sorry, something went wrong.
Sorry, something went wrong.
|
Ok, let me try that, thanks :) |
Sorry, something went wrong.
|
IMO, let's merge this PR as-is though? We can start unblocking other PRs and this has been around for a while |
Sorry, something went wrong.
|
OK. @jbonofre Could you create a new issue for this and update the PR title/description for the latest changes? |
Sorry, something went wrong.
Sorry, something went wrong.
|
@jbonofre do you mind if I just file the issue and close this out? |
Sorry, something went wrong.
|
@lidavidm maybe we can start by merging this one and create another issue/PR as follow up ? |
Sorry, something went wrong.
…tion (apache#47915) ### Rationale for this change We need to use `orc_SOURCE` not `ORC_SOURCE` for Apache ORC and `glog_SOURCE` not `GLOG_SOURCE` for Google logging. ### What changes are included in this PR? Fix referred variable names. ### Are these changes tested? Yes in apache/arrow-java#865 . ### Are there any user-facing changes? Yes. * GitHub Issue: apache#47914 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
…tion (apache#47915) ### Rationale for this change We need to use `orc_SOURCE` not `ORC_SOURCE` for Apache ORC and `glog_SOURCE` not `GLOG_SOURCE` for Google logging. ### What changes are included in this PR? Fix referred variable names. ### Are these changes tested? Yes in apache/arrow-java#865 . ### Are there any user-facing changes? Yes. * GitHub Issue: apache#47914 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
| Back | FazBrowse Home | New Git URL |
Closes #898.