Add support to configure the shuffle client library with a property. · alanma/DataflowJavaSDK@e64446c · GitHub

1. alanma
2. DataflowJavaSDK

Repository navigation

• Code
• Pull requests
• Actions
• Projects
• Security and quality
• Insights

Commit e64446c

Browse filesBrowse the repository at this point in the historyBrowse files

tudorm

authored and

davorbonaci

committed

Add support to configure the shuffle client library with a property.

----Release Notes---- [] ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=113307012

1 parent 76b8689 commit e64446c
Copy full SHA for e64446c

1 file changed

+6-2Lines changed: 6 additions & 2 deletions

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

• sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker
  • ShuffleLibrary.java

Expand file treeCollapse file tree

Open diff view settings

Collapse file

‎sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/ShuffleLibrary.java‎

Copy file name to clipboardExpand all lines: sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/ShuffleLibrary.java

+6-2Lines changed: 6 additions & 2 deletions

Original file line number	Diff line number	Diff line change
@@ -32,9 +32,13 @@ class ShuffleLibrary {
32	32	*/
33	33	static void load() {
34	34	try {
	35	+ final String shuffleClientLibraryPropertyKey = "batch.shuffle_client_library";
	36	+ String shuffleClientLibrary = "libshuffle_client_jni.so.stripped";
	37	+ if (System.getProperties().containsKey(shuffleClientLibraryPropertyKey)) {
	38	+ shuffleClientLibrary = System.getProperty(shuffleClientLibraryPropertyKey);
	39	+ }
35	40	File tempfile = File.createTempFile("libshuffle_client_jni", ".so");
36		- InputStream input = ClassLoader.getSystemResourceAsStream(
37		- "libshuffle_client_jni.so.stripped");
	41	+ InputStream input = ClassLoader.getSystemResourceAsStream(shuffleClientLibrary);
38	42	Files.copy(input, tempfile.toPath(), StandardCopyOption.REPLACE_EXISTING);
39	43	System.load(tempfile.getAbsolutePath());
40	44	} catch (IOException e) {

0 commit comments

Comments

0 (0)