fix(spark): address review feedback on staging filesystem PR · feast-dev/feast@feec382 · GitHub

feast-dev / feast Public

• Notifications You must be signed in to change notification settings
• Fork 1.4k
• Star 7.2k

• Code
• Issues 214
• Pull requests 189
• Discussions
• Actions
• Security and quality 1
• Insights

Additional navigation options

Commit feec382

Browse filesBrowse the repository at this point in the historyBrowse files

abhijeet-dhumal

committed

fix(spark): address review feedback on staging filesystem PR

Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>

1 parent 11b765c commit feec382
Copy full SHA for feec382

1 file changed

+6-6Lines changed: 6 additions & 6 deletions

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

• sdk/python/feast/infra/offline_stores/contrib/spark_offline_store
  • spark.py

Expand file treeCollapse file tree

Open diff view settings

Collapse file

‎sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py‎

Copy file name to clipboardExpand all lines: sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py

+6-6Lines changed: 6 additions & 6 deletions

Original file line number	Diff line number	Diff line change
@@ -492,9 +492,9 @@ def _resolve_staging_filesystem(
492	492	"""Return (pyarrow filesystem, prefix-stripped paths) for staging URIs."""
493	493	sample = paths[0]
494	494
495		- if sample.startswith("s3://") or sample.startswith("s3a://"):
496		- import pyarrow.fs as pafs
	495	+ import pyarrow.fs as pafs
497	496
	497	+ if sample.startswith("s3://") or sample.startswith("s3a://"):
498	498	endpoint = os.environ.get("AWS_ENDPOINT_URL_S3") or os.environ.get(
499	499	"AWS_S3_ENDPOINT", ""
500	500	)
@@ -504,16 +504,16 @@ def _resolve_staging_filesystem(
504	504	kwargs: Dict[str, Any] = {"region": region}
505	505	if endpoint:
506	506	kwargs["endpoint_override"] = (
507		- endpoint.rstrip("/").replace("https://", "").replace("http://", "")
	507	+ endpoint.rstrip("/")
	508	+ .removeprefix("https://")
	509	+ .removeprefix("http://")
508	510	)
509	511	kwargs["scheme"] = "https" if endpoint.startswith("https") else "http"
510	512	fs = pafs.S3FileSystem(**kwargs)
511		- stripped = [p.replace("s3a://", "").replace("s3://", "") for p in paths]
	513	+ stripped = [p.removeprefix("s3a://").removeprefix("s3://") for p in paths]
512	514	return fs, stripped
513	515
514	516	if sample.startswith("gs://"):
515		- import pyarrow.fs as pafs
516		-
517	517	fs = pafs.GcsFileSystem()
518	518	stripped = [p[len("gs://") :] for p in paths]
519	519	return fs, stripped

0 commit comments

Comments

0 (0)