MAST: Read product function for MastMissions - #3593
Conversation
Sorry, something went wrong.
Codecov Report❌ Patch coverage is 88.46154% with 6 lines in your changes missing coverage. Please review.
@@ Coverage Diff @@
## main #3593 +/- ##
==========================================
+ Coverage 73.42% 73.44% +0.02%
==========================================
Files 230 230
Lines 21360 21400 +40
==========================================
+ Hits 15684 15718 +34
- Misses 5676 5682 +6 ☔ View full report in Codecov by Harness.
|
Sorry, something went wrong.
|
Until we have a more mature version ready, can we please ensure documentation / tutorials call out that for FITS files read_product() still downloads the file locally, so people working on Roman work who may want to use this to download HST or JWST FITS files aren't surprised if read_product() is using up credits writing to local cloud storage on them? I presume this can be avoided once there's a "get_s3_uris()" endpoint in MissionMAST for all the supported missions beyond Roman? |
Sorry, something went wrong.
bsipocz
left a comment
There was a problem hiding this comment.
code changes look good, but I have problem with the proposed changes in pyproject.toml, IMO we should avoid workaround for [all] and should not include optional dependencies to be installed for all the test jobs.
Sorry, something went wrong.
| "sphinx-astropy>=1.5", | ||
| "scipy", | ||
| "fsspec[http]", | ||
| "asdf" |
There was a problem hiding this comment.
where is it used in the docs?
Sorry, something went wrong.
There was a problem hiding this comment.
I think I thought I had to include it for the code blocks. Removed!
Sorry, something went wrong.
| "fsspec[http]", | ||
| "fsspec[http,s3]", | ||
| "moto[s3]", | ||
| "s3fs", | ||
| "asdf" |
There was a problem hiding this comment.
I don't think that optional dependencies should be installed for all test runs. The fact that we have fsspec and s3fs listed here is a problem of its own and should be cleaned up. So, I would not think we should add asdf.
Sorry, something went wrong.
A function in the MastMissions class that can read products into memory.
For FITS files, we use astropy.io.fits.open. The product files are downloaded and cached by Astropy, and then read into a HDUList object.
For ASDF files, we use fsspec and asdf. Our downloads for Roman data are redirected to S3, so these files are not downloaded and are streamed directly into asdf.AsdfFile objects.