FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

S3 second-pass rewrite by ixcat · Pull Request #408 · datajoint/datajoint-python · GitHub

  1. datajoint
  2. datajoint-python

Repository navigation

S3 second-pass rewrite - #408

Merged
dimitri-yatsenko merged 24 commits into
datajoint:masterfrom
ixcat:s3-nu
Dec 14, 2017
ConversationCommits (24)ChecksFiles changed
Merged

S3 second-pass rewrite#408
dimitri-yatsenko merged 24 commits into
datajoint:masterfrom
ixcat:s3-nu

Conversation

ixcat commented Dec 9, 2017 •
edited by dimitri-yatsenko
Loading

Copy link
Copy Markdown

simplified based update of S3 / #404 based on feedback from dimitri.

includes the 'central' cache work cherry-picked from his branch.

fix #204

dimitri-yatsenko and others added 16 commits December 8, 2017 15:18
implement external storage caching
e53fbaf
fix tests, increment version to 0.10.0
09a6f6e
correct logic of external storage caching
9cc9013
improve external storage caching logic
da23337
implement safe_write for external storage
0875074
s3-nu: initial rework of procedural s3 api. tests tbd.
0b36307
tests/test_s3.py -> tests/test_s3_class.py
085d048
datajoint/__init__.py: zap s3 symbols; these are used directly within…
b89feca 
… external.py
tests/test_s3_class.py: revamp to match procedural api
c802895
datajoint/s3.py: make config error diagnostic more useful
63a45af
add tests/test_s3_mock.py: test s3/ExternalTable w/moto mock logic
18b8f86
tests/test_s3_mock.py: zap old note
7c4c683
zap use of uri-like syntax for s3 bucket
54a7c88
Merge remote-tracking branch 'upstream/master' into s3-nu
46f5b4a
fix tests to work for classes with dropped tables
b541ee7
Add tests/test_s3_00_real.py - real-world AWS/S3 test.
0eeffd3 
See commments for details concerning name (prevent moto errors by test sequence)
and configuration (set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET).

coveralls commented Dec 9, 2017 •
edited
Loading

Copy link
Copy Markdown

Coverage increased (+0.1%) to 91.017% when pulling d7f1a7e on ixcat:s3-nu into 55bfc61 on datajoint:master.

dimitri-yatsenko requested changes Dec 11, 2017
View reviewed changes

dimitri-yatsenko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Will you please merge master?

coveralls commented Dec 13, 2017 •
edited
Loading

Copy link
Copy Markdown

Coverage decreased (-0.07%) to 90.766% when pulling 72bd9b8 on ixcat:s3-nu into c815421 on datajoint:master.

dimitri-yatsenko requested changes Dec 13, 2017
View reviewed changes
Comment thread datajoint/external.py Outdated

def safe_write(filename, blob):
"""
A two-step write.

dimitri-yatsenko Dec 13, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Please remove this safe_write. It's implemented below.

dimitri-yatsenko requested changes Dec 13, 2017
View reviewed changes
Comment thread datajoint/external.py Outdated
except FileNotFoundError:
raise DataJointError('Lost external blob')
elif protocol == 's3':
blob = s3.get(self.database, store, blob, hash)

dimitri-yatsenko Dec 13, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Bug: hash should be blob_hash

coveralls commented Dec 13, 2017 •
edited
Loading

Copy link
Copy Markdown

Coverage increased (+0.08%) to 90.917% when pulling da87545 on ixcat:s3-nu into c815421 on datajoint:master.

coveralls commented Dec 13, 2017 •
edited
Loading

Copy link
Copy Markdown

Coverage increased (+0.08%) to 90.917% when pulling da87545 on ixcat:s3-nu into c815421 on datajoint:master.

dimitri-yatsenko requested changes Dec 13, 2017
View reviewed changes
Comment thread datajoint/external.py Outdated
except FileNotFoundError:
raise DataJointError('Lost external blob')
elif protocol == 's3':
blob = s3.get(self.database, store, blob, blob_hash)

dimitri-yatsenko Dec 13, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

bug. s3.get does not take a blob as an input.

ixcat changed the title S3 second-pass rewrite [WIP] S3 second-pass rewrite Dec 13, 2017

coveralls commented Dec 13, 2017 •
edited
Loading

Copy link
Copy Markdown

Coverage increased (+0.08%) to 90.917% when pulling d5d1dd6 on ixcat:s3-nu into c815421 on datajoint:master.

1 similar comment

coveralls commented Dec 13, 2017

Copy link
Copy Markdown

Coverage increased (+0.08%) to 90.917% when pulling d5d1dd6 on ixcat:s3-nu into c815421 on datajoint:master.

coveralls commented Dec 13, 2017 •
edited
Loading

Copy link
Copy Markdown

Coverage increased (+0.08%) to 90.917% when pulling d1ae669 on ixcat:s3-nu into c815421 on datajoint:master.

dimitri-yatsenko approved these changes Dec 14, 2017
View reviewed changes
Comment thread datajoint/s3.py
def make_rpath_name(location, database, blob_hash):
rpath = '{l}/{d}/{h}'.format(l=location, d=database, h=blob_hash)
# s3 is '' rooted; prevent useless '/' top-level 'directory'
return rpath[1:] if rpath[0] == '/' else rpath

dimitri-yatsenko Dec 14, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality 
return rpath.lstrip('/')

eywalker changed the title [WIP] S3 second-pass rewrite S3 second-pass rewrite Dec 14, 2017
dimitri-yatsenko merged commit 70eb28e into datajoint:master Dec 14, 2017
ixcat deleted the s3-nu branch October 31, 2018 13:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

dimitri-yatsenko dimitri-yatsenko approved these changes
eywalker Awaiting requested review from eywalker

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support AWS S3 storage of blobs

3 participants

Back | FazBrowse Home | New Git URL