Repository navigation
Support AWS S3 storage of blobs #204
Description
Since S3 storage is less expensive than RDS storage, let's implement transparent storage of blobs in S3 buckets. Blobs represent 99% of the overall data volume of a typical datajoint database.
Let's discuss the implementation of this feature.
Metadata
Metadata
Assignees
Labels
No labels
Activity
fabiansinz commented on Apr 21, 2016
Why limit to S3? I vote for defining an interface that can possibly hook into many storage backends like simple hard-drives, Dropbox, sqlite ... . We don't necessarily need to implement it right now but we should program it such that it can easily be done.
My idea is to translate blobs into general file identifiers. The problem there will be to come up with a way that works reliably across systems and programming languages. How that identifier is translated into the exact path depends on the backend which we set in the configuration. The question is: Do we want to implement it for all blobs at once or only for some blobs? In the former case we could use a new data type, in the latter case we can still go with blobs.
dimitri-yatsenko commented on Apr 21, 2016
Should we add a setting to config to specify the storage of blobs and the path format? We may set a size threshold above which the blob goes to storage or add a special 'stored_blob' datatype.
dimitri-yatsenko commented on Apr 21, 2016
Specification:
A new attribute datatype will be added for externally stored data. The datatype name will be external.
The config will specify external_storage_type, which will be one of ("file-system", "aws-s3"). This list may be extended in the future.
The config will specify external_storage_path, that might correspond to the root path for the file system or the bucket-name/folder-name for S3.
Finally, config will specify external_storage_filename template which can use variables such as $TABLE_NAME, $SCHEMA_NAME, $KEY_VALUE, $KEY_HASH, $DATA_HASH. For example, config.external_storage_filename="$SCHEMA_NAME/$TABLE_NAME/$KEY_VALUE-$DATA_HASH".
$KEY_VALUE is the ascii serialized human-readable representation of the key values, in order. E.g. 381-1-2.
insert will exit with an error and will fail to insert if the saving of an external field failed.
If a table has multiple external fields, then insert will also ensure that either all or none of the fields are stored. If none are stored, the tuple is not inserted.
insert will throw an error if the external field already exists. Versioning will not be supported.
delete and drop will attempt to delete all the external fields after the tuple is deleted.
peabody124 commented on Apr 22, 2016
Why not just have the external datatype be stored as a string with a uri?
fabiansinz commented on Apr 22, 2016
@peabody124: In the end it will be a string in the database.
A few comments to the proposed specification:
All classes that implement external resource storage should inherit from a class ExternalResourceIO that defines the following abstract methods: store, fetch, exists, delete.
Here is a list of things that need to be changed for that specification. Please feel free to complete:
dimitri-yatsenko commented on Apr 22, 2016
@fabiansinz:
So we agree that a new datatype for external storage will be added alongside in addition to the the conventional blobs?
fabiansinz commented on Apr 25, 2016
@dimitri-yatsenko
Yes, I think a new datatype is the best way to go.
Re 1: What if the user chooses something that is not unique for each tuple? We could avoid that problem if we require to have at least one hash in external_storage_filename.
Re 2: I vote for storage.*
dimitri-yatsenko commented on Oct 20, 2017
This feature will be implemented according to this spec: https://docs.datajoint.io/data-definition/External-store.html