| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -15,10 +15,16 @@ | |||
| 15 | 15 | * :mod:`dbm.ndbm` | |
| 16 | 16 | ||
| 17 | 17 | If none of these modules are installed, the | |
| 18 | - slow-but-simple implementation in module :mod:`dbm.dumb` will be used. There | ||
| 18 | + slow-but-simple implementation in module :mod:`dbm.dumb` will be used. There | ||
| 19 | 19 | is a `third party interface <https://www.jcea.es/programacion/pybsddb.htm>`_ to | |
| 20 | 20 | the Oracle Berkeley DB. | |
| 21 | 21 | ||
| 22 | + .. note:: | ||
| 23 | + None of the underlying modules will automatically shrink the disk space used by | ||
| 24 | + the database file. However, :mod:`dbm.sqlite3`, :mod:`dbm.gnu` and :mod:`dbm.dumb` | ||
| 25 | + provide a :meth:`!reorganize` method that can be used for this purpose. | ||
| 26 | + | ||
| 27 | + | ||
| 22 | 28 | .. exception:: error | |
| 23 | 29 | ||
| 24 | 30 | A tuple containing the exceptions that can be raised by each of the supported | |
@@ -186,6 +192,17 @@ or any other SQLite browser, including the SQLite CLI. | |||
| 186 | 192 | The Unix file access mode of the file (default: octal ``0o666``), | |
| 187 | 193 | used only when the database has to be created. | |
| 188 | 194 | ||
| 195 | + .. method:: sqlite3.reorganize() | ||
| 196 | + | ||
| 197 | + If you have carried out a lot of deletions and would like to shrink the space | ||
| 198 | + used on disk, this method will reorganize the database; otherwise, deleted file | ||
| 199 | + space will be kept and reused as new (key, value) pairs are added. | ||
| 200 | + | ||
| 201 | + .. note:: | ||
| 202 | + While reorganizing, as much as two times the size of the original database is required | ||
| 203 | + in free disk space. However, be aware that this factor changes for each :mod:`dbm` submodule. | ||
| 204 | + | ||
| 205 | + .. versionadded:: next | ||
| 189 | 206 | ||
| 190 | 207 | :mod:`dbm.gnu` --- GNU database manager | |
| 191 | 208 | --------------------------------------- | |
@@ -284,6 +301,10 @@ functionality like crash tolerance. | |||
| 284 | 301 | reorganization; otherwise, deleted file space will be kept and reused as new | |
| 285 | 302 | (key, value) pairs are added. | |
| 286 | 303 | ||
| 304 | + .. note:: | ||
| 305 | + While reorganizing, as much as one time the size of the original database is required | ||
| 306 | + in free disk space. However, be aware that this factor changes for each :mod:`dbm` submodule. | ||
| 307 | + | ||
| 287 | 308 | .. method:: gdbm.sync() | |
| 288 | 309 | ||
| 289 | 310 | When the database has been opened in fast mode, this method forces any | |
@@ -438,6 +459,11 @@ The :mod:`!dbm.dumb` module defines the following: | |||
| 438 | 459 | with a sufficiently large/complex entry due to stack depth limitations in | |
| 439 | 460 | Python's AST compiler. | |
| 440 | 461 | ||
| 462 | + .. warning:: | ||
| 463 | + :mod:`dbm.dumb` does not support concurrent read/write access. (Multiple | ||
| 464 | + simultaneous read accesses are safe.) When a program has the database open | ||
| 465 | + for writing, no other program should have it open for reading or writing. | ||
| 466 | + | ||
| 441 | 467 | .. versionchanged:: 3.5 | |
| 442 | 468 | :func:`~dbm.dumb.open` always creates a new database when *flag* is ``'n'``. | |
| 443 | 469 | ||
@@ -460,3 +486,15 @@ The :mod:`!dbm.dumb` module defines the following: | |||
| 460 | 486 | .. method:: dumbdbm.close() | |
| 461 | 487 | ||
| 462 | 488 | Close the database. | |
| 489 | + | ||
| 490 | + .. method:: dumbdbm.reorganize() | ||
| 491 | + | ||
| 492 | + If you have carried out a lot of deletions and would like to shrink the space | ||
| 493 | + used on disk, this method will reorganize the database; otherwise, deleted file | ||
| 494 | + space will not be reused. | ||
| 495 | + | ||
| 496 | + .. note:: | ||
| 497 | + While reorganizing, no additional free disk space is required. However, be aware | ||
| 498 | + that this factor changes for each :mod:`dbm` submodule. | ||
| 499 | + | ||
| 500 | + .. versionadded:: next | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -75,8 +75,15 @@ Two additional methods are supported: | |||
| 75 | 75 | ||
| 76 | 76 | Write back all entries in the cache if the shelf was opened with *writeback* | |
| 77 | 77 | set to :const:`True`. Also empty the cache and synchronize the persistent | |
| 78 | - dictionary on disk, if feasible. This is called automatically when the shelf | ||
| 79 | - is closed with :meth:`close`. | ||
| 78 | + dictionary on disk, if feasible. This is called automatically when | ||
| 79 | + :meth:`reorganize` is called or the shelf is closed with :meth:`close`. | ||
| 80 | + | ||
| 81 | + .. method:: Shelf.reorganize() | ||
| 82 | + | ||
| 83 | + Calls :meth:`sync` and attempts to shrink space used on disk by removing empty | ||
| 84 | + space resulting from deletions. | ||
| 85 | + | ||
| 86 | + .. versionadded:: next | ||
| 80 | 87 | ||
| 81 | 88 | .. method:: Shelf.close() | |
| 82 | 89 | ||
@@ -116,6 +123,11 @@ Restrictions | |||
| 116 | 123 | * On macOS :mod:`dbm.ndbm` can silently corrupt the database file on updates, | |
| 117 | 124 | which can cause hard crashes when trying to read from the database. | |
| 118 | 125 | ||
| 126 | + * :meth:`Shelf.reorganize` may not be available for all database packages and | ||
| 127 | + may temporarely increase resource usage (especially disk space) when called. | ||
| 128 | + Additionally, it will never run automatically and instead needs to be called | ||
| 129 | + explicitly. | ||
| 130 | + | ||
| 119 | 131 | ||
| 120 | 132 | .. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8') | |
| 121 | 133 | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -89,13 +89,30 @@ New modules | |||
| 89 | 89 | Improved modules | |
| 90 | 90 | ================ | |
| 91 | 91 | ||
| 92 | + dbm | ||
| 93 | + --- | ||
| 94 | + | ||
| 95 | + * Added new :meth:`!reorganize` methods to :mod:`dbm.dumb` and :mod:`dbm.sqlite3` | ||
| 96 | + which allow to recover unused free space previously occupied by deleted entries. | ||
| 97 | + (Contributed by Andrea Oliveri in :gh:`134004`.) | ||
| 98 | + | ||
| 99 | + | ||
| 92 | 100 | difflib | |
| 93 | 101 | ------- | |
| 94 | 102 | ||
| 95 | 103 | * Improved the styling of HTML diff pages generated by the :class:`difflib.HtmlDiff` | |
| 96 | 104 | class, and migrated the output to the HTML5 standard. | |
| 97 | 105 | (Contributed by Jiahao Li in :gh:`134580`.) | |
| 98 | 106 | ||
| 107 | + | ||
| 108 | + shelve | ||
| 109 | + ------ | ||
| 110 | + | ||
| 111 | + * Added new :meth:`!reorganize` method to :mod:`shelve` used to recover unused free | ||
| 112 | + space previously occupied by deleted entries. | ||
| 113 | + (Contributed by Andrea Oliveri in :gh:`134004`.) | ||
| 114 | + | ||
| 115 | + | ||
| 99 | 116 | ssl | |
| 100 | 117 | --- | |
| 101 | 118 | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -9,16 +9,14 @@ | |||
| 9 | 9 | - seems to contain a bug when updating... | |
| 10 | 10 | ||
| 11 | 11 | - reclaim free space (currently, space once occupied by deleted or expanded | |
| 12 | - items is never reused) | ||
| 12 | + items is not reused exept if .reorganize() is called) | ||
| 13 | 13 | ||
| 14 | 14 | - support concurrent access (currently, if two processes take turns making | |
| 15 | 15 | updates, they can mess up the index) | |
| 16 | 16 | ||
| 17 | 17 | - support efficient access to large databases (currently, the whole index | |
| 18 | 18 | is read when the database is opened, and some updates rewrite the whole index) | |
| 19 | 19 | ||
| 20 | - - support opening for read-only (flag = 'm') | ||
| 21 | - | ||
| 22 | 20 | """ | |
| 23 | 21 | ||
| 24 | 22 | import ast as _ast | |
@@ -289,6 +287,34 @@ def __enter__(self): | |||
| 289 | 287 | def __exit__(self, *args): | |
| 290 | 288 | self.close() | |
| 291 | 289 | ||
| 290 | + def reorganize(self): | ||
| 291 | + if self._readonly: | ||
| 292 | + raise error('The database is opened for reading only') | ||
| 293 | + self._verify_open() | ||
| 294 | + # Ensure all changes are committed before reorganizing. | ||
| 295 | + self._commit() | ||
| 296 | + # Open file in r+ to allow changing in-place. | ||
| 297 | + with _io.open(self._datfile, 'rb+') as f: | ||
| 298 | + reorganize_pos = 0 | ||
| 299 | + | ||
| 300 | + # Iterate over existing keys, sorted by starting byte. | ||
| 301 | + for key in sorted(self._index, key = lambda k: self._index[k][0]): | ||
| 302 | + pos, siz = self._index[key] | ||
| 303 | + f.seek(pos) | ||
| 304 | + val = f.read(siz) | ||
| 305 | + | ||
| 306 | + f.seek(reorganize_pos) | ||
| 307 | + f.write(val) | ||
| 308 | + self._index[key] = (reorganize_pos, siz) | ||
| 309 | + | ||
| 310 | + blocks_occupied = (siz + _BLOCKSIZE - 1) // _BLOCKSIZE | ||
| 311 | + reorganize_pos += blocks_occupied * _BLOCKSIZE | ||
| 312 | + | ||
| 313 | + f.truncate(reorganize_pos) | ||
| 314 | + # Commit changes to index, which were not in-place. | ||
| 315 | + self._commit() | ||
| 316 | + | ||
| 317 | + | ||
| 292 | 318 | ||
| 293 | 319 | def open(file, flag='c', mode=0o666): | |
| 294 | 320 | """Open the database file, filename, and return corresponding object. | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -15,6 +15,7 @@ | |||
| 15 | 15 | STORE_KV = "REPLACE INTO Dict (key, value) VALUES (CAST(? AS BLOB), CAST(? AS BLOB))" | |
| 16 | 16 | DELETE_KEY = "DELETE FROM Dict WHERE key = CAST(? AS BLOB)" | |
| 17 | 17 | ITER_KEYS = "SELECT key FROM Dict" | |
| 18 | + REORGANIZE = "VACUUM" | ||
| 18 | 19 | ||
| 19 | 20 | ||
| 20 | 21 | class error(OSError): | |
@@ -122,6 +123,9 @@ def __enter__(self): | |||
| 122 | 123 | def __exit__(self, *args): | |
| 123 | 124 | self.close() | |
| 124 | 125 | ||
| 126 | + def reorganize(self): | ||
| 127 | + self._execute(REORGANIZE) | ||
| 128 | + | ||
| 125 | 129 | ||
| 126 | 130 | def open(filename, /, flag="r", mode=0o666): | |
| 127 | 131 | """Open a dbm.sqlite3 database and return the dbm object. | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -171,6 +171,11 @@ def sync(self): | |||
| 171 | 171 | if hasattr(self.dict, 'sync'): | |
| 172 | 172 | self.dict.sync() | |
| 173 | 173 | ||
| 174 | + def reorganize(self): | ||
| 175 | + self.sync() | ||
| 176 | + if hasattr(self.dict, 'reorganize'): | ||
| 177 | + self.dict.reorganize() | ||
| 178 | + | ||
| 174 | 179 | ||
| 175 | 180 | class BsdDbShelf(Shelf): | |
| 176 | 181 | """Shelf implementation using the "BSD" db interface. | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -135,6 +135,67 @@ def test_anydbm_access(self): | |||
| 135 | 135 | assert(f[key] == b"Python:") | |
| 136 | 136 | f.close() | |
| 137 | 137 | ||
| 138 | + def test_anydbm_readonly_reorganize(self): | ||
| 139 | + self.init_db() | ||
| 140 | + with dbm.open(_fname, 'r') as d: | ||
| 141 | + # Early stopping. | ||
| 142 | + if not hasattr(d, 'reorganize'): | ||
| 143 | + self.skipTest("method reorganize not available this dbm submodule") | ||
| 144 | + | ||
| 145 | + self.assertRaises(dbm.error, lambda: d.reorganize()) | ||
| 146 | + | ||
| 147 | + def test_anydbm_reorganize_not_changed_content(self): | ||
| 148 | + self.init_db() | ||
| 149 | + with dbm.open(_fname, 'c') as d: | ||
| 150 | + # Early stopping. | ||
| 151 | + if not hasattr(d, 'reorganize'): | ||
| 152 | + self.skipTest("method reorganize not available this dbm submodule") | ||
| 153 | + | ||
| 154 | + keys_before = sorted(d.keys()) | ||
| 155 | + values_before = [d[k] for k in keys_before] | ||
| 156 | + d.reorganize() | ||
| 157 | + keys_after = sorted(d.keys()) | ||
| 158 | + values_after = [d[k] for k in keys_before] | ||
| 159 | + self.assertEqual(keys_before, keys_after) | ||
| 160 | + self.assertEqual(values_before, values_after) | ||
| 161 | + | ||
| 162 | + def test_anydbm_reorganize_decreased_size(self): | ||
| 163 | + | ||
| 164 | + def _calculate_db_size(db_path): | ||
| 165 | + if os.path.isfile(db_path): | ||
| 166 | + return os.path.getsize(db_path) | ||
| 167 | + total_size = 0 | ||
| 168 | + for root, _, filenames in os.walk(db_path): | ||
| 169 | + for filename in filenames: | ||
| 170 | + file_path = os.path.join(root, filename) | ||
| 171 | + total_size += os.path.getsize(file_path) | ||
| 172 | + return total_size | ||
| 173 | + | ||
| 174 | + # This test requires relatively large databases to reliably show difference in size before and after reorganizing. | ||
| 175 | + with dbm.open(_fname, 'n') as f: | ||
| 176 | + # Early stopping. | ||
| 177 | + if not hasattr(f, 'reorganize'): | ||
| 178 | + self.skipTest("method reorganize not available this dbm submodule") | ||
| 179 | + | ||
| 180 | + for k in self._dict: | ||
| 181 | + f[k.encode('ascii')] = self._dict[k] * 100000 | ||
| 182 | + db_keys = list(f.keys()) | ||
| 183 | + | ||
| 184 | + # Make sure to calculate size of database only after file is closed to ensure file content are flushed to disk. | ||
| 185 | + size_before = _calculate_db_size(os.path.dirname(_fname)) | ||
| 186 | + | ||
| 187 | + # Delete some elements from the start of the database. | ||
| 188 | + keys_to_delete = db_keys[:len(db_keys) // 2] | ||
| 189 | + with dbm.open(_fname, 'c') as f: | ||
| 190 | + for k in keys_to_delete: | ||
| 191 | + del f[k] | ||
| 192 | + f.reorganize() | ||
| 193 | + | ||
| 194 | + # Make sure to calculate size of database only after file is closed to ensure file content are flushed to disk. | ||
| 195 | + size_after = _calculate_db_size(os.path.dirname(_fname)) | ||
| 196 | + | ||
| 197 | + self.assertLess(size_after, size_before) | ||
| 198 | + | ||
| 138 | 199 | def test_open_with_bytes(self): | |
| 139 | 200 | dbm.open(os.fsencode(_fname), "c").close() | |
| 140 | 201 | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -1365,6 +1365,7 @@ Milan Oberkirch | |||
| 1365 | 1365 | Pascal Oberndoerfer | |
| 1366 | 1366 | Géry Ogam | |
| 1367 | 1367 | Seonkyo Ok | |
| 1368 | + Andrea Oliveri | ||
| 1368 | 1369 | Jeffrey Ollie | |
| 1369 | 1370 | Adam Olsen | |
| 1370 | 1371 | Bryan Olson | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,2 @@ | |||
| 1 | + :mod:`shelve` as well as underlying :mod:`!dbm.dumb` and :mod:`!dbm.sqlite` now have :meth:`!reorganize` methods to | ||
| 2 | + recover unused free space previously occupied by deleted entries. | ||
| Back | FazBrowse Home | New Git URL |
0 commit comments