FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-78502: Add a trackfd parameter to mmap.mmap() by ZackerySpytz · Pull Request #25425 · python/cpython · GitHub

/ cpython Public

gh-78502: Add a trackfd parameter to mmap.mmap() - #25425

Merged
encukou merged 16 commits into
python:mainfrom
ZackerySpytz:bpo-34321-mmap-trackfd-parameter
Jan 16, 2024
Merged

gh-78502: Add a trackfd parameter to mmap.mmap()#25425
encukou merged 16 commits into
python:mainfrom
ZackerySpytz:bpo-34321-mmap-trackfd-parameter

Conversation

ZackerySpytz commented Apr 15, 2021
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Contributor

If trackfd is False, the file descriptor specified by fileno
will not be duplicated.

https://bugs.python.org/issue34321

If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.
ZackerySpytz force-pushed the bpo-34321-mmap-trackfd-parameter branch from f46e71e to d42762c Compare April 15, 2021 17:06

Copy link
Copy Markdown
Contributor

Could you make the new trackfd argument keyword-only? When the number of arguments starts getting large, especially when it's an argument that's boolean and not self-describing, it seems like there is a preference for making the new arguments keyword-only both to prevent using the API in needlessly unclear ways and to avoid a long term compatibility requirement that prevents changing the argument order (see evolution of subprocess.Popen and subprocess.run). I suspect, in hindsight, mmap would have liked to keep more arguments keyword-only to avoid the issue with differing argument orders on UNIX-likes and Windows, but that ship has sailed; at least we can avoid making it any worse.

Also, is there a reason you're not doing this for Windows? The same feature seems perfectly reasonable in Windows too.

github-actions Bot commented Jun 3, 2021

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

github-actions Bot added the stale Stale PR or inactive for long period of time. label Jun 3, 2021
github-actions Bot removed the stale Stale PR or inactive for long period of time. label Aug 6, 2022
erlend-aasland changed the title bpo-34321: Add a trackfd parameter to mmap.mmap() gh-78502: Add a trackfd parameter to mmap.mmap() Jan 5, 2024

Copy link
Copy Markdown
Contributor

cc. @encukou / @serhiy-storchaka: would one of you like to take a look at the proposed solution?

encukou self-assigned this Jan 5, 2024

encukou commented Jan 8, 2024

Copy link
Copy Markdown
Member

This is a good start! @ZackerySpytz, do you want to continue or should I finish the feature?

IMO, it's fine to keep it *nix-only, at least initially. It should be documented as such though.
Also, size/resize should be tested to ensure they raise the proper exception rather than crash, and their docs should mention the new error condition. (
I'd also prefer to test other functionality, like __setitem__, flush. If/when Windows support is added, the test should detect any platform differences.

Copy link
Copy Markdown
Contributor

This is a good start! @ZackerySpytz, do you want to continue or should I finish the feature?

Zackery is not very active these days; I think it is fine to take over the PR.

encukou commented Jan 9, 2024

Copy link
Copy Markdown
Member

OK! Here are my updates.

I've realized that resize() method works fine without a fd -- it'll resize the map, but not the underlying file, so if the map grows, the “new” part becomes unmapped.
The latest commit here fixes that -- at the cost of an additional flag. (The memory cost in negligible IMO -- and could be optimized away -- but there's a maintenance cost & mental overhead.)
I'm not sure if resize() with trackfd=False would ever be useful. Making it fail is a safer initial implementation; it can always be enabled.

encukou marked this pull request as ready for review January 9, 2024 12:47

serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think that it would be interesting to add the following tests:

  • Test that it fails on Windows with trackfd=False and trackfd=True.
  • Test with fd=-1.
  • Test with original fd closed after creating a mmap.
  • Test whether mmap can be used after fork.

Comment thread Doc/library/mmap.rst Outdated
Comment thread Doc/library/mmap.rst Outdated
Comment thread Lib/test/test_mmap.py Outdated
Comment thread Modules/mmapmodule.c Outdated

encukou commented Jan 9, 2024

Copy link
Copy Markdown
Member

Test whether mmap can be used after fork.

I think that's too much: it'd be testing the platform, rather than CPython itself.

Comment thread Doc/whatsnew/3.10.rst Outdated
Comment thread Lib/test/test_mmap.py
Comment on lines +309 to +312
with self.assertRaises(OSError):
m.size()
with self.assertRaises(TypeError):
m.resize(size // 2)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Does it work the same for mmap(-1, size, trackfd=True)? I have not found tests for this.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

trackfd=True is the default. It's tested in test_resize_past_pos.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

It has different behavior. Should trackfd=False have any effect for fd=-1? Should combination fd=-1 and trackfd=False be allowed?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Conceptually, trackfd=False should be the only possible behaviour for fd=-1 -- the fd is not tracked.
However, I don't think it's worth it to include a third value for “trackfd=default”. As it is, the default (True) works as before. Setting False has the documented effect -- disabling resize. This isn't very useful, but I don't think it's worth handling specially.

Comment thread Modules/mmapmodule.c Outdated

serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

By default, the mmap is always created from non-inheritable file descriptor. But if trackfd is false, it can be created from inheritable file descriptor. I wondered what is the difference and how can it be tested.

But if it works after closing the original file descriptor, perhaps there is no difference. cc @vstinner

Comment thread Doc/library/mmap.rst Outdated
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

encukou commented Jan 11, 2024

Copy link
Copy Markdown
Member

There's always another test that could be added. I think we're too far into diminishing returns at this point :)

Copy link
Copy Markdown
Member

There is currently a discussion about inheritance of file descriptors in #113817. I just want to be sure that we do not miss some details here (most likely it is all correct). cc @vstinner, @zooba, @eryksun.

zooba left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Our other discussion was specific to Windows, so this issue doesn't apply.

But since I got mentioned, could we get the docs phrased in a way that explains why I might ever want to use the new option? Right now it sounds like a bad thing.

Comment thread Doc/library/mmap.rst
defaults to 0. *offset* must be a multiple of :const:`ALLOCATIONGRANULARITY`
which is equal to :const:`PAGESIZE` on Unix systems.

If *trackfd* is ``False``, the file descriptor specified by *fileno* will

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'd love to have some idea of why I might want to use this parameter. Right now it only describes the downsides.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

On Windows, the internally duplicated handle probably references an open that lacks delete access. It thus prevents deleting the file, even if the mapped section otherwise allows it (e.g. the section is mapped readonly). For example:

>>> f = open('spam.txt')
>>> m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
>>> f.close()
>>> os.remove('spam.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'spam.txt'

>>> # I manually closed the internal handle via Process Explorer.
>>> os.remove('spam.txt')
>>> m[:]
b'spam'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

That sounds like a reason to at least add the argument for all platforms, which I'm generally in favour of anyway. It can have more appropriate semantics on Windows if needed (i.e. "doesn't hold an extra HANDLE" rather than "FD").

It's probably actually pretty useful to be able to immediately delete the file but keep the mapping open (which will keep the file on disk on Windows at least, so you can't reuse the name while it's in use). And it looks like the mapping doesn't lock out deletes, so I guess it'll work as intended.

I'm not going to hold up this PR for it though. All I'll say is that if we ever do add that option, it should be trackfd=False to "activate" it, for consistency between platforms.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

That sounds like a reason to at least add the argument for all platforms, which I'm generally in favour of anyway. It can have more appropriate semantics on Windows if needed (i.e. "doesn't hold an extra HANDLE" rather than "FD").

I think trackfd would be fine on Windows. The fileno parameter is a C file descriptor, not a native OS handle.

It's probably actually pretty useful to be able to immediately delete the file but keep the mapping open (which will keep the file on disk on Windows at least, so you can't reuse the name while it's in use). And it looks like the mapping doesn't lock out deletes, so I guess it'll work as intended.

NTFS supports POSIX delete, in which a deleted file gets renamed to a reserved system directory until all references to the file object have been closed. That includes the internal pointer reference to a file object that's held by the memory manager for the mapped section. The internal file reference doesn't count toward the file's share mode, i.e. a memory-mapped file can be deleted even if the source open didn't share delete access. Actually, I just checked that the delete is allowed nowadays even if the mapped section has write access to the file, so my assumption was wrong that it would only work for a readonly mapping.

You can observe this in Process Explorer. Switch the lower-pane view to DLLs (file- and pagefile-backed memory mappings), and add the name and path columns to the view. You'll see that the backing file gets moved to the "\$Extend\$Deleted" system directory on the volume after the file is 'deleted'.

encukou commented Jan 15, 2024

Copy link
Copy Markdown
Member

I'd prefer if the Windows functionality was added in a separate PR. I don't have a Windows box set up (yet), so I won't send one myself.
Other than that, if the added note is OK this looks good to merge.

serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM.

encukou merged commit 8fd287b into python:main Jan 16, 2024
kulikjak pushed a commit to kulikjak/cpython that referenced this pull request Jan 22, 2024
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants


Back | FazBrowse Home | New Git URL