Repository navigation
C extension naming doesn't take bitness into account #67169
Description
| BPO | 22980 |
|---|---|
| Nosy | @malemburg, @warsaw, @doko42, @ncoghlan, @pitrou, @larryhastings, @tjguk, @ned-deily, @ericsnowcurrently, @berkerpeksag, @zware, @1st1, @zooba, @dstufft |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2015-09-12.16:13:19.053>
created_at = <Date 2014-12-02.14:35:25.510>
labels = ['deferred-blocker', 'type-feature']
title = "C extension naming doesn't take bitness into account"
updated_at = <Date 2017-01-14.07:25:16.112>
user = 'https://github.com/pitrou'bugs.python.org fields:
activity = <Date 2017-01-14.07:25:16.112>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2015-09-12.16:13:19.053>
closer = 'larry'
components = []
creation = <Date 2014-12-02.14:35:25.510>
creator = 'pitrou'
dependencies = []
files = ['37379', '37384', '38155', '38964', '39027', '40420']
hgrepos = []
issue_num = 22980
keywords = ['patch']
message_count = 96.0
messages = ['232000', '232001', '232002', '232003', '232008', '232014', '232016', '232026', '232030', '232031', '232033', '232034', '232036', '232037', '232038', '232039', '232040', '232041', '232042', '232043', '232044', '232045', '232047', '232052', '232063', '232065', '232251', '232256', '232266', '232268', '232270', '232272', '232277', '232297', '232710', '232711', '232729', '232740', '232759', '232760', '232764', '232765', '232807', '233822', '236109', '236111', '237558', '237559', '240516', '240548', '240746', '240748', '240750', '240761', '240762', '240769', '240775', '241065', '241127', '241139', '241141', '241159', '241167', '241183', '241211', '241223', '241228', '241230', '241231', '241236', '241242', '241248', '241249', '241250', '241251', '241252', '241253', '241254', '241283', '241285', '241287', '241291', '241512', '241513', '246247', '249725', '250045', '250299', '250302', '250308', '250341', '250424', '250425', '250431', '250524', '285464']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'doko', 'ncoghlan', 'pitrou', 'larry', 'tim.golden', 'ned.deily', 'Arfrever', 'python-dev', 'eric.snow', 'berker.peksag', 'zach.ware', 'yselivanov', 'steve.dower', 'dstufft']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22980'
versions = ['Python 3.5']
Activity
pitrou commented on Dec 2, 2014
Currently, C extensions are named something like "_helperlib.cpython-34dm.so". This doesn't take into account the bitness of the interpreter (32- vs. 64-bit), which makes it awkward to use the same working copy with two different interpreters (you have to rebuild everything each time you switch bitnesses).
Worse, under Windows it seems ABI tags aren't even used, giving generic names such as "_helperlib.pyd". Why is that?
vstinner commented on Dec 2, 2014
See also the PEP-3149.
pitrou commented on Dec 2, 2014
PEP-3149 says """It is not currently clear that the facilities in this PEP are even useful for Windows""". Well, it seems I have found a use for it :-)
pitrou commented on Dec 2, 2014
Ideally, we would use distutils.util.get_platform(). However, there are two cases where it relies on other modules:
Of course, ideally we should be able to hardcode this into the compiled CPython executable...
pitrou commented on Dec 2, 2014
As a side-note, it is interesting to note that Python currently wrongly identifies 32-bit builds under 64-bit Linux:
Python 3.5.0a0 (default:64a54f0c87d7, Nov 2 2014, 17:18:13) [GCC 4.9.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys, os, sysconfig >>> sys.maxsize 2147483647 >>> os.uname() posix.uname_result(sysname='Linux', nodename='fsol', release='3.16.0-25-generic', version='#33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 2014', machine='x86_64') >>> sysconfig.get_platform() 'linux-x86_64'AFAIU, sysconfig.get_platform() (or its sibling distutils.util.get_platform()) is used for the naming of binary distributions...
pitrou commented on Dec 2, 2014
The MULTIARCH variable can help at least under Linux:
vstinner commented on Dec 2, 2014
There is also platform.architecture(). I don't like its implementation, it relies on the external file program :-(
zooba commented on Dec 2, 2014
I'm very much in favor of adding this for .pyds on Windows.
I assume the hard part will be getting the details for Linux (doesn't bitness have to be compiled in there? For Windows it can be determined at compile-time...), but preferring an extension with the ABI tag and falling back on one without seems easy enough.
(Would/could this also work for .py files? So a 2.7/3.x or Jython/CPython/IronPython package could include tags in pure-Python code files?)
malemburg commented on Dec 2, 2014
Note that there's a difference between the platform's architecture (which is what get_platform() returns) and the pointer size of the currently running Python executable.
On 64-bit Linux, it's rather rare to have an application built as 32-bit executable. On 64-bit Windows, it's rather common to have 32-bit applications running.
The best way to determine 32-bit vs. 64-bit is by using the struct module:
This should be portable across all platforms and will always refer to the pointer size of the currently running Python executable.
pitrou commented on Dec 2, 2014
Yes, that's pointed out above.
pitrou commented on Dec 2, 2014
Nothing new should be necessary for pyc files under Windows:
Python 3.4.2 |Continuum Analytics, Inc.| (default, Oct 22 2014, 11:51:45) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.implementation.cache_tag 'cpython-34'The problem is with C extensions:
>>> import _imp >>> _imp.extension_suffixes() ['.pyd']Compare with Linux:
>>> import _imp >>> _imp.extension_suffixes() ['.cpython-35dm.so', '.abi3.so', '.so']pitrou commented on Dec 2, 2014
Sticking to bitness should be easy (although I wonder if it would be desirable for platforms with fat binaries - Ned?). If we can go the extra mile and include platform identification all the better, of course.
zooba commented on Dec 2, 2014
I was more interested in source file resolution than bytecode caching. If Python 3.5 would prefer "spam.cpython-35.py" or "spam.cpython-3.py" over "spam.py" and Python 2 preferred "spam.py", then I can more easily separate the code that won't parse in the alternative.
Happy to be told it's unrelated and I should raise it separately, but from my POV resolving .pyd filenames looks very similar to resolving .py files.
malemburg commented on Dec 2, 2014
On 02.12.2014 19:02, Antoine Pitrou wrote:
I hear the "can of worms" alarm ringing :-)
Seriously, I think that putting platform infos into the file name
is bound to cause more trouble than it tries to solve. Fat builds
leave the decision to the linker, which is a good method and avoids
the file name clashes.
I think we should only focus on platforms where fat builds are
uncommon, while at the same time you do have to support multiple
architectures, like e.g. Windows:
http://en.wikipedia.org/wiki/Fat_binary
Note that on Linux, 32-bit and 64-bit versions are typically placed
into different directory trees:
http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
so I'm not sure whether it's a real problem on Linux.
72 remaining items
ned-deily commented on Jul 4, 2015
I think we should add something to the 3.5 "What's New" document about these changes and which platforms are affected. Otherwise is there anything left to do before closing?
larryhastings commented on Sep 4, 2015
I sure hope not.
larryhastings commented on Sep 7, 2015
I'm leaving this open just because we're apparently waiting on some "What's New" docs.
larryhastings commented on Sep 9, 2015
Here's an attempt at a What's New section for this change. I expect it's wrong! Maybe someone can fix it. Maybe it's actually better than not having one at all.
Can we maybe get a round or two of edits on this and get something in for 3.5 final?
berkerpeksag commented on Sep 9, 2015
Adding Yury since he and Elvis are working on Doc/whatsnew/3.5.rst and they might want to take a look at the latest patch.
zooba commented on Sep 9, 2015
Only thing I'd add is that the extra tag is optional (on Windows at least), and Python will happily import extensions without it. But extensions with a mismatched tag won't be loaded.
doko42 commented on Sep 9, 2015
thanks for the draft!
I'm not sure how to describe this properly. The extension names are derived from https://wiki.debian.org/Multiarch/Tuples
and this again is derived from the GNU triplets/quadruplets.
there is no "cpu" and "os" part, depending on the architecture some ABI parts are either encoded in the "cpu" part or the "os" part.
So what about just enumerating the most common cases (i386-linux-gnu, x86_64-linux-gnu, arm-linux-gnueabi (still used for the old Raspberry Pi), arm-linux-gnueabihf), and then point to the "spec"? The above examples have some irregular cases, most other cases just follow the triplets.
I wouldn't mention x86_64-linux-gnux32 explicitly. Until now there are only unreleased or experimental distros.
Not sure if it is worth mentioning that this would allow distributing "fat" wheels.
python-dev commented on Sep 10, 2015
New changeset 1744d65705d0 by Yury Selivanov in branch '3.5':
whatsnew/3.5: Describe changes in issue bpo-22980
https://hg.python.org/cpython/rev/1744d65705d0
New changeset cfbcb3a6a848 by Yury Selivanov in branch 'default':
Merge 3.5 (issue bpo-22980, whatsnew/3.5)
https://hg.python.org/cpython/rev/cfbcb3a6a848
1st1 commented on Sep 10, 2015
Larry, Matthias, Steve, Berker - I've mentioned this issue in the whatsnew (applied Larry's patch with some modifications to address comments from Steve and Matthias). Please review.
zooba commented on Sep 11, 2015
There's no dot before the debug marker on Windows. Otherwise, looks good to me. Thanks for writing this up.
larryhastings commented on Sep 12, 2015
It's fixed! So it's finally closed.
python-dev commented on Jan 14, 2017
New changeset 80fc40a9ae47 by Martin Panter in branch '3.5':
Issue bpo-22980: Skip a sysconfig test if _ctypes is not available.
https://hg.python.org/cpython/rev/80fc40a9ae47