Crash when using pythonnet with nuitka · Issue #904 · pythonnet/pythonnet · GitHub

Environment

Pythonnet version: 2.4.0
Python version: Python 2.7.13
Operating System: Windows 10

Details

Describe what you were trying to get done.

I want to use a .net library that I access through pythonnet in combination with a library that was compiled with Nuitka (the contents of the library do not seem to matter, nor the version of Nuitka). However, when I import the module in the pyd first, calling 'import clr' after that crashes pythonnet and then Python.

I was wondering if this is a known issue/how easy it would be to fix it/if there would be some straightforward workaround so the order of imports would not matter?

What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.

    import foo
    import clr

Will crash pythonnet and then Python. Changing the order of the imports will make it work, however it will make working with the .net library very annoying.

Relevant files are in the zip:
minimal_example.zip

If there was a crash, please include the traceback here.

    Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Python.Runtime.Runtime.PyModule_GetDict(IntPtr module)
   at Python.Runtime.PythonEngine.Initialize(IEnumerable`1 args, Boolean setSysArgv, Boolean initSigs)
   at Python.Runtime.PythonEngine.InitExt()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at clrModule.initclr()

I create a branch and tried to fix it, it seems worked for my PC but failed on CI, I still not figure it out.
amos402@3591d92
Here's the binaries, could you tell me what will happen for your environment?
Python.Runtime.zip

Hi, thanks for the reply :)

I changed the dll in my site-packages with the one you sent. Now

import foo
import clr

works.

However, doing:

import clr

by itself:

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Python.Runtime.ModuleObject.StoreAttribute(String name, ManagedType ob) in I:\repos\pythonnet\src\runtime\moduleobject.cs:line 178
   at Python.Runtime.ModuleObject.InitializeModuleMembers() in I:\repos\pythonnet\src\runtime\moduleobject.cs:line 233
   at Python.Runtime.ModuleObject..ctor(String name) in I:\repos\pythonnet\src\runtime\moduleobject.cs:line 58
   at Python.Runtime.CLRModule..ctor() in I:\repos\pythonnet\src\runtime\moduleobject.cs:line 319
   at Python.Runtime.ImportHook.Initialize() in I:\repos\pythonnet\src\runtime\importhook.cs:line 37
   at Python.Runtime.Runtime.Initialize(Boolean initSigs) in I:\repos\pythonnet\src\runtime\runtime.cs:line 315
   at Python.Runtime.PythonEngine.Initialize(IEnumerable`1 args, Boolean setSysArgv, Boolean initSigs) in I:\repos\pythonnet\src\runtime\pythonengine.cs:line 168
   at Python.Runtime.PythonEngine.Initialize(Boolean setSysArgv, Boolean initSigs) in I:\repos\pythonnet\src\runtime\pythonengine.cs:line 145
   at Python.Runtime.PythonEngine.InitExt() in I:\repos\pythonnet\src\runtime\pythonengine.cs:line 261
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at clrModule.initclr()```

Hi, any update on this? If it does not seem to be realistic to fix this one in the near future, could you explain me the problem, and why you did these fixes in the commit? No promises here, but if it is a relatively easy fix, I could maybe pick it up myself. I'm asking because I need to make a decision if I'm continuing using pythonnet to access this library or if I'm going for a different approach.

The major cause is ModuleObject is not a Python module object, so it cannot get the dict by PyModule_GetDict, some initialize behavior just different between import clr.pyd and start with nPython.exe directly.
You can just start with nPython.exe, it may works. I will try to fix it again. But since I can't reproduce your errors, I'm just not sure about that......

It is strange that you cannot recreate the error (what platform do you use?). When I build the project in vs2019 as DebugWin x64, use a plain Python 2.17.12 interpreter, and just run 'break_it.py' in a folder with a freshly built clrmodule and runtime it breaks every time for me.

Anyway, I've started to look into it a bit. I hope I'm not just repeating stuff you already found out.

(Note that I'm using the code from master, yours seems to keep crashing for me when I build it in debug mode in vs2019. I have the feeling that after your modification stuff is written to illegal parts of the memory, but I do not yet understand why.)

The main difference between the case that does work (import clr and then foo), and that does not work (import foo and then clr), is that the following 'hack' does not have an effect in the faulty case:

            // This hackery is required in order to allow a plain Python to
            // import the managed runtime via the CLR bootstrapper module.
            // The standard Python machinery in control at the time of the
            // import requires the module to pass PyModule_Check. :(
            if (!hacked)
            {
                IntPtr type = tpHandle;
                IntPtr mro = Marshal.ReadIntPtr(type, TypeOffset.tp_mro);
                IntPtr ext = Runtime.ExtendTuple(mro, Runtime.PyModuleType);
                Marshal.WriteIntPtr(type, TypeOffset.tp_mro, ext);
                Runtime.XDecref(mro);
                hacked = true;
            }

when the hack works, I suddenly can get the dict through PyModule_GetDict, if I understand it correctly because you change the MRO to include a PyModuleType, thus making it a PyModule. In the case that it does not work, this hack fails, and it will probably, like you said, not be recognized as a PyModule.

Extending the MRO tuple however seems to succeed, the Python runtime does not throw any exceptions (Runtime.CheckExceptionOccurred() does not report anything).

The logic seems kind of robust to me (no reason why Runtime.PyModuleType would not be added). For my understanding: why are we adding the PyModuleType to mro instead of bases? Otherwise it might be that after the import of foo, Runtime.PyModuleType does not point to a ModuleType anymore (ideas on how I can check that?).

Any thoughts on this? Or do you think I am looking in a wrong direction?

Update:

It seems that the problem is that Nuitka changes the type of the 'builtin' module, and the Runtime code deduces the PyModuleType based on the type of the __builtin__ module, probably causing everything to fail. (notice that foo is the compiled module).

import __builtin__

print type(__builtin__), id(__builtin__)
print __builtin__
import foo

print type(__builtin__), id(__builtin__)
print __builtin__

Results in:

<type 'module'> 43593992  <--- notice 'module'
<module '__builtin__' (built-in)>
<type 'compiled_module'> 43593992  <--- notice 'compiled_module'
<module '__builtin__' (built-in)>

Solution seems to be in having a safer way to deduce Runtime.PyModuleType

Fix so that Runtime.PyModuleType is retrieved via Python.dll (pythonn…

Fix so that Runtime.PyModuleType is retrieved via Python.dll (#904) (#…

Fix so that Runtime.PyModuleType is retrieved via Python.dll (pythonn…

Crash when using pythonnet with nuitka · Issue #904 · pythonnet/pythonnet · GitHub

Repository navigation

Crash when using pythonnet with nuitka #904

Environment

Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Crash when using pythonnet with nuitka · Issue #904 · pythonnet/pythonnet · GitHub

Repository navigation

Crash when using pythonnet with nuitka #904

Description

Environment

Details

Activity

amos402 commented on Jul 2, 2019

ftreurni commented on Jul 2, 2019

ftreurni commented on Jul 15, 2019

amos402 commented on Jul 15, 2019

ftreurni commented on Aug 1, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions