WIP: Combine #546 and #531 by fractus · Pull Request #557 · pythonnet/pythonnet · GitHub

What does this implement/fix? Explain your changes.

Supersedes two PRs by merging Build system improvements pack - addressing comments to PR and some issues. #546 and Interop methods with Py_ssize_t works differently in NetCoreApp 2.0 #531 as suggested by @dmitriyse in the discussion of Interop methods with Py_ssize_t works differently in NetCoreApp 2.0 #531.
Fixed a typo in runtime.cs
Commented a case-sensitive copy that break the build on macOS in Python.Runtime.15.csproj

Does this close any currently open issues?

No, it just now builds without problems on macOS.

Any other comments?

Versions used: python=3.6.3, mono=5.2.0.224, dotnet=2.0.0 on High Sierra (10.13).
Build with --xplat and tested on Mono/DotNetCore projects.
@dmitriyse if this not appropriate; I can submit the minor fixes in a new PR

Checklist

Check all those that are applicable and complete.

Make sure to include one or more tests for your change
If an enhancement PR, please create docs and at best an example
Add yourself to AUTHORS
Updated the CHANGELOG

Hi! Thank you for testing under Mac and your fixes.
I reviewed your changes and will approve them once CI finish builds.

I don't know how but ms fixed their Case issue with System.XML.dll(that should be System.Xml.dll) (probably with help of CoreCLR 2.0.0 improvement or silent myget package update).
Otherwise travis CI builds should fails.

PRs merging process currently is too slow.
We needs to do something with this.

Codecov Report

Merging #557 into master will decrease coverage by 0.08%.
The diff coverage is 69.76%.

@@            Coverage Diff             @@
##           master     #557      +/-   ##
==========================================
- Coverage   77.16%   77.08%   -0.09%     
==========================================
  Files          65       65              
  Lines        5610     5651      +41     
  Branches      888      888              
==========================================
+ Hits         4329     4356      +27     
- Misses        993     1007      +14     
  Partials      288      288

Flag	Coverage Δ
#setup_linux	71.83% <0%> (-1.82%)	⬇️
#setup_windows	76.23% <69.76%> (-0.08%)	⬇️

Impacted Files	Coverage Δ
src/runtime/exceptions.cs	80.35% <100%> (ø)	⬆️
src/runtime/runtime.cs	88.23% <68.57%> (-2.75%)	⬇️
setup.py	88.97% <71.42%> (-0.85%)	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df1c224...4cb92d4. Read the comment docs.

Codecov Report

Merging #557 into master will decrease coverage by 0.05%.
The diff coverage is 69.44%.

@@            Coverage Diff             @@
##           master     #557      +/-   ##
==========================================
- Coverage   77.14%   77.08%   -0.06%     
==========================================
  Files          65       65              
  Lines        5617     5651      +34     
  Branches      888      888              
==========================================
+ Hits         4333     4356      +23     
- Misses        996     1007      +11     
  Partials      288      288

Flag	Coverage Δ
#setup_linux	71.83% <ø> (ø)	⬆️
#setup_windows	76.23% <69.44%> (-0.06%)	⬇️

Impacted Files	Coverage Δ
src/runtime/exceptions.cs	80.35% <100%> (ø)	⬆️
src/runtime/runtime.cs	88.23% <68.57%> (-2.75%)	⬇️
setup.py	88.97% <0%> (ø)	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5a002f...e09749c. Read the comment docs.

@fractus why do you have WIP in your PR title? Let me also check tomorrow morning on my Mac.

@denfromufa Thanks -- the reason for the WIP is the potentially breaking change in the csproj; I also want to check on Win10 and Linux.

On MSBuild 14 (VS2015) (Win7), the branch builds (python setup.py install) without any problems.

On MSBuild 15 and dotnet2.0 (Win10), the branch fails; standard pip install works for 2.3.0. In particular:

No xplat:

> python setup.py install --dry-run
running install
running build
running build_ext
Checking for updates from https://www.nuget.org/api/v2/.
Currently running NuGet.exe 4.3.0.
NuGet.exe is up to date.
MSBuild auto-detection: using msbuild version '15.3.409.57025' 
from 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\bin'.
All packages listed in packages.config are already installed.
Microsoft (R) Manifest Tool
Copyright (c) Microsoft Corporation.
All rights reserved.
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Microsoft (R) Build Engine version 15.3.409.57025 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Microsoft\NuGet\15.0\Microsoft.NuGet.targets(178,5): 
error : Your  project.json doesn't have a runtimes section. You should add '"runtimes": { "win": { } }' to your project.json and 
then re-run NuGet restore. [c:\dev\pythonnet\src\runtime\Python.Runtime.csproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Microsoft\NuGet\15.0\Microsoft.NuGet.targets(178,5): 
error : Your  project.json doesn't have a runtimes section. You should add '"runtimes": { "win-x64": { } }' to your project.json and 
then re-run NuG et restore. [c:\dev\pythonnet\src\clrmodule\clrmodule.csproj]
Traceback (most recent call last):
  File "setup.py", line 516, in <module>
    zip_safe=False,
  ...
  File "setup.py", line 259, in build_extension
    subprocess.check_call(" ".join(cmd + ["/t:Build"]), shell=use_shell)
  File "c:\tools\Anaconda3\envs\c36\lib\subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe" pythonnet.sln 
/p:Configuration=ReleaseWin /p:Platform=x64 
/p:DefineConstants="PYTHON36%3BPYTHON3%3BUCS2" 
/p:PythonBuildDir="c:\dev\pythonnet\build\lib.win-amd64-3.6" /p:PythonInteropFile="interop36.cs" 
/verbosity:minimal /p:PythonManifest="c:\dev\pythonnet\build\lib.win-amd64-3.6\app.manifest" 
/t:Build' returned non-zero exit status 1.

With xplat:

> python setup.py install --xplat --dry-run
running install
running build
running build_ext
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for c:\dev\pythonnet\src\embed_tests\Python.EmbeddingTest.15.csproj...
  Restoring packages for c:\dev\pythonnet\src\runtime\Python.Runtime.15.csproj...
  Restoring packages for c:\dev\pythonnet\src\console\Console.15.csproj...
  Restore completed in 18.12 ms for c:\dev\pythonnet\src\clrmodule\clrmodule.15.csproj.
  Restoring packages for c:\dev\pythonnet\src\testing\Python.Test.15.csproj...
  Restore completed in 181.26 ms for c:\dev\pythonnet\src\testing\Python.Test.15.csproj.
  Restore completed in 203.27 ms for c:\dev\pythonnet\src\runtime\Python.Runtime.15.csproj.
  Restore completed in 253.6 ms for c:\dev\pythonnet\src\console\Console.15.csproj.
  Restore completed in 590.73 ms for c:\dev\pythonnet\src\embed_tests\Python.EmbeddingTest.15.csproj.
Microsoft (R) Manifest Tool
Copyright (c) Microsoft Corporation.
All rights reserved.
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

c:\dev\pythonnet\src\testing\Python.Test.15.csproj(2,31): error MSB4236: The SDK 'Microsoft.NET.Sdk' specified could not be found.
c:\dev\pythonnet\src\console\Console.15.csproj(2,31): error MSB4236: The SDK 'Microsoft.NET.Sdk' specified could not be found.
c:\dev\pythonnet\src\clrmodule\clrmodule.15.csproj(3,31): error MSB4236: The SDK 'Microsoft.NET.Sdk' specified could not be found.
c:\dev\pythonnet\src\embed_tests\Python.EmbeddingTest.15.csproj(2,31): error MSB4236: The SDK 'Microsoft.NET.Sdk' specified could not b e found.
c:\dev\pythonnet\src\runtime\Python.Runtime.15.csproj(2,31): error MSB4236: The SDK 'Microsoft.NET.Sdk' specified could not be found.
Traceback (most recent call last):
  File "setup.py", line 516, in <module>
    zip_safe=False,
  ...
  File "setup.py", line 258, in build_extension
    subprocess.check_call(" ".join(cmd + ["/t:Clean"]), shell=use_shell)
  File "c:\tools\Anaconda3\envs\c36\lib\subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe" pythonnet.15.sln 
/p:Configuration=ReleaseWin /p:Platform=x64 
/p:CustomDefineConstants="PYTHON36%3BPYTHON3%3BUCS2" 
/p:PythonBuildDir="c:\dev\pythonnet\build\lib.win-amd64-3.6" /p:PythonInteropFile="interop36.cs" 
/verbosity:minimal /p:PythonManifest="c:\dev\pythonnet\build\lib.win-amd64-3.6\app.manifest" 
/t:Clean' returned non-zero exit status 1.

The following "manual" builds work:

> dotnet build src\runtime\Python.Runtime.15.csproj /p:Configuration=ReleaseWinPY3

    8 Warning(s)
    0 Error(s)

Time Elapsed 00:00:04.50

but this one fails:

> dotnet build src\clrmodule\clrmodule.15.csproj /p:Configuration=DebugWinPY3
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  clrmodule.15 -> c:\dev\pythonnet\src\clrmodule\bin\clrmodule.dll
C:\Users\vpapakos\.nuget\packages\unmanagedexports\1.2.7\tools\RGiesecke.DllExport.targets(58,3): error MSB4062: 
The "RGiesecke.DllExport.MSBuild.DllExportAppDomainIsolatedTask" 
task could not be loaded from the assembly 
C:\Users\vpapakos\.nuget\packages\unmanagedexports\1.2.7\tools\RGiesecke.DllExport.MSBuild.dll. 
Could not load file or assembly 'Microsoft.Build.Utilities, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. 
The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, 
and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [c:\dev\pythonnet\src\clrmodule\clrmodule.15.csproj]

Build FAILED.

I will try to debug this problem. --dry-run switch was never debugged with --xplat

Something that may be helpful: on Win10 + MSBuild15, installing the wheel file from the PR artifacts worked:

C:\temp
(c36) λ pip install pythonnet-2.4.0.dev0-cp36-cp36m-win_amd64.whl
Processing c:\temp\pythonnet-2.4.0.dev0-cp36-cp36m-win_amd64.whl
Installing collected packages: pythonnet
Successfully installed pythonnet-2.4.0.dev0

C:\temp
(c36) λ python
Python 3.6.3 |Anaconda, Inc.| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> clr
<module 'clr' from 'C:\\tools\\anaconda3\\envs\\c36xplat\\lib\\site-packages\\clr.pyd'>
>>> clr.__version__
'2.4.0.dev0'

> dotnet build src\clrmodule\clrmodule.15.csproj /p:Configuration=DebugWinPY3
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  clrmodule.15 -> c:\dev\pythonnet\src\clrmodule\bin\clrmodule.dll
C:\Users\vpapakos\.nuget\packages\unmanagedexports\1.2.7\tools\RGiesecke.DllExport.targets(58,3): error MSB4062: 
The "RGiesecke.DllExport.MSBuild.DllExportAppDomainIsolatedTask" 
task could not be loaded from the assembly 
C:\Users\vpapakos\.nuget\packages\unmanagedexports\1.2.7\tools\RGiesecke.DllExport.MSBuild.dll. 
Could not load file or assembly 'Microsoft.Build.Utilities, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. 
The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, 
and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [c:\dev\pythonnet\src\clrmodule\clrmodule.15.csproj]

Build FAILED.

I figured out the problem

dotnet msbuild (it's worked on dotnet core)
and
VS 2017 Dev Env -> msbuild (it's worked on .Net Framework 4.x)
Is not the same.

So you should execute (under VS 2017 DevEnvironment)

msbuild src\clrmodule\clrmodule.15.csproj /p:Configuration=DebugWinPY3

@fractus @dmitriyse so is there still any open issue on mac or windows. did you try to build this under linux?

Some work still needed to handle --dry-run correctly.

dotnet msbuild .... will never work on windows, so @fractus should change his test scheme.

Win

Agreed with @dmitriyse on dotnet msbuild -- we should drop it from Windows
[PENDING] Still failing with MSBuild15

Linux

Build without problems with Mono 5.4 and Python 3.6; also with --xplat (with dotnet)

(Sorry, for the message pollution -- closed it by mistake)

@fractus , sorry but problem is solely in your environment.
I tested the same commands on a super fresh windows 10, VS 2017, python 3.6, and problem was not reproduced.

Please read some related post.
https://andrewlock.net/the-sdk-microsoft-net-sdk-web-specified-could-not-be-found/

Ensure that you are able to build any .net core 2.0 project with your VS 2017 msbuild.

@fractus, please merge my new commit from #546 (6457aa8)

Our build is broken even for the master. So Travis-CI will fail.

@dmitriyse thanks for troubleshooting mono bug, @fractus this should now build fine

…tCoreApp 2.0).

…m interop*.cs file inside IDE.

…Debug/Release.

dmitriyse self-assigned this Oct 15, 2017

dmitriyse self-requested a review October 15, 2017 21:12

dmitriyse removed their assignment Oct 15, 2017

dmitriyse approved these changes Oct 15, 2017

View reviewed changes

den-run-ai requested review from dmitriyse, filmor, tonyroberts and vmuriart November 4, 2017 04:03

den-run-ai added the requires review label Nov 4, 2017

fractus closed this Nov 4, 2017

fractus reopened this Nov 4, 2017

dse and others added 6 commits November 28, 2017 21:50

Interop fix for Py_ssize_t methods (it's produces problems only on Ne…

b8165d2

…tCoreApp 2.0).

PYTHONNET_INTEROP_FILE env var introduced to allow working with custo…

6c51ed5

…m interop*.cs file inside IDE.

pdb generation improved for all conditions Net 4.0/NetStandard 2.0 x …

ddffaf2

…Debug/Release.

Missing ; in runtime.cs

f8dcbe8

Commented case-sensitive copy

7556ff2

Fixed rebase conflicts

e09749c

den-run-ai mentioned this pull request Feb 10, 2018

Interop methods with Py_ssize_t works differently in NetCoreApp 2.0 #531

Merged

4 tasks

den-run-ai closed this Feb 14, 2018

WIP: Combine #546 and #531 by fractus · Pull Request #557 · pythonnet/pythonnet · GitHub

Repository navigation

WIP: Combine #546 and #531 - #557

WIP: Combine #546 and #531#557fractus wants to merge 6 commits intopythonnet:masterpythonnet/pythonnet:masterfrom fractus:pr546+531Copy head branch name to clipboard

Conversation

fractus commented Oct 15, 2017

What does this implement/fix? Explain your changes.

Does this close any currently open issues?

Any other comments?

Checklist

Uh oh!

dmitriyse commented Oct 15, 2017

Uh oh!

dmitriyse commented Oct 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 15, 2017

Codecov Report

Uh oh!

codecov Bot commented Oct 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

den-run-ai commented Oct 15, 2017

Uh oh!

fractus commented Oct 15, 2017

Uh oh!

fractus commented Oct 17, 2017

Uh oh!

dmitriyse commented Oct 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fractus commented Oct 26, 2017

Uh oh!

dmitriyse commented Oct 26, 2017

Uh oh!

den-run-ai commented Nov 4, 2017

Uh oh!

dmitriyse commented Nov 4, 2017

Uh oh!

dmitriyse commented Nov 4, 2017

Uh oh!

fractus commented Nov 4, 2017

Uh oh!

fractus commented Nov 4, 2017

Uh oh!

dmitriyse commented Nov 4, 2017

Uh oh!

dmitriyse commented Nov 4, 2017

Uh oh!

dmitriyse commented Nov 4, 2017

Uh oh!

den-run-ai commented Nov 7, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WIP: Combine #546 and #531#557
fractus wants to merge 6 commits into
pythonnet:masterfrom
fractus:pr546+531

dmitriyse commented Oct 15, 2017 •
edited
Loading

codecov Bot commented Oct 15, 2017 •
edited
Loading

dmitriyse commented Oct 17, 2017 •
edited
Loading