Changing dtype in example causes "Internal error" · Issue #116 · xtensor-stack/xtensor-python · GitHub

First of all, it was really easy to get started with xtensor-python using your cookiecutter template. Thanks!

I tried modifying one of the examples in a seemingly trivial way: I replaced double with uint8. And now it crashes! Any idea what's going on?

Here's the condensed example, adapted from example2():

#include "pybind11/pybind11.h"
#include "xtensor/xarray.hpp"

#define FORCE_IMPORT_ARRAY
#include "xtensor-python/pyarray.hpp"
#include "xtensor-python/pyvectorize.hpp"

namespace py = pybind11;
using std::uint8_t;

// I'm using uint8_t here instead of double
inline xt::pyarray<uint8_t> example2(xt::pyarray<uint8_t> &m)
{
    return m + 2;
}

PYBIND11_PLUGIN(example_module)
{
    using namespace pybind11::literals;
    xt::import_numpy();
    py::module m("example_module", "");
    m.def("example2", example2);
    return m.ptr();
}

I tried to run this test program:

import numpy as np
from example_module import example2

a = np.zeros((10,), np.uint8)
b = example2(a)

And it crashes with the following error:

Traceback (most recent call last):
  File "test2.py", line 5, in <module>
    b = example2(a)
RuntimeError: Precondition violation!
Internal error: trivial_assigner called with unequal types.
  /miniforge/envs/flyem-forge/include/xtensor/xassign.hpp(377)

I tried other dtypes, too:

Crashes: uint8_t, uint16_t, int8_t, int16_t
No crash: uint32_t, uint64_t, int32_t, int64_t, float, double

Setup details: I'm using OSX and clang with libc++, with the following conda packages:

numpy                     1.13.1                   py36_0
python                    3.6.2                         0
xtensor                   0.12.1                        0    conda-forge
xtensor-python            0.14.0                        0    conda-forge
pybind11                  2.1.1                    py36_0    conda-forge

Hi,

the problem is that m + 2 returns a xexpression<int> and not xexpression<uint8>. This is the generic C++ behavior, e.g.:

uint a = 3, b = 5;
auto c = a + b;
std::cout << std::is_same<decltype(c), int>::value << std::endl; // will print true!

However, we've become recently aware of this issue (2 days ago) and we'll be working on a fix. In the meantime, changing the return type from your example2 function to pyarray<int> should solve the issue.

Thanks for the detailed report @stuarteberg !

Thanks very much for the explanation.

the problem is that m + 2 returns a xexpression<int>

I had wondered about that, so I naïvely tried m + static_cast<uint8_t>(2), which made no difference. But after your explanation, I see why.

In the meantime, changing the return type ...

Instead of changing the return type, can I just cast the output to the correct type (without incurring large temporaries)?

Does xtensor already have a static_cast-like xfunction? Here's what I tried, which seems to work:

template <typename R, typename T>
inline R cast_element(T x)
{
	return static_cast<R>(x);
}

template <typename result_value_type, typename xpr_t>
auto xcast(xpr_t && xpr)
{
	auto cast = cast_element<result_value_type, typename xpr_t::value_type>;
	return xt::vectorize(cast)(xpr);
}

inline xt::pyarray<uint8_t> example2(xt::pyarray<uint8_t> &m)
{
	return xcast<uint8_t>(m+2);
}

Can you spot any problems with that approach?

... and, sorry, one more question. Now that it works for uint8_t, I want to expose this function for all dtypes. I tried this:

template <typename T> std::string typestring() { return "Unknown"; }
template <> std::string typestring<uint8_t>()  { return "uint8_t"; }
template <> std::string typestring<uint16_t>() { return "uint16_t"; }
template <> std::string typestring<uint32_t>() { return "uint32_t"; }
template <> std::string typestring<uint64_t>() { return "uint64_t"; }

template<typename T>
inline xt::pyarray<T> example2(xt::pyarray<T> &m)
{
	std::cout << "example2<" << typestring<T>() << ">" << std::endl;
	return xcast<T>(m+2);
}
PYBIND11_PLUGIN(example_module)
{
    using namespace pybind11::literals;
    xt::import_numpy();
    py::module m("example_module", "");
    m.def("example2", example2<uint8_t>);
    m.def("example2", example2<uint16_t>);
    m.def("example2", example2<uint32_t>);
    m.def("example2", example2<uint64_t>);
    return m.ptr();
}

With this test program:

import numpy as np
from example_module import example2

def test(dtype):
    print("Trying: " + str(dtype.__name__))
    b = example2(np.ones((10,), dtype))
    print("")

for dtype in [np.uint8, np.uint16, np.uint32, np.uint64]:
    test(dtype)

And it produces this output:

Trying: uint8
example2<uint8_t>

Trying: uint16
example2<uint16_t>

Trying: uint32
example2<uint32_t>

Trying: uint64
example2<uint8_t>

But as far as I can tell, pyarray is designed to do the right thing here. The pyarray typecaster does indeed check the dtype of the numpy array before permitting pybind11 to make the conversion.

What am I doing wrong here?

and it didn't work? looks good to me

btw. great solution! xcast is neat. maybe we'll integrate it into xtensor

(Sorry, I hit "enter" on my last comment too soon. See my edits for the complete question.)

Oh no! I see the bug in my test program. Stand by...

Sorry, I had a copy+paste error in that test program. But the proper test still shows a problem. I edited the code above. See the output for uint64. What's going on?

hmm, so pybind basically chooses the wrong overload, right?

Looks that way...

ok, what happens if you reorder the functions?

E.g.

PYBIND11_PLUGIN(example_module)
{
    using namespace pybind11::literals;
    xt::import_numpy();
    py::module m("example_module", "");
    m.def("example2", example2<uint64_t>);
    m.def("example2", example2<uint8_t>);
    m.def("example2", example2<uint16_t>);
    m.def("example2", example2<uint32_t>);
    return m.ptr();
}

and can you also try to remove the other overloads?

Good idea. In that case, the output is correct:

Trying: uint8
example2<uint8_t>

Trying: uint16
example2<uint16_t>

Trying: uint32
example2<uint32_t>

Trying: uint64
example2<uint64_t>

It seems that the uint64 call just defaults to whichever overload is listed first. When I put uint32 first, then I get this:

PYBIND11_PLUGIN(example_module)
{
    using namespace pybind11::literals;
    xt::import_numpy();
    py::module m("example_module", "");
    m.def("example2", example2<uint32_t>);
    m.def("example2", example2<uint8_t>);
    m.def("example2", example2<uint16_t>);
    m.def("example2", example2<uint64_t>);
    return m.ptr();
}

Trying: uint8
example2<uint8_t>

Trying: uint16
example2<uint16_t>

Trying: uint32
example2<uint32_t>

Trying: uint64
example2<uint32_t>

Thanks for testing. I am not sure where the overload resolution goes wrong. Either it's in the pybind11 code, or in our code. Will need to be investigated!

@yoshipon do you know if this issue could be caused by the type caster?

Worth noting: I am using pybind11 2.1.1 instead of the latest (2.2.1), because that's the newest version that is permitted by the current xtensor-python recipe on conda-forge.

I suppose I should try a newer version of pybind11 before reporting the problem to them...

I have to pause my investigation for now, but I'll try to get back to this later tonight or tomorrow.

I don't think that it should not make much of a difference (pybind11 2.2.1 vs 2.1.1). We pinned to 2.1.x because there was a breaking change in 2.2.0 that we reported and should be fixed in 2.2.1.

I just tested the overload resolution with the pybind11 cmake example and it works correctly. Let me investigate if it is our pybind11 variant or our code.

Sorry, this was maybe not clear: I tested with the pybind11 py::array that can also accept a numpy array.

Hi @stuarteberg thanks again for the report. I found the issue, and I am working on a fix.
I hope I'll be able to push a fix today.

Cheers,

Wolf

Changing dtype in example causes "Internal error" · Issue #116 · xtensor-stack/xtensor-python · GitHub

Repository navigation

Changing dtype in example causes "Internal error" #116

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Changing dtype in example causes "Internal error" · Issue #116 · xtensor-stack/xtensor-python · GitHub

Repository navigation

Changing dtype in example causes "Internal error" #116

Description

Activity

wolfv commented on Oct 25, 2017

SylvainCorlay commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

wolfv commented on Oct 25, 2017

wolfv commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

wolfv commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

wolfv commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

wolfv commented on Oct 25, 2017

wolfv commented on Oct 25, 2017

stuarteberg commented on Oct 25, 2017

SylvainCorlay commented on Oct 25, 2017

wolfv commented on Oct 26, 2017

wolfv commented on Oct 26, 2017

wolfv commented on Oct 26, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions