I've been testing it a bit through a PR in my fork, and it all builds well, up to one point in the Cython sources where it fails in step 107/114: https://github.com/agriyakhetarpal/python-flint/actions/runs/13576857716/job/37954995494?pr=1

with the following error trace:

FAILED: src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o 
/tmp/tmp8t79rr74/cc -Isrc/flint/types/_gr.cpython-312-wasm32-emscripten.so.p -Isrc/flint/types -I../src/flint/types -I/opt/hostedtoolcache/Python/3.12.9/x64/include/python3.12 -I/home/runner/work/python-flint/python-flint/wasm-library-dir/include -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -MD -MQ src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o -MF src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o.d -o src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/meson-generated_src_flint_types__gr.pyx.c.o -c src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/src/flint/types/_gr.pyx.c
src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/src/flint/types/_gr.pyx.c:19051:65: error: incompatible integer to pointer conversion passing 'mp_limb_t' (aka 'unsigned long') to parameter of type 'const fmpz *' (aka 'const long *') [-Wint-conversion]
 19051 |   gr_ctx_init_fq_nmod(__pyx_v_ctx->__pyx_base.__pyx_base.ctx_t, __pyx_v_p, __pyx_v_d, __pyx_v_name);
       |                                                                 ^~~~~~~~~
/home/runner/work/python-flint/python-flint/wasm-library-dir/include/flint/gr.h:1366:53: note: passing argument to parameter 'p' here
 1366 | void gr_ctx_init_fq_nmod(gr_ctx_t ctx, const fmpz_t p, slong d, const char * var);
      |                                                     ^
src/flint/types/_gr.cpython-312-wasm32-emscripten.so.p/src/flint/types/_gr.pyx.c:19150:65: error: incompatible integer to pointer conversion passing 'mp_limb_t' (aka 'unsigned long') to parameter of type 'const fmpz *' (aka 'const long *') [-Wint-conversion]
 19150 |   gr_ctx_init_fq_zech(__pyx_v_ctx->__pyx_base.__pyx_base.ctx_t, __pyx_v_p, __pyx_v_d, __pyx_v_name);
       |                                                                 ^~~~~~~~~
/home/runner/work/python-flint/python-flint/wasm-library-dir/include/flint/gr.h:1367:53: note: passing argument to parameter 'p' here
 1367 | void gr_ctx_init_fq_zech(gr_ctx_t ctx, const fmpz_t p, slong d, const char * var);
      |                                                     ^
2 errors generated.

FWIW; I hardly have any Cython development experience, so I will have to wrap my head around this a bit, unless someone has an idea for a fix. The basis for the failure here is that WASM is stricter about type conversions than compilation for other platforms, which is why this mismatch is causing errors.

Okay – the cause of the error, more specifically, is that gr_fq_nmod_ctx and gr_fq_zech_ctx pass a ulong parameter p

but flint/gr.h has fmpz_t in version 3.0.1: https://github.com/flintlib/flint/blob/c05007be863b1aae7221f11a9c135d673d968638/src/gr.h#L1366-L1367

and this has apparently been fixed in version v3.1.0. I'll try updating flint to v3.1.0 (before proceeding to try out v3.1.2, if I do)

Successful build: https://github.com/agriyakhetarpal/python-flint/actions/runs/13577635441?pr=1 ✨

The next task is to run tests, which I can take a look at later in the day.

I haven't looked through this yet but thanks!

but flint/gr.h has fmpz_t in version 3.0.1:

Best to keep with latest FLINT version for the pyodide build. There is no need to support a range of versions in pyodide. Generally python-flint tracks the latest version but has some #ifdef type stuff for older versions so that someone could build against them. No one is going to build against old versions in WASM though.

Sounds good to me! I think the tests will have a bunch of function signature mismatches, though 😬 based on what I noticed through my fork's run. Also, building itself takes ~20 minutes in total – we could get to half of that if we have prebuilt WASM binaries. I guess they could be hosted somewhere in a GitHub release when things are ready.

This can take a bit of time. I'm disabling tests and running it in on my fork to see how widespread the signature mismatches are, and I managed to get test_all.py to run till the end – now we have some more in test_docstrings.py. My next tasks will be to debug these mismatches, which I haven't had fun doing, as, i. compilation of these libraries to WASM is messy on a macOS and ii. a test here contains several assert statements, instead of several tests containing a few of them. Please bear with me as I proceed with this slowly and steadily. We could also wholly ignore test_docstrings.py if you're not too keen on that being tested. (I think it's good to test, though).

I don't quite understand what the problem with the tests is. Is it just caused by the Cython/C signatures not matching?

There is now FLINT 3.2.0-rc1. We could perhaps update all the Cython declarations to that version and use that for the WASM build. I'm not sure when 3.2.0 final will be released but we would want to update to that straight away when it is.

I don't quite understand what the problem with the tests is. Is it just caused by the Cython/C signatures not matching?

Yes, majorly. If there's even a slight mismatch between the upstream signatures for Flint and the ones we have here, WASM's type safety guarantees that it will terminate the program, which makes Pyodide raise a fatal error. These can happen at the time of linking (like when it happened with version 3.0.1 above), or at runtime, like we are facing now. I have to drop these completely instead of xfailing them as the running Pyodide interpreter is no longer possible to be used after it has encountered fatal errors.

There are so many of these with SciPy, especially from its wrapping of Fortran libraries that we can't compile to WASM directly without having to f2c them over. I hope the new Flang from LLVM-19 can do this better: pyodide/pyodide#5268

There is now FLINT 3.2.0-rc1. We could perhaps update all the Cython declarations to that version and use that for the WASM build. I'm not sure when 3.2.0 final will be released but we would want to update to that straight away when it is.

Yes, this sounds good to me, and I hope they've fixed a few of these upstream. I could start another PR for this when I have enough time to do so; I assume it will help me learn Cython a bit. :)

I've opened gh-264 to bump the FLINT version to 3.2.0-rc1 and update all of the Cython declarations.

The Cython declarations are set from the FLINT docs by running the bin/all_rst_to_pxd.py script. There can often be mismatches resulting from parsing the signatures from the docs rather than the actual C code e.g. the docs may be inaccurate but usually in the Cython -> C -> compiled extension modules the differences even out largely because a lot of FLINT's functions are not actually called directly by python-flint.

I've opened gh-264 to bump the FLINT version to 3.2.0-rc1 and update all of the Cython declarations.

I've just merged this to main. If you rebase/merge with main then this PR will get the updated declarations.

There might still be some wrong declarations that we can fix manually for now but then upstream to FLINT afterwards.

Ah, yes, this time it compiles. Here is the output from emmake make check:

make: make check
add_ssaaaa...
add_ssaaaa                                        0.00   (PASS)
add_sssaaaaaa...
add_sssaaaaaa                                     0.09   (PASS)
flint_clz...
flint_clz                                         0.00   (PASS)
flint_ctz...
flint_ctz                                         0.00   (PASS)
flint_fprintf...
flint_fprintf                                     0.00   (FAIL)
Could not open temporary file "tmp"
Aborted()
/Users/agriyakhetarpal/Desktop/flint/build/test/main:380
  var e = new WebAssembly.RuntimeError(what);
          ^

RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
    at abort (/Users/agriyakhetarpal/Desktop/flint/build/test/main:380:11)
    at __abort_js (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3388:7)
    at main.wasm.abort (wasm://wasm/main.wasm-083fa0d6:wasm-function[8641]:0x4647b4)
    at main.wasm.flint_abort (wasm://wasm/main.wasm-083fa0d6:wasm-function[40]:0x90a0)
    at main.wasm.main (wasm://wasm/main.wasm-083fa0d6:wasm-function[38]:0x9010)
    at Module._main (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3728:102)
    at callMain (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3754:15)
    at doRun (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3793:24)
    at run (/Users/agriyakhetarpal/Desktop/flint/build/test/main:3806:5)
    at removeRunDependency (/Users/agriyakhetarpal/Desktop/flint/build/test/main:348:7)

Node.js v20.18.0
make: *** [build/test/main_TEST_RUN] Error 1
emmake: error: 'make check' failed (returned 2)

The failing test is fine; we can't access the file system without mounting a NodeFS or a similar one. I don't think we should dwell on it.

Note that I had to manually modify the generated Makefile as it did not interpret that build/test/main was a JavaScript file, so I had to prepend node before the %_TEST_RUN: % rules. It should build main.js if the HOST_OS matches emscripten and use Node.js in that case by default – but that was just some misconfiguration that we can deal with. At least from this, we can see that most tests pass and WASM compilation works.

I hope we are not missing any tests, though. Looks like we are :/

I hope we are not missing any tests, though. Looks like we are :/

Yes, usually there are more than 5 tests!

I added NODERAWFS=1 to my TESTCFLAGS for allowing access to my file system, and this has let the tests go forward much more:

make: make check
add_ssaaaa...
add_ssaaaa                                        0.00   (PASS)
add_sssaaaaaa...
add_sssaaaaaa                                     0.04   (PASS)
flint_clz...
flint_clz                                         0.00   (PASS)
flint_ctz...
flint_ctz                                         0.00   (PASS)
flint_fprintf...
flint_fprintf                                     0.01   (PASS)
flint_printf...
flint_printf                                          (SKIPPED)
memory_manager...
memory_manager                                    0.00   (PASS)
sdiv_qrnnd...
sdiv_qrnnd                                        0.01   (PASS)
smul_ppmm...
smul_ppmm                                         0.01   (PASS)
flint_sort...
flint_sort                                        0.00   (PASS)
sub_dddmmmsss...
sub_dddmmmsss                                     0.05   (PASS)
sub_ddmmss...
sub_ddmmss                                        0.00   (PASS)
udiv_qrnnd...
udiv_qrnnd                                        0.01   (PASS)
udiv_qrnnd_preinv...
udiv_qrnnd_preinv                                 0.00   (PASS)
umul_ppmm...
umul_ppmm                                         0.01   (PASS)
thread_pool...
thread_pool                                       0.10   (PASS)
thread_support_parallel_binary_splitting...
thread_support_parallel_binary_splitting          0.00   (PASS)
thread_support_parallel_do...
thread_support_parallel_do                        0.00   (PASS)
n_addmod...
n_addmod                                          0.00   (PASS)
n_cbrt_binary_search...
n_cbrt_binary_search                              0.00   (PASS)
n_cbrt...
n_cbrt                                            0.01   (PASS)
n_cbrt_chebyshev_approx...
n_cbrt_chebyshev_approx                           0.00   (PASS)
n_cbrtrem...
n_cbrtrem                                         0.01   (PASS)
n_clog_2exp...
n_clog_2exp                                       0.00   (PASS)
n_clog...
n_clog                                            0.00   (PASS)
compute_primes...
Aborted(OOM)
/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:380
  var e = new WebAssembly.RuntimeError(what);
          ^

RuntimeError: Aborted(OOM). Build with -sASSERTIONS for more info.
    at abort (/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:380:11)
    at abortOnCannotGrowMemory (/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:3949:7)
    at _emscripten_resize_heap (/Users/agriyakhetarpal/Desktop/flint/build/ulong_extras/test/main:3955:7)
    at main.wasm.sbrk (wasm://wasm/main.wasm-084fa6f2:wasm-function[8876]:0x4812bd)
    at main.wasm.emscripten_builtin_malloc (wasm://wasm/main.wasm-084fa6f2:wasm-function[8868]:0x47f4c6)
    at main.wasm._flint_malloc (wasm://wasm/main.wasm-084fa6f2:wasm-function[155]:0x186f6)
    at main.wasm.flint_malloc (wasm://wasm/main.wasm-084fa6f2:wasm-function[147]:0x1850f)
    at main.wasm.n_compute_primes (wasm://wasm/main.wasm-084fa6f2:wasm-function[189]:0x1a0b1)
    at main.wasm.n_primes_arr_readonly (wasm://wasm/main.wasm-084fa6f2:wasm-function[271]:0x1fe54)
    at main.wasm.test_compute_primes (wasm://wasm/main.wasm-084fa6f2:wasm-function[27]:0x50f5)

Node.js v20.18.0
make: *** [build/ulong_extras/test/main_TEST_RUN] Error 1
emmake: error: 'make check' failed (returned 2)

The relevant section of the main file that Emscripten has set up looks like this (lines 3944–3956), but I don't think this helps much:

  var getHeapMax = () =>
      HEAPU8.length;
  var _emscripten_get_heap_max = () => getHeapMax();

  var abortOnCannotGrowMemory = (requestedSize) => {
      abort('OOM');
    };
  var _emscripten_resize_heap = (requestedSize) => {
      var oldSize = HEAPU8.length;
      // With CAN_ADDRESS_2GB or MEMORY64, pointers are already unsigned.
      requestedSize >>>= 0;
      abortOnCannotGrowMemory(requestedSize);
    };

And from the CI jobs for FLINT, I see that none of the nmod_vec_, flint_mpn, fmpz, fmpq, arb, gr_ tests ran; this isn't the full test suite either, so it would be nice if there were a pytest-like way to skip tests that we know are going to abort the program without having to recompile everything, or to allow a selective portion of the tests of our choice to run, similar to the -k option.

it would be nice if there were a pytest-like way to skip tests that we know are going to abort the program without having to recompile everything, or to allow a selective portion of the tests of our choice to run, similar to the -k option.

I'm not sure what options there are but you can just run the individual test files like:

$ build/acb/test/main
acb_acos...
acb_acos                                          0.00   (PASS)
acb_acosh...
acb_acosh                                         0.00   (PASS)
acb_agm1...
acb_agm1                                          0.15   (PASS)
acb_agm...
acb_agm                                           0.09   (PASS)
...

You can list the hundred or so test executables and maybe put them in a script and comment out the ones you don't want:

$ ls build/*/test/main
build/acb_calc/test/main               build/fq_default_mat/test/main
build/acb_dft/test/main                build/fq_default_poly_factor/test/main
build/acb_dirichlet/test/main          build/fq_default_poly/test/main
build/acb_elliptic/test/main           build/fq_default/test/main
...

Given the things that currently don't work in this PR I would be most interested in acb_modular, nf, nf_elem, and the gr_* tests.

Okay, so the CI job passes now. I think that a good milestone here would be to get something merged so that we have emscripten running in CI and nightly wheels being uploaded even if there are some outstanding known issues.

I suggest we:

• Preserve this PR with its many commits and long discussion intact for posterity.
• Open a new PR that squashes the commits and does any remaining clean ups.
• Then I will follow up after with something to clean up the test changes and come up with a better solution for the doctests that are skipped or sorting of polynomial factors.

Lastly what remains is to see if we can identify and fix the problems with acb_modular_hilbert_class_poly and gr_nf but unless we can quickly identify the issue there I think it is better to leave that for later.

Sounds good to me! I will create another PR from here and also obtain some additional WASM test results from FLINT-main in a new issue – hopefully by tonight or early tomorrow. As most of the tests pass, I think the next steps, after yours, are to:

• get patches for updated versions of FLINT and python-flint accordingly to https://github.com/pyodide/pyodide-recipes/ so that we can ensure that they are included in Pyodide 0.28 (which will come in a few weeks from now).
• use Pyodide 0.28 for testing SymPy in its Pyodide CI, and include the python-flint tests that were being skipped because it's not installed, and address any problems there (perhaps it makes sense to address problems here first, I can't say)
• Install python-flint alongside SymPy for SymPy Live, and try to resolve Display the SymPy version in the shell prompt sympy/live#23

Not sure how urgent this is: should we hold up the 3.3 release of FLINT to try to get all tests to pass on WASM first, or is it OK if we release 3.3 now and push any further fixes for WASM compatibility to a future release?

I think for now we can track FLINT's main branch for the WASM build so I think the FLINT release doesn't matter. At least the current state in this PR worked good enough with FLINT main and so if FLINT 3.3 is released now then it should be possible to get at least as good as we have here by pinning to FLINT 3.3.

How long it would take to iron these things out is unknown so it is probably best not to wait on it.

I'm reopening to test after gh-286 which updates the FLINT version to 3.3.0 and also updates the cython declarations for FLINT C functions.

The problem with fmpz_poly.hilbert_class_poly seems resolved. The problem with gr_nf is not (and is the only outstanding issue in this PR):

../.venv-pyodide/lib/python3.13/site-packages/flint/test/test_docstrings.py::test_docstrings[flint.types._gr.new13] PASSED
Pyodide has suffered a fatal error. Please report this to the Pyodide maintainers.
The cause of the fatal error was:
RuntimeError: memory access out of bounds
    at wasm://wasm/020eeec2:wasm-function[9770]:0x4883e2
    at wasm://wasm/02dce452:wasm-function[15599]:0x9273aa
    at wasm://wasm/02dce452:wasm-function[15583]:0x926e37
    at wasm://wasm/02dce452:wasm-function[14736]:0x8d16a2
    at wasm://wasm/02fbdc16:wasm-function[422]:0xc40a5
    at wasm://wasm/020eeec2:wasm-function[[230](https://github.com/flintlib/python-flint/actions/runs/15597884038/job/43932170844#step:15:231)6]:0x1b8b1d
    at wasm://wasm/02fbdc16:wasm-function[685]:0xf20ad
    at wasm://wasm/020eeec2:wasm-function[2306]:0x1b8b1d
    at wasm://wasm/020eeec2:wasm-function[1962]:0x1a0d0a
    at wasm://wasm/020eeec2:wasm-function[1949]:0x1a01c4 {
  pyodide_fatal_error: true
}

https://github.com/flintlib/python-flint/actions/runs/15597884038/job/43932170844

The gr_nf issue was revealed again by reenabling the doctests in ee057b4.

I think we can close this one now. Much progress was made elsewhere. There are still some FLINT/emscripten hickups but I don't think this PR is useful for those any more.