I'm leaving my dev session for now so I'll be back tomorrow if needs arise.

Mmh, I'll need some configure hacks when only BLAKE2 is present and other hash functions are disabled (otherwise the build bot will never build and it's not very useful).

I don't think there's any issue with shared objects containing simd instructions, as long as the files are compiled with compiler flags consistent with their suffix (e.g. -mavx2 only for Hacl_*_Simd256) prior to linking into a shared object (which we already do properly).

Thanks for looking into this!

Ok, it's not really a simplification of the build, but I managed to eliminate static libraries.

!buildbot FIPS

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 8d86ff6 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F132438%2Fmerge

The command will test the builders whose names match following regular expression: FIPS

The builders matched are:

• AMD64 CentOS9 FIPS No Builtin Hashes PR
• AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR
• AMD64 RHEL8 FIPS No Builtin Hashes PR
• AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR

Ok, so for WASI I don't know what to do :D

Ok, so nothing was fixed. I give up. I don't know how to do it for WASI. Maybe static linking is the only way to do it?

Ok, so here is how I eventually fix:

• HACL* can be built statically or dynamically. In other words, it's not possible to just use Setup.stdlib.in. Why? if I were to specify a .o file, it would actually create rules automatically and I won't be able to customize the HACL* build itself.
• So, what I do is the following: I just say "ok build this extension module but don't bother about the dependencies, I'll tell you which ones are dependencies".

For regular builds, there are two kinds of dependencies:

• the Makefile rule prequisites.
• the actual files to pass to the linker when building the final *.so or *.a file.

For instance, to build the HACL*-based MD5 module I first need to build the HACL* MD5 part and then build md5module.o itself, and finally link _md5.so.

We have:

Modules/_hacl/Hacl_Hash_MD5.o: $(srcdir)/Modules/_hacl/Hacl_Hash_MD5.c $(LIBHACL_MD5_HEADERS)
	$(CC) -c $(LIBHACL_CFLAGS) -o $@ $(srcdir)/Modules/_hacl/Hacl_Hash_MD5.c
$(LIBHACL_MD5_LIB_STATIC): $(LIBHACL_MD5_OBJS)
	-rm -f $@
	$(AR) $(ARFLAGS) $@ $(LIBHACL_MD5_OBJS)

This step builds the HACL* part. The $(LIBHACL_MD5_LIB_STATIC) rule may not necessarily be used later but it's the rule needed to build statically the HACL* MD5 part. It's as if I was building a separate project.

Then, I have explicitly defined:

MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_MD5_HEADERS) $(LIBHACL_MD5_LIB_SHARED)
MODULE__MD5_LDEPS=$(LIBHACL_MD5_LIB_SHARED)

The first variable is for everything related to rule dependencies. In this example, to build the md5 (cpython) module, I need the HACL* library and some other stuff. When calling the linker, I only need to pass the .o files but I also need them to be built separately, which is why they are marked as LDEPS. The LDEPS variable is only used when I want to indicate additional linker rule prerequisites. Example:

Modules/md5module.o: $(srcdir)/Modules/md5module.c $(MODULE__MD5_DEPS) $(MODULE_DEPS_SHARED) $(PYTHON_HEADERS)
	$(CC) $(MODULE__MD5_CFLAGS) $(PY_STDMODULE_CFLAGS) $(CCSHARED) -c $(srcdir)/Modules/md5module.c -o Modules/md5module.o
Modules/_md5$(EXT_SUFFIX):  Modules/md5module.o $(MODULE__MD5_LDEPS)
	$(BLDSHARED)  Modules/md5module.o $(MODULE__MD5_LDFLAGS) $(LIBPYTHON) -o Modules/_md5$(EXT_SUFFIX)

Note that the rules for Modules/_md5$(EXT_SUFFIX) are automatically created by makesetup and I cannot just change them. I need to tell makesetup that I have an extra prerequisite for that rule but I cannot tell it via Setup.stdlib.in. So I introduced the LDEPS variable instead. Note that it can be different from the LDFLAGS variable (in this case, it's not).

Now what was the issue with WASI? well... before, I didn't add the .a file as a rule prequisite (namely, I forgot to also include it in the DEPS variable). Because of that, when executing the libpython rule, there was a missing prequisite, which is why the static HACL* library was never built, leading to a missing file error. It's because LIBPYTHON was built before the extension modules were built.

It works!!

🤖 New build scheduled with the buildbot fleet by @picnixz for commit c87bb27 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F132438%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All failures are unrelated or already known. So I think we're good with this change. I'll now actually check if freeze.py works (I actually didn't test this...).

The following works:

$ ./configure -q --prefix="$(pwd)/build" --libdir="$(pwd)/build/lib" --exec-prefix="$(pwd)/build" && make -j12 && make install
$ echo 'print("Hello world")' >> hello.py
$ ./python ./Tools/freeze/freeze.py -o frozenhello hello.py
$ make -C frozenhello

looks like Modules/Setup was missed as part of this patch -- I opened #133012 to update that as well!