Bug report

_hashlib calls notify_ssl_error_occurred_in() and notify_smart_ssl_error_occurred_in() inside HASHLIB_EXTERNAL_INSTRUCTIONS_{LOCKED,UNLOCKED} blocks, that is, while the thread
state is detached (and, for the LOCKED variant, while the object's mutex is held).

These helpers call PyErr_* functions; thus, if the libssl's update function fails on a buffer larger than HASHLIB_GIL_MINSIZE, the interpreter crashes instead of raising a ValueError:

• _hashlib.HASH.update(): _hashlib_HASH_hash() raises when EVP_DigestUpdate() fails.
• _hashlib.new() and the named constructors (e.g. _hashlib.openssl_sha256(data)): same helper, called on the initial data.
• _hashlib.HMAC.update() and _hashlib.hmac_new(): hashlib_openssl_HMAC_update_once() raises when EVP_MAC_update() (or HMAC_Update()) fails. This one is a regression introduced by Port HMAC implementation to new OpenSSL APIs #134531. On 3.14, the exception was raised after the GIL was re-acquired.

Note that the default OpenSSL provider never fails in these update functions. This is solely to protect provides that can fail. For buffers of at most 2048 bytes (GIL held), the same failure correctly raises a ValueError.

The same class of bug was fixed in _hmac by gh-135561 and gh-159786. The fix here is the same: report the libssl error after the thread state is re-attached and the lock is released. The OpenSSL error queue is per OS thread, so the message is preserved.

• The HASH constructor and update() paths are affected on 3.13 and later.
• The HMAC path is affected since 3.15.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-159811: Raise OpenSSL errors in _hashlib.HASH after re-attaching the thread state #159817
• gh-159811: Raise OpenSSL errors in _hashlib.HMAC after re-attaching the thread state #159818