Repository navigation
Port HMAC implementation to new OpenSSL APIs #134531
Description
Feature or enhancement
Our current OpenSSL implementation of HMAC is based on the deprecated HMAC API.
We should migrate to the new EVP_MAC-HMAC API instead.
Linked PRs
- gh-134531: cleanup _hashopenssl.c to support EVP_MAC #134626
- [3.14] gh-134531: cleanup _hashopenssl.c to support EVP_MAC (GH-134626) #134703
- gh-134531: use EVP_MAC API for _hashlib.HMAC #135235
- gh-134531: fix _hashlib clinic directive post GH-134626 #135249
- gh-134531: refactor _hashlib logic for mapping NIDs to EVP_MD objects #135254
- gh-134531: prefer using _hashlib module state instead of module object #136865
- gh-134531: simplify code for computing HMAC digests #138046
Activity
picnixz commented on May 23, 2025
I can take care of this. Note that EVP_MAC-HMAC is only available since OpenSSL 3.x and the requirements for CPython are OpenSSL 1.1.1 and later. We still need to maintain the old API (though, we're already doing it through PY_EVP_MD macros) since our build requirements OpenSSL 3.0.9 is the recommended minimum version for the ssl and hashlib extension modules.
picnixz commented on May 24, 2025
Ok, so I had a quick look at how I would plan this. I'll need multiple PRs because _hashopenssl is a bit messy and now that we will use EVP_MAC in addition to EVP_MD, we should rename some functions. To that end, I'll first make a "cleanup" PR which prepares everything and a bit of refactoring especially to reuse functions later.
Note: I eventually decided against the backports, even if they could ease the life of my future self (see #134626 (comment) and #134703 (comment)).
gh-134531: cleanup `_hashopenssl.c` to support `EVP_MAC` (#134626)
pythongh-134531: cleanup `_hashopenssl.c` to support `EVP_MAC` (pytho…
picnixz commented on Jun 7, 2025
Ok, so here's the plan. I've finished writing the implementation, but the PR is huge:
So I'll break down the commits, because I also found multiple issues elsewhere. Well, not real issues, but annoying interfaces that should be refactored first.
There is one issue that I introduced in gh-134531: cleanup _hashopenssl.c to support EVP_MAC #134626, namely I renamed the structure types, but I forgot to rename them in clinic directives. It's not an issue because it's never used but we should first fix this. [fixed by 4372011]
Next, we have a way to map EVP_MD objects to NID (thank you OpenSSL), and digestmods to EVP_MD, and EVP_MD/digestmods to names. That's great. But the names in question are the "preferred" Python names, not the OpenSSL ones (well sometimes they can be OpenSSL ones as we use it as a fallback). Now, for EVP_MAC objects, there is no way to recover the underlying digest. Why? because MAC objects are not necessarily HMAC, and thus there's not always an underlying "digest" (sometimes it's a cipher). [see https://github.com/gh-134531: refactor _hashlib logic for mapping NIDs to EVP_MD objects #135254]
This is extremely annoying. Also there is no mapping from EVP_MAC to NIDs, and we can only know whether a MAC is a [HPC]MAC etc, not what the parameters for the underlying algorithm itself were. Now, the goods news is: since we provide a way to create HMAC objects using digestmods, we always get an EVP_MD object behind the scenes. From there, we can instantiate HMAC+digest correctly and remember the NID of that EVP_MD. To recover the underlying digest name, we just need to resolve the NID.
The last task will be to rely on digestmods <-> NIDS <-> OpenSSL digest names -> HMAC to construct the EVP_MAC_CTX object (we only need one EVP_MAC object because it's always HMAC).
The deprecated HMAC API is deprecated because it's actually wrapping an EVP_MD object. The new EVP_MAC API generalize MACs by delegating everything to providers, and callback functions, hiding the implementation of the actual MAC algorithm itself. Now, it's the provider that is wrapping an HMAC_CTX object instead of directly wrapping the EVP_MD objects. Long story short: we still have an EVP_MD somewhere but it's no more recoverable from EVP_MAC objects themselves (and there's no EVP_HMAC).
I have a branch with everything ready so I'll just make different PRs. At the same time, I'll also refactor _hashopenssl.c because the naming is a bit messy (we don't have a clear convention and I think it's better if we had one good convention when dealing with this module). Later, I'll need to refactor out everything so that we have a unified interface that can be used by _hashlib and _hmac for recognizing "digestmods", but for now, we'll keep things separate (this task is #131876).
gh-134531: fix `_hashlib` clinic directive post GH-134626 (#135249)
gh-134531: refactor `_hashlib` logic for handling NIDs and EVP_MDs (#…
pythongh-134531: fix `_hashlib` clinic directive post pythonGH-134626 (…
pythongh-134531: refactor `_hashlib` logic for handling NIDs and EVP_…
pythongh-134531: cleanup `_hashopenssl.c` to support `EVP_MAC` (pytho…
pythongh-134531: fix `_hashlib` clinic directive post pythonGH-134626 (…
2 remaining items
pythongh-134531: cleanup `_hashopenssl.c` to support `EVP_MAC` (pytho…
pythongh-134531: fix `_hashlib` clinic directive post pythonGH-134626 (…
pythongh-134531: refactor `_hashlib` logic for handling NIDs and EVP_…
pythongh-134531: prefer using `_hashlib` module state instead of modu…
pythongh-134531: fix `_hashlib` clinic directive post pythonGH-134626 (…
pythongh-134531: refactor `_hashlib` logic for handling NIDs and EVP_…
pythongh-134531: prefer using `_hashlib` module state instead of modu…
gh-134531: simplify code for computing HMAC digests (#138046)
gh-134531: use `EVP_MAC` API for `_hashlib.HMAC` when possible (#135235)
pythongh-134531: cleanup `_hashopenssl.c` to support `EVP_MAC` (pytho…
pythongh-159811: Raise OpenSSL errors in `_hashlib.HMAC` after re-att…
gh-159811: Raise OpenSSL errors in `_hashlib.HMAC` after re-attaching…
[3.15] gh-159811: Raise OpenSSL errors in `_hashlib.HMAC` after re-at…