FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix constexpr evaluation of validate_utf8_with_counts by lemire · Pull Request #7 · BenjaminBucher/simdutf · GitHub

fix constexpr evaluation of validate_utf8_with_counts - #7

Open
lemire wants to merge 1 commit into
BenjaminBucher:simdunicode-portfrom
lemire:fix-constexpr-counts
Open

fix constexpr evaluation of validate_utf8_with_counts#7
lemire wants to merge 1 commit into
BenjaminBucher:simdunicode-portfrom
lemire:fix-constexpr-counts

Conversation

lemire commented Aug 14, 2026

Copy link
Copy Markdown

Targets simdunicode-port (the branch behind simdutf#974).

The problem

scalar::utf8::validate_with_counts is marked simdutf_constexpr23, and the span overload in implementation.h dispatches to it from an if consteval branch. But its block-wise ASCII fast path calls std::memcpy, which is not usable during constant evaluation. So any constant-evaluated call with 16 or more bytes fails to compile:

include/simdutf/scalar/utf8.h:271: error: 'memcpy(((void*)(& v1)),
  ((const void*)"the quick brown fox jumps"), 8)' is not a constant expression

Inputs shorter than 16 bytes never reach the fast path, which is why this went unnoticed.

scalar::utf8::validate() in the same header already handles this with an if !consteval guard; validate_with_counts was ported without it.

The fix

Wrap the fast path in the same #if SIMDUTF_CPLUSPLUS23 / if !consteval / #endif guard used by validate(), and match its uint64_t v1{} initialization.

Also adds constexpr assertions to tests/validate_utf8_with_counts_boundary_tests.cpp (guarded by SIMDUTF_CPLUSPLUS23 && SIMDUTF_SPAN, no CMake change needed), covering a pure-ASCII input longer than one fast-path block, a long ASCII run followed by 2-, 3- and 4-byte sequences, and an invalid byte past the first block.

Verification

On a Xeon Gold 6548N / Fedora box:

  • The new test reproduces the bug: with the test in place but the header guard reverted, the C++23 build fails with the memcpy ... is not a constant expression errors above.
  • With the guard, builds and passes for g++-14 and clang++-18 across C++17 / C++20 / C++23 (all six combinations).
  • Full ctest at C++23: 93/93 pass.
  • clang-format reports no changes on either file.

ubuntu24-cxxstandards.yml runs cppversion: [17, 20, 23], so the C++23 leg of CI exercises the new assertions.

No behavioural change at run time: the guard only affects constant evaluation.

The block-wise ASCII fast path in scalar::utf8::validate_with_counts uses
std::memcpy, which is not usable during constant evaluation. The function
is marked simdutf_constexpr23 and implementation.h dispatches to it from an
'if consteval' branch, so any constant-evaluated call with 16 or more bytes
failed to compile. Guard the fast path with 'if !consteval', exactly as
scalar::utf8::validate() in the same header already does.

Adds constexpr assertions to the boundary tests; they reproduce the failure
without the guard and are exercised by the C++23 leg of CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL