make detect encoding functions constexpr by shikharish · Pull Request #920 · simdutf/simdutf · GitHub

Closes #891

Signed-off-by: Shikhar <shikharish05@gmail.com>

@shikharish thanks. This will nicely close our gap. Let us include this in our next release.

Thanks for the PR! I am in a hurry so sorry for not doing a thorough review.
I think it is mostly fine but I have some questions.

let's use a macro guarded c++20 concept here to restrict the bytepr.

should be really_inline

why is this included twice?

it is ok to move around the includes, but remove those that are duplicates

why is this needed?

why was this changed?

please add a macro guarded c++20 concept check.

(and reused the already existing concept and style)

Signed-off-by: Shikhar <shikharish05@gmail.com>

@shikharish Could you have a look at @pauldreik's comments ? There are a few that are apparently outstanding (meaning that it is unclear whether you answered or changed the code accordingly).

@lemire I made all the requested changes. There were a few things I misunderstood before but I've fixed them in the last few commits.

@pauldreik Can you check that your concerns were addressed?

please look at the other constexpr changes recently merged and reuse the already existing concepts.

(and reused the already existing concept and style)

Signed-off-by: Shikhar <shikharish05@gmail.com>

@pauldreik Please review.

@shikharish when you get review comments, please go through them one by one and fix/answer them. otherwise I have to go through everything again. there are still issues left from earlier comments.

what is also left to do:

update the README which says these two functions miss constexpr functionality.
a function template is already implicitly inline, don't mark as inline.
the changes in base64, why are they there? surprised to see them in this PR.

these don't seem to be used.

template <endianness big_endian, typename InputPtr>
#if SIMDUTF_CPLUSPLUS20
  requires simdutf::detail::indexes_into_utf16<InputPtr>
#endif
simdutf_warn_unused simdutf_constexpr23 bool

this should be a simdutf really inline

Signed-off-by: Shikhar <shikharish05@gmail.com>

@shikharish Could you have a look at the comments ?

@lemire The last commit addressed all the previous comments.

@shikharish Ok, can you access the web interface at #920 and look at @pauldreik's comments. For each one of them, can you reply to Paul. It can be as simple as 'ok, I did it in my last commit', and so forth.

Otherwise, we have to go through all the comments and cross-reference them with you code changes and make sure that they have been addressed. It should be far easier for you to check each comment and confirm that it was addressed, with explanations where necessary.

Thanks!!!

make detect encoding functions constexpr

83388fb

Signed-off-by: Shikhar <shikharish05@gmail.com>

shikharish force-pushed the encoding-constexpr branch from 187b3c6 to 83388fb Compare January 13, 2026 10:47

minor fixes

0a0d74c

Signed-off-by: Shikhar <shikharish05@gmail.com>

pauldreik reviewed Jan 14, 2026

View reviewed changes

minor changes

d18b2ae

Signed-off-by: Shikhar <shikharish05@gmail.com>

shikharish requested review from lemire and pauldreik January 15, 2026 12:02

revert c++23 condition check

b831db8

Signed-off-by: Shikhar <shikharish05@gmail.com>

shikharish force-pushed the encoding-constexpr branch from 2b17857 to b831db8 Compare January 15, 2026 16:37

pauldreik requested changes Jan 16, 2026

View reviewed changes

improvements

46290ea

Signed-off-by: Shikhar <shikharish05@gmail.com>

shikharish requested a review from pauldreik January 21, 2026 11:39

pauldreik requested changes Feb 1, 2026

View reviewed changes

minor changes

ca05f19

Signed-off-by: Shikhar <shikharish05@gmail.com>

-              simdutf_warn_unused encoding_type check_bom(const uint8_t *byte, size_t length);
-              simdutf_warn_unused encoding_type check_bom(const char *byte, size_t length);
+              template <typename BytePtr>
+              simdutf_warn_unused inline simdutf_constexpr14 encoding_type

               #include <simdutf/scalar/utf32_to_utf8/utf32_to_utf8.h>
               #include <simdutf/scalar/utf32_to_utf8/valid_utf32_to_utf8.h>
               #include <simdutf/scalar/utf8.h>
+              #include <simdutf/scalar/detect.h>

+              #include <simdutf/scalar/utf8.h>
+              #include <simdutf/scalar/utf16.h>
+              #include <simdutf/scalar/utf32.h>
+              #include <simdutf/scalar/detect.h>

                * of units processed if successful.
                */
-              simdutf_warn_unused result
+              simdutf_warn_unused simdutf_constexpr23 result

+              #if SIMDUTF_CPLUSPLUS20
+                #include <concepts>
+              template <typename BytePtr>
+              concept check_bom_byteptr = requires(BytePtr p, size_t i) {

+                return encoding_type::unspecified;
+              }
+              simdutf_warn_unused inline encoding_type check_bom(const char *byte,

+              #if SIMDUTF_SPAN
+              template <endianness big_endian, simdutf::detail::indexes_into_utf16 InputPtr>
+              #else
+              template <endianness big_endian, typename InputPtr>
+              #endif

               }
+              template <endianness big_endian>
+              inline simdutf_warn_unused simdutf_constexpr23 bool

		#include <iostream>
		#include <array>

make detect encoding functions constexpr by shikharish · Pull Request #920 · simdutf/simdutf · GitHub

Uh oh!

make detect encoding functions constexpr - #920

make detect encoding functions constexpr#920shikharish wants to merge 6 commits intosimdutf:mastersimdutf/simdutf:masterfrom shikharish:encoding-constexprshikharish/simdutf:encoding-constexprCopy head branch name to clipboard

Conversation

shikharish commented Jan 13, 2026

Uh oh!

shikharish commented Jan 13, 2026

Uh oh!

lemire commented Jan 13, 2026

Uh oh!

pauldreik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lemire commented Jan 16, 2026

Uh oh!

shikharish commented Jan 16, 2026

Uh oh!

lemire commented Jan 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shikharish commented Feb 1, 2026

Uh oh!

pauldreik commented Feb 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lemire commented Apr 6, 2026

Uh oh!

shikharish commented Apr 14, 2026

Uh oh!

lemire commented Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

make detect encoding functions constexpr#920
shikharish wants to merge 6 commits into
simdutf:masterfrom
shikharish:encoding-constexpr