FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add C++20 char8_t string accessors to the On-Demand API (#2731) by lemire · Pull Request #2843 · simdjson/simdjson · GitHub

Add C++20 char8_t string accessors to the On-Demand API (#2731) - #2843

Open
lemire wants to merge 3 commits into
masterfrom
u8string-ondemand
Open

Add C++20 char8_t string accessors to the On-Demand API (#2731)#2843
lemire wants to merge 3 commits into
masterfrom
u8string-ondemand

Conversation

lemire commented Aug 24, 2026

Copy link
Copy Markdown
Member

Short title (summary): Add C++20 char8_t string accessors to the On-Demand API

Description

PR #2786 added get_u8string() to dom::element. This PR extends the same idea to the On-Demand API, and makes std::u8string_view and std::u8string first-class types in get<T>() on both APIs. Closes #2731.

Every string simdjson produces is valid UTF-8, so all of the new accessors are zero-copy aliases of their char-based counterparts: the same bytes, at the same address, no conversion.

New accessors

char accessor char8_t accessor On
get_string() get_u8string() ondemand::value, ondemand::document, ondemand::document_reference (+ simdjson_result<> forwarders)
field.unescaped_key() field.unescaped_u8key() ondemand::field (+ forwarder)
field.escaped_key() field.escaped_u8key() ondemand::field (+ forwarder)

Type-based access

  • get<std::u8string_view>() on ondemand::value, ondemand::document, ondemand::document_reference and dom::element — the DOM specialization also brings dom::element::get(std::u8string_view&) and is<std::u8string_view>() along for free.
  • get<std::u8string>(), plus containers and optionals of it (std::vector<std::u8string>, std::optional<std::u8string>, …), through the C++20 deserialization path. This is a new concepts::constructible_from_u8string_view concept and a matching tag_invoke overload; std::u8string cannot use the existing constructible_from_string_view path because the character types differ.
  • The templated receiver overloads get_string(receiver) and unescaped_key(receiver) now accept a std::u8string or std::u8string_view receiver. Previously they did not compile for those types, because the implementation did a plain receiver = content; from a char-based std::string_view. That single assignment is now routed through an internal::assign_utf8() helper whose general (unconstrained) overload is the old code verbatim, so no existing receiver type changes behavior.

Notes

  • A new SIMDJSON_SUPPORTS_CHAR8_T macro in compiler_check.h replaces the raw #if defined(__cpp_char8_t) && __cpp_char8_t >= 201811L feature test that PR Add C++20 u8 variants of string access methods (#2731) #2786 repeated at every call site. It follows the style of the neighbouring SIMDJSON_SUPPORTS_CONCEPTS / SIMDJSON_SUPPORTS_DESERIALIZATION macros, and is user-overridable.
  • Everything is purely additive and guarded, so C++11/14/17 builds are byte-for-byte unaffected.
  • get_wobbly_string() deliberately has no u8 variant. Wobbly strings may be invalid UTF-8 (WTF-8), so they have no business in a std::u8string_view.
  • No serialization/builder changes: this issue is about string access.

Type of change

  • Bug fix
  • Optimization
  • New feature
  • Refactor / cleanup
  • Documentation / tests
  • Other

How to verify / test

  • New test target tests/ondemand/ondemand_u8string_tests.cpp (17 tests): value/document/document_reference/field accessors, the allow_replacement flag, INCORRECT_TYPE and error forwarding, iterate_many, get<T> for both u8 types, containers and optionals, u8 receivers, a byte-for-byte check that the u8 view matches get_string(), and a regression check that the char-based paths still behave.
  • tests/dom/basictests.cpp gains u8string_value_get_template() covering dom::element::get<std::u8string_view>(), get(T&), is<T>() and the error path.
  • Full suite passes under both standards on Apple clang 17 (arm64): 121/121 with SIMDJSON_CXX_STANDARD=20 and 121/121 with SIMDJSON_CXX_STANDARD=17 (where the whole feature compiles out).
  • Also compiled the library and a driver under -std=c++11, -std=c++14, -std=c++17 and with -DSIMDJSON_EXCEPTIONS=0.

Checklist before submitting

  • I added/updated tests covering my change
  • Code builds locally and passes my check
  • Documentation / README updated if needed (new u8string_view and u8string (C++20) section in doc/basics.md, note in doc/dom.md)
  • Commits are atomic and messages are clear
  • I linked the related issue

https://claude.ai/code/session_01EKoa1vNn8ZKwJH4qaZTU2M

lemire force-pushed the u8string-ondemand branch from 8d817ec to 2ed4c7b Compare August 25, 2026 20:35
lemire added 3 commits August 26, 2026 13:35
PR #2786 added dom::element::get_u8string(). This extends the same idea to
the On-Demand API and makes std::u8string_view / std::u8string first-class
types in get<T>().

New accessors (all zero-copy aliases of their char-based counterparts, since
every string simdjson produces is valid UTF-8):

  * ondemand::value::get_u8string(bool allow_replacement = false)
  * ondemand::document / document_reference::get_u8string(...)
  * ondemand::field::unescaped_u8key(...) and escaped_u8key()
  * the matching simdjson_result<> forwarders

Type-based access:

  * get<std::u8string_view>() on ondemand::value, ondemand::document,
    ondemand::document_reference and dom::element (which also gives
    dom::element::get(std::u8string_view&) and is<std::u8string_view>())
  * get<std::u8string>() and containers/optionals of it through the C++20
    deserialization path (new concepts::constructible_from_u8string_view)
  * get_string(receiver) and unescaped_key(receiver) now accept a
    std::u8string or std::u8string_view receiver

The new SIMDJSON_SUPPORTS_CHAR8_T macro replaces the raw __cpp_char8_t
feature test that PR #2786 repeated at each call site. Nothing is added
when char8_t is unavailable, so C++11/14/17 builds are unaffected.

get_wobbly_string() deliberately has no u8 variant: wobbly strings may be
invalid UTF-8 (WTF-8), so they do not belong in a std::u8string_view.
Claude-Session: https://claude.ai/code/session_01EKoa1vNn8ZKwJH4qaZTU2M
Under -DSIMDJSON_STATIC_REFLECTION_MODE=ON, std::u8string is a class type
with no reflectable members, so user_defined_type<T> accepted it and the
reflected-struct tag_invoke overload competed with the new u8 string
overload. Every tag_invoke call on std::u8string was therefore ambiguous,
custom_deserializable<std::u8string> was false, and get<std::u8string>()
(and any container of it) failed the static_assert in value.h.

std::string and std::string_view were already excluded by name for the
same reason; this adds their char8_t counterparts, guarded by
SIMDJSON_SUPPORTS_CHAR8_T so nothing changes for C++17 and earlier.

The gcc 16 job builds with static reflection on, which is why this only
showed up there: ondemand_u8string_tests passes in every other
configuration.

Claude-Session: https://claude.ai/code/session_01K1fw1eGejVLMVvHjAFD8CU
lemire force-pushed the u8string-ondemand branch from 2ed4c7b to fd95241 Compare August 26, 2026 17:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add u8 variant of string access methods

1 participant


Back | FazBrowse Home | New Git URL