FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add ktxTexture2_GetLevelFileInfo and the ktxLevelProcessor API (streaming per-level transcode, part 1) by SashaRX · Pull Request #1251 · KhronosGroup/KTX-Software · GitHub

Add ktxTexture2_GetLevelFileInfo and the ktxLevelProcessor API (streaming per-level transcode, part 1) - #1251

Open
SashaRX wants to merge 6 commits into
KhronosGroup:mainfrom
SashaRX:feat/level-file-info
Open

SashaRX wants to merge 6 commits into
KhronosGroup:mainfrom
SashaRX:feat/level-file-info

Conversation

SashaRX commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

First part of the per-level streaming API discussed in #1224. This draft carries the truncated-stream regression test, the source-side query and the public surface of the level processor, so the header can be reviewed against what was agreed in the thread before the codec work lands. ktxLevelProcessor_ProcessLevel is intentionally a stub here: argument validation per the final contract, then KTX_UNSUPPORTED_FEATURE.

To answer the question in #1224 first: yes, the truncated-stream regression test was successful. A ktxTexture2 constructs from exactly the serialized metadata prefix without KTX_TEXTURE_CREATE_LOAD_IMAGE_DATA_BIT on unmodified main, for BasisLZ, Zstd-supercompressed and unsupercompressed sources, and the constructor never reads past that prefix. No prerequisite fix to the constructor was needed; the test is the first commit.

What is in the three commits

  1. Metadata-only construction tests (test-only). Construction from the prefix through the end of SGD/KVD, as applicable, checked against an instrumented bounded custom ktxStream on all platforms and, on POSIX, a PROT_NONE guard page immediately after the prefix. Also covers the clean failure modes: LOAD bit on a prefix, late ktxTexture2_LoadImageData on a shell, metadata cut one byte into the last section. Files are generated in-process; no new binary resources.

  2. ktxTexture2_GetLevelFileInfo — where a level's stored payload lives in the serialized source: absolute byteOffset, byteLength, and the Level Index uncompressedByteLength. A thin public exposure of the existing rebased level index plus _firstLevelFileOffset. Per the thread, ktxTexture2_levelFileOffset() itself is left private and unchanged; the public query wraps its logic and returns KTX_INVALID_OPERATION once the texture no longer has serialized-source state (created with ktxTexture2_Create, or after image data has been loaded). If you would rather expose ktxTexture2_levelFileOffset() directly, in addition or instead, that is a small change.

  3. ktxLevelProcessor skeleton — the header as drafted in Feature request: transcode individual mip levels from a partially-downloaded KTX2 file (progressive texture streaming) #1224: opaque type, ktxLevelProcessor_CreateBasis, GetOutputVkFormat, the three layout queries (GetLevelSize / GetImageSize / GetImageOffset, level-relative), ProcessLevel, Destroy. CreateBasis validates the source the same way ktxTexture2_TranscodeBasis does, rejects video with KTX_UNSUPPORTED_FEATURE (follow-up, as agreed), builds the private KTX_TEXTURE_CREATE_NO_STORAGE prototype the queries delegate to, and performs the transcoder's process-wide initialization through the same guarded mechanism TranscodeBasis uses — once, never per processor. To make both paths validate sources, resolve targets and initialize identically, three pieces of TranscodeBasis are factored into internal helpers declared in a new basis_transcode.h: the source gate (transcodable color model, SGD presence), the target resolution (alpha content, automatic-selection mapping, VkFormat with transfer function, PVRTC1 pow2 check, transcoder-availability check) and the guarded basisu_transcoder_init() call. Mechanical; no behavior change to TranscodeBasis. CreateBasis requires the source to retain its serialized-source state and rejects a texture whose image data has been loaded (or one created with ktxTexture2_Create) with KTX_INVALID_OPERATION: ProcessLevel consumes the serialized level payload that only such a source can describe, and loading a Zstd/Zlib source rewrites its level index to the inflated sizes. ProcessLevel re-checks that state through GetLevelFileInfo and validates srcSize against the byteLength it reports; the three layout queries report out-of-range arguments uniformly as KTX_INVALID_VALUE. The processor struct lives in a private header, as texture2.h does for ktxTexture2_private, and level_processor.cpp is added to the libktx Doxygen inputs, so the generated reference documents the ktxLevelProcessor class with its functions and nothing internal.

Settled in #1224 and reflected here

  • Caller-provided destination; a complete level (all layers, faces, slices) with no inter-level padding; layout from the queries, not derived from the transcode enum.
  • Opaque, reusable processor: generic type ktxLevelProcessor, codec-specific factory ktxLevelProcessor_CreateBasis. The prototype texture stays private.
  • ProcessLevel takes exactly the bytes GetLevelFileInfo describes, as stored (BasisLZ / raw or Zstd/Zlib UASTC / HDR 6x6 intermediate); srcSize must equal byteLength. No second mode where src is a slice of loaded pData; no partial payloads.
  • ktxTexture2_levelFileOffset() behavior unchanged; the serialized layout is considered discarded once image data has been loaded.
  • Video rejected in this PR, supported in a follow-up before release.
  • Out of this PR: single-level GL/Vulkan upload; JS binding as a small follow-up (sketch in the thread); Java and Python bindings needed before release.

Still open — your call

  1. Processor lifetime. The processor currently borrows the source: it must stay alive and unmodified until ktxLevelProcessor_Destroy (documented on the type and on CreateBasis). This keeps the PR small and avoids duplicating DFD/SGD/level index; bindings retain the source until all processors are gone. Is borrowing acceptable for the C API, or should the processor own a copy of the codec/container metadata from the start?
  2. Sharing with the whole-texture path. Plan for the next commits: a private per-level primitive shared by a serialized public adapter (ProcessLevel) and a loaded-data private adapter used by ktxTexture2_TranscodeBasis, so the public contract stays serialized-only while the codec logic is single-sourced; the UASTC HDR 4x4 → ASTC relabel stays a metadata-only fast path; the existing ETC1S video orchestration in TranscodeBasis is preserved. Does that satisfy "usable and called by TranscodeBasisEx for each level it processes", or would you prefer TranscodeBasis to literally call the public function?

Tests

32 new gtest cases in transcodetests, all generated in-process: the construction/over-read/truncation matrix × 3 source variants; GetLevelFileInfo values byte-for-byte against the serialized index, availability before and invalidation after load, invalid arguments, no-serialized-source; processor creation validation and every layout query cross-checked against the actual output of ktxTexture2_TranscodeBasis on an ETC1S 2D-array source. Full library/tools/tests build clean on Linux; libktx.doc builds without new warnings.

Out of scope here (per the thread)

Batch level ranges, cancellation, caching, single-level GL/Vulkan upload, format-property queries, JS/Java/Python bindings (follow-ups), video (follow-up before release), XUASTC sources (rejected by the same source-family gate as TranscodeBasis).

Pin down the constructor behavior that the per-level streaming work
discussed in issue 1224 relies on: a ktxTexture2 constructs successfully from
exactly the serialized metadata prefix (through the end of SGD/KVD)
without KTX_TEXTURE_CREATE_LOAD_IMAGE_DATA_BIT, for BasisLZ, Zstd and
no-supercompression variants, and the constructor never reads past that
prefix. Over-reads are detected two ways: an instrumented bounded custom
ktxStream on all platforms, and a PROT_NONE guard page on POSIX where an
over-read faults instead of returning an error.

Also covers the failure modes a streaming consumer must be able to rely
on being clean errors rather than crashes: the LOAD bit against a
metadata-only buffer, a later LoadImageData on the shell, metadata cut
one byte into the last section, and a prefix one byte short (which may
legitimately succeed for scheme 0, where only alignment padding is
missing).

Test files are generated in-process with the write API; no new binary
resources.
Public query reporting where a mip level's stored payload lives in the
serialized KTX2 source — absolute byteOffset, byteLength and the Level
Index's uncompressedByteLength — so a streaming consumer can fetch a
level's bytes (e.g. with an HTTP Range request) without parsing the
container itself. First piece of the per-level streaming API discussed
in issue 1224.

The implementation is a thin public exposure of existing state: the
in-memory level index rebased by _firstLevelFileOffset, exactly what the
private ktxTexture2_levelFileOffset() computes. The query is valid only
while the texture retains its serialized-source state; for a texture
created with ktxTexture2_Create, or once image data has been fully
loaded (LOAD bit or LoadImageData), it returns KTX_INVALID_OPERATION,
matching the library's existing lifetime handling which zeroes the base
after a full load. The rebase-to-absolute conversion is guarded against
overflow from a corrupt index with KTX_FILE_DATA_ERROR.

Tests cover: values matching the serialized level index byte-for-byte
across BasisLZ/Zstd/raw variants on a metadata-only shell, availability
on a fully-buffered texture before LoadImageData and invalidation after
it, the LOAD-bit construction path, invalid arguments, and the
no-serialized-source case.
The streaming counterpart of ktxTexture2_TranscodeBasis agreed in issue 1224:
an object created for a Basis-compressed source (typically a
metadata-only texture constructed from the serialized file prefix) and a
chosen target, that will process mip levels one at a time from
caller-provided payload bytes into caller-provided buffers, with layout
queries describing the processed output. This commit lands the public
surface, creation/validation and the layout queries; ProcessLevel
validates its arguments per the final contract and returns
KTX_UNSUPPORTED_FEATURE until the codec paths land in the next commit of
this series.

ktxLevelProcessor_CreateBasis validates the source exactly as
TranscodeBasis does (transcodable color model, SGD presence for
BasisLZ/6x6-intermediate; unknown color models fail the same gate),
rejects video sources with KTX_UNSUPPORTED_FEATURE (agreed follow-up),
and builds a private KTX_TEXTURE_CREATE_NO_STORAGE prototype — the same
prototype technique TranscodeBasis uses, minus the full-mip-chain
allocation. The queries delegate to it: GetLevelSize/GetImageSize are
the existing size calculations, GetImageOffset is rebased to be relative
to the level's destination buffer, GetOutputVkFormat reports the
resolved format.

So the processor validates sources, resolves targets and initializes
identically to TranscodeBasis, three pieces of it are factored out and
shared via the new internal basis_transcode.h: the source gate
(transcodable color model, presence of the supercompression global data
the scheme requires) as ktxTexture2_validateBasisSource, the target
resolution logic (alpha-content detection, automatic-selection mapping,
VkFormat resolution with the source's transfer function, PVRTC1
power-of-two validation and the transcoder-availability check) as
ktxTexture2_resolveBasisTargetFormat, and the guarded process-wide
basisu_transcoder_init() call as ktxInitBasisTranscoder. Creating a
processor performs that initialization once, never per processor. The
colorModel to basis_tex_format mapping both users need is a file-local
helper. No behavior change to TranscodeBasis.

CreateBasis requires the source to retain its serialized-source state:
a texture whose image data has been loaded, or one created with
ktxTexture2_Create, is rejected with KTX_INVALID_OPERATION, because
ProcessLevel consumes the serialized level payload that only such a
source can describe (loading a Zstd/Zlib source even rewrites its
level index to the inflated sizes). ProcessLevel re-checks that state
through ktxTexture2_GetLevelFileInfo, propagating its result, and
validates srcSize against the byteLength it reports. The three layout
queries validate their arguments themselves so an out-of-range level,
layer or faceSlice is KTX_INVALID_VALUE from all of them.

The processor struct lives in a private header (level_processor.h, as
texture2.h does for ktxTexture2_private) and level_processor.cpp is
added to the libktx Doxygen inputs, so the reference documents the
ktxLevelProcessor class with its functions and nothing internal.

Tests verify the resolved format and every layout query against the
actual output of ktxTexture2_TranscodeBasis on the same ETC1S 2D-array
source, plus creation validation and the ProcessLevel
validate-then-report contract.
KHRONOS_STATIC was defined for transcodetests so the always-static
basisu C binding's declarations would not be dllimport. It also empties
KTX_API in ktx.h, so when linking the shared libktx the imported data
constant KTX_ETC1S_DEFAULT_COMPRESSION_LEVEL was referenced without
__imp_ and the executable failed to link (LNK2001, all Windows CI
jobs).

Define KTX_BASISU_API instead. It suppresses dllimport only in
basisu_c_binding.h, leaving ktx.h's KTX_API correct for both the
shared and the static libktx configurations.
EXPECT_EQ compared ktxSupercmpScheme with a raw ktx_uint32_t parsed
from the file header. MSVC flags the mixed signed/unsigned template
instantiation as C4389, fatal under the Windows CI KTX_WERROR=ON build.
Cast the parsed value to the enum so both operands share the type.
SashaRX marked this pull request as ready for review September 21, 2026 21:47

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL