* spacemit: add mtmd media refactor

* ci: update spacemit mtmd workflow branches

* ci: inline spacemit ohos build commands

* refactor: rename smt mtmd tools directory

* refactor: rename smt mtmd server build option

* refactor: share SMT media wrapper helpers

* feat(spacemit): add Qwen3-TTS llama runtime support

- allow Qwen3 embedding-only contexts to skip the language model head
- support compact context padding for the TTS talker and code predictor
- reject malformed context padding overrides deterministically

Assisted-by: OpenAI Codex

* feat(server): add Qwen3-TTS SMT speech backend

- run text embedding, talker, code predictor, and codec decoding in process
- expose WAV synthesis through the OpenAI-compatible audio speech endpoint
- support request hotwords and startup-selected speaker reference binaries

Assisted-by: OpenAI Codex

* perf(spacemit): accelerate single-row quantized GEMM

- distribute Q4_0 and Q8_0 single-row output tiles across worker threads

Assisted-by: OpenAI Codex

---------

Co-authored-by: muggle-stack <muggle-stack@gmail.com>

Rescale HP activation relative scales and block scale inversely to preserve the final product while reducing fp16 range pressure in the Q4_0 fast path.

- ggml-cpu.c: bypass threadpool on RISCV64_SPACEMIT, run whole graph
  inside AI cores via spacemit_spert_launch_graph_kernel; redirect
  ggml_barrier to spert grid sync
- spacemit_spert_adapter.{h,cpp}: C/C++ boundary for spine-runtime ABI
  (stream require/dispatch/release, grid sync via spine_ctx_sync C ABI,
  per-tile TCM buffer)
- ime.cpp/ime.h: use spert shared buffer and grid sync
- CMakeLists.txt: link spine-runtime (SPERT_DIR)
- .github: fetch spine-runtime (spert) release for glibc and OHOS builds,
  pass SPERT_DIR to cmake

* feat(server): integrate isolated multi-ASR service

* fix(server): guard SMT ASR fields for non-SMT builds

* feat(asr): optimize continuous batching pipeline

Implement dynamic mixed embedding-prefill and token-decode batching for multi-ASR, with FIFO admission and bounded decoder slots. Keep audio encoding serialized while allowing encoder and decoder execution to overlap, and preserve per-sequence logits and sampling across batched steps.

* chore: bump version to 0.1.9

The decoder built its context from zeroed common_params, so the server's
-t/-tb never reached it: n_threads fell back to GGML_DEFAULT_N_THREADS
(4) and the spert whole-graph path checked out only 4 tiles from the
preferred core set, leaving the remaining requested cores idle.

Co-authored-by: Claude Code <noreply@anthropic.com>

* feat: scaffold ggml-spacemit backend structure

- Add GGML_SPACEMIT option to ggml/CMakeLists.txt
- Register Spacemit backend in ggml-backend-reg.cpp
- Add ggml-spacemit.h public API header
- Create ggml-spacemit/ directory with CMakeLists.txt
- Copy kernel files from ggml-cpu/spacemit/ (independent copy,
  not a reference) to make this backend self-contained

Backend implementation (ggml-spacemit.cpp, session, opnode, fusion)
will be added in subsequent commits.

* feat: implement ggml-spacemit backend

Add the main backend implementation following the ggml-hexagon pattern:

- spacemit-opnode.h: opnode struct with fusion support (mirrors htp-opnode)
- spacemit-session.h: session struct (hardware metadata + graph cache,
  no spert::Stream -- created per graph_compute)
- spacemit-kernels.h: kernel dispatch declarations
- ggml-spacemit.cpp: full backend implementation
  * Buffer management reusing spine_mem_pool (alloc/free)
  * Buffer type interface (alignment 64, is_host=true)
  * Op fusion: 4 patterns (RMS_NORM+MUL, MUL_MAT+ADD, QKV merge, FFN merge)
  * graph_compute: creates spert::Stream per forward (RAII),
    guarded by #if __has_include(<spert.hpp>)
  * supports_op: MUL_MAT, MUL_MAT_ID, RMS_NORM, ADD, UNARY, ROPE, SOFT_MAX,
    view ops
  * Device interface and registry following hexagon pattern

Assisted-by: Hermes Agent

* feat: add spine-runtime as submodule, make it a hard dependency

- Add thirdparty/spine-runtime as a git submodule
- spine-runtime CMakeLists.txt is included via add_subdirectory
- spert.hpp is now #included directly (no #if __has_include guard)
- CMakeLists.txt: include FindSMTIME.cmake for IME1/IME2 detection
- CMakeLists.txt: pass full -march flags to both spert and ggml-spacemit
- Verified: cross-compiles for riscv64 with spacemit toolchain
  (libggml-spacemit.a + libspert.so.0.6.0 produced)

Assisted-by: Hermes Agent

* fix: supports_op returns false for compute ops in Phase 1

Kernel dispatch is not yet implemented (Phase 2). Claiming only view ops
(NONE/RESHAPE/VIEW/PERMUTE/TRANSPOSE) lets the scheduler route compute
ops to CPU, avoiding segfaults from uninitialized tensor data.

Also fix CMakeLists.txt to inline IME detection instead of depending
on GGML_CPU_RISCV64_SPACEMIT flag.

Verified on K3 (k3-dev-006):
  Qwen3-0.6B Q8_0: pp128=80.94 t/s, tg32=11.30 t/s
  Qwen3-1.7B Q4_0: pp128=43.81 t/s, tg32=10.24 t/s

Assisted-by: Hermes Agent

* feat: graph_compute launches entire graph via spert::Stream

Replace per-op TODO stub with a single stream.launch that processes all
nodes in the graph as one SPMD kernel:

- Create spert::Stream per graph_compute (RAII)
- Copy node pointers to a flat array for CC core access
- Each CC core (ith/nth from program_id/grid_dim) iterates all nodes
- Dispatch MUL_MAT via tensor_traits->compute_forward, fallback to
  ggml_compute_forward_mul_mat
- Dispatch RMS_NORM/ADD via spacemit RVV kernels
- Dispatch ROPE/UNARY/SOFT_MAX via ggml-cpu generic ops
- ctx->sync() grid barrier between ops preserves graph dependencies
- fut.sync() waits for completion before returning

Currently supports_op only claims view ops (NONE/RESHAPE/VIEW/PERMUTE/
TRANSPOSE) to avoid segfaults when compute kernels run on CC cores.
The host-side ggml_compute_forward_* functions may not be safe to call
from CC core context (threadpool/thread-local dependencies). This will
be addressed in Phase 2 by writing CC-core-native kernel wrappers.

Verified on K3: pp64=90.86 t/s (Qwen3-0.6B Q8_0)

Assisted-by: Hermes Agent

* refactor: replace ime_env.cpp with spert::backend_info()

- Remove ime_env.cpp from CMakeLists.txt (no longer compiled)
- Provide global_spine_env_info in ggml-spacemit.cpp, initialized from:
  * spert::backend_info() for arch_id, vlen, num_cores (fallback)
  * GGML_SPACEMIT_WORKERS env var for num_cores (takes priority)
  * SPACEMIT_MEM_BACKEND / SPACEMIT_DISABLE_TCM env vars
- Force GGML_CPU_RISCV64_SPACEMIT=OFF when GGML_SPACEMIT=ON
- spine_core_info::get_spine_core_info returns empty (no /proc/cpuinfo)

Verified on K3:
  GGML_SPACEMIT_WORKERS=4: pp64=93.73 t/s
  default (8 cores): pp128=95.13 t/s, tg32=25.72 t/s

Assisted-by: Hermes Agent

* feat: migrate full compute pipeline from ggml-cpu/spacemit

- graph_compute uses ggml_graph_plan + ggml_graph_compute (standard
  ggml-cpu threadpool pipeline) instead of spert::Stream::launch
- init_tensor calls ggml_riscv64_spacemit_get_optimal_repack_type()
  to set tensor->extra with proper tensor_traits
- set_tensor calls ggml_riscv64_spacemit_repack_tensor() for data repack
- buffer_type reuses ggml_backend_cpu_riscv64_spacemit_buffer_type()
  from ime.cpp so extra_buffer_type::get_tensor_traits() can match
- Added get_tensor implementation (was nullptr, caused segfault)
- Added fallback memcpy in set_tensor when no traits
- Define GGML_USE_CPU_RISCV64_SPACEMIT on ggml-cpu so ggml_graph_compute
  uses spacemit thread launch path
- Disable TCM in spert mode (no perfer_core_ids)
- Export extern C functions with visibility(default)
- supports_op claims MUL_MAT, RMS_NORM, ADD, ROPE, SOFT_MAX, UNARY, etc.

Verified on K3 (Qwen3-0.6B Q4_0, -t 8):
  pp128 = 265.96 t/s, tg128 = 6.63 t/s

Assisted-by: Hermes Agent

* feat: graph_compute uses spert::Stream::launch for CC core compute

Replace ggml_graph_compute (CPU threadpool) with spert::Stream::launch:
- Each CC core gets ith/nth from ctx->program_id/grid_dim
- ctx->sync() grid barrier between ops (replaces ggml_barrier)
- threadpool=nullptr: ggml_barrier calls made conditional in ime.cpp
- tcm_buffer=nullptr: forward_mul_mat falls through to Path C (no TCM)
- Dispatch: ggml_cpu_extra_compute_forward for MUL_MAT (tensor_traits),
  then manual switch for ADD/ROPE/SOFT_MAX/UNARY/CONCAT/GET_ROWS/etc

Verified on K3 (Qwen3-0.6B Q4_0):
  pp128 = 108.86 t/s, tg128 = 2.78 t/s

Assisted-by: Hermes Agent

* feat: use spert ctx->alloc_shared for TCM, ctx->sync for in-kernel barrier

- Allocate TCM from spert shared memory via ctx->alloc_shared() in
  the launch lambda, stored into tls_context.tcm_buffer
- forward_mul_mat now takes Path A/B (TCM) instead of Path C
- Phase 1->2 barrier: when threadpool==nullptr, use ctx->sync()
  via tls_context.spert_ctx (stored spert::Context pointer)
- Device get_memory reports host memory so scheduler can assign
  tensors to SPACEMIT backend
- Added TCM buffer accessor wrappers (set/get) and spert_ctx setter
  with visibility(default) for cross-TU access
- Added #include <spert.hpp> to ime.cpp for Context type

Verified on K3 (-sm none, Qwen3-0.6B Q4_0):
  TCM=393216 bytes, Path A/B, pp128=107.80, tg128=2.81

Assisted-by: Hermes Agent

* feat: independent SPACEMIT buffer type, no longer needs -sm none

- SPACEMIT device now has its own buffer type (separate from CPU's
  ggml_backend_cpu_riscv64_spacemit_buffer_type) with its own
  extra_buffer_type context
- ggml_buft_is_spacemit() helper matches both CPU_RISCV64_SPACEMIT
  and SPACEMIT buft names so get_tensor_traits works for both
- get_alloc_size delegates to CPU spacemit buft for repacked size
- Added get_tensor to SPACEMIT buffer interface
- Factory function ggml_spacemit_create_extra_buffer_type() to avoid
  cross-TU class visibility issues
- No longer requires -sm none; scheduler auto-assigns to SPACEMIT

Verified on K3 (default scheduler, Qwen3-0.6B Q4_0):
  pp128=113.69, tg128=2.82

Assisted-by: Hermes Agent

* refactor: remove ggml-cpu fallback, use SPACEMIT own dispatch

- ggml_spacemit_compute_forward: directly calls SPACEMIT's own
  extra_buffer_type::get_tensor_traits->compute_forward
- No ggml_cpu_extra_compute_forward, no ggml_compute_forward_* fallback
- graph_compute launch lambda: single ggml_spacemit_compute_forward call
- extra_buffer_type: public inheritance + public access specifier
- Removed unused ops.h/binary-ops.h includes from ggml-spacemit.cpp

Assisted-by: Hermes Agent

* 1

* refactor spacemit backend ops

* feat(spacemit): keep Qwen3 decode graph on CC cores

Add ROPE, SET_ROWS, and SWIGLU kernels, fix FLASH attention VLEN capability detection, reuse the spert stream, and preserve cross-core synchronization correctness.

* fix(spacemit): serialize the shared persistent stream

Protect stream creation and launch with the device-session mutex so multiple backend handles share the single hardware stream without a C++ data race.

* fix(spacemit): release stream resources after graph compute

Keep spert::Stream scoped to graph_compute because its lifetime owns hardware resources. Retain only a host mutex in the device session to serialize exclusive hardware use across backend handles.

* refactor(spacemit): reuse session compute state

Address MR review by simplifying architecture detection, using ggml logging, growing a session-owned workspace only when needed, and reusing per-core SPACEMIT contexts.

* Agent/ggml spacemit backend update (#36)

* ggml-spacemit: fix --no-mmap correctness by returning false from buffer_type_is_host

The buffer_type is_host callback returned true, which made the model
loader treat the SPACEMIT weight buffer as plain host memory. In the
--no-mmap path (llama_model_loader::load_all_data), that shortcut reads
tensor data directly into cur->data via file->read_raw, bypassing
ggml_backend_tensor_set and therefore the buffer's set_tensor callback.

Since SPACEMIT weights are physically repacked inside set_tensor
(ggml_riscv64_spacemit_repack_tensor), skipping it left the weights in
the original GGUF layout while the IME/RVV kernels expect the repacked
layout, producing garbage output with --no-mmap.

Returning false routes the --no-mmap load through ggml_backend_tensor_set
so repack runs, matching the mmap path. Verified on K3/A100 with
Qwen3-0.6B-Q4_0 --no-mmap: output is now coherent.

* ggml-spacemit: add SILU and SOFT_MAX ops to eliminate CPU fallback

* ggml-spacemit: extend F16 support for CONCAT/GET_ROWS, add CPY type-convert, ROPE freq_factors

* ggml-spacemit: add F16xF32 MUL_MAT RVV kernel for dense weight matmul

* ggml-spacemit: add general CONT kernel for F32/F16 non-contiguous tensors

* ggml-spacemit: add YaRN ROPE support (ext_factor != 0)

* ggml-spacemit: add IMROPE/MROPE support in forward_rope_impl

* ggml-spacemit: add scalar kernels for missing Qwen3.5 ops (MUL_MAT F32, L2_NORM, PAD, TRI, etc.)

* ggml-spacemit: implement GATED_DELTA_NET and SSM_CONV kernels (fix Qwen3.5 garbled output)

* ggml-spacemit: migrate fallback operators and use worker shared buffer

* ggml: add trace profiling support

* ggml-spacemit: optimize rope fallback with RVV

* ggml-spacemit: use row copies for contiguous fallback

* ggml-spacemit: use legacy rope path for small F32 shapes

* ggml-spacemit: trim masked flash attention tiles

* ggml-spacemit: restore fused GDN RVV path

* build(spacemit): use prebuilt spine-runtime

Assisted-by: Codex

* args: refactor mlock/mmap/directio into load-mode (#20834)

---------

Co-authored-by: cailinxi <cailinxi@spacemit.com>

* ggml-spacemit: type-gate direct GEMV to q4_0, fuse GDN decode row update (#37)

MUL_MAT direct path (IME reads B straight from DRAM, decode only):
restrict to q4_0 and drop the q8_0 arm together with its
gemm_n <= 2*NB_COLS*64 crossover. Paired A/B on K3 shows q8_0 has no
winning shape: Qwen3.5-35B dense q8_0 decode GEMVs all fall under the
old crossover and still measure 21.8-26.7 GB/s direct vs 28.9-30.8
staged through TCM (tg128 6.71 -> 6.91 with direct disabled), while
q4_0 gains from direct (Qwen3-0.6B tg128 57.45 -> 59.05, +2.8%; pp
unchanged). 35B carries no physical q4_0 weights after repack, so the
gate leaves it on the staged path entirely.

GDN decode: add a fused single-pass row update used for n_tokens <= 4.
State rows are independent within a token, so the reference's memcpy +
decay/dots pass + update pass (3 reads, 3 writes of the state per head)
collapse into one load/store pair per row, with k/q/decay held in
vector registers across the row loop; decode reads the input state and
writes the output state directly with no intermediate copy. Kernel time
at n_tokens == 1 drops 424.6 -> 216.0 us; paired A/B tg128 7.58 -> 7.93
(+4.6%), pp128 unchanged. Chunks with more tokens keep the reference
two-pass form: their state working copy is L2-resident so the extra
passes already run at the bandwidth floor, and the fused form loses
there to register spills and the store-at-row-end reduction dependency
stalling row loads on the in-order cores (measured crossover ~8
tokens; chunks are 64, or 16 with kda).

Co-authored-by: Claude Code <noreply@anthropic.com>

* ggml-spacemit: bind stream cores via SPACEMIT_PERFER_CORE_ID

Port the core-pinning behavior of the old ggml-cpu/spacemit spert adapter:
spine_env_info parses SPACEMIT_PERFER_CORE_ID (comma-separated physical
CC-core ids) into perfer_core_ids and caps num_cores by the pool size;
graph_compute then creates the spert Stream with
StreamConfig{n_cores, core_ids} so decoder tiles are pinned to the
requested CC cores instead of the default all-core grant, which collided
with the ONNX encoder EP cores.

Verified on K3 board (multi-asr 4+4: enc 8-11 / dec 12-15): during
decode only cores 12-15 are busy and 8-11 stay idle.

Co-Authored-By: Claude Code <noreply@anthropic.com>

---------

Co-authored-by: Alex <jinghui.huang@spacemit.com>
Co-authored-by: cailinxi <cailinxi@spacemit.com>
Co-authored-by: Claude Code <noreply@anthropic.com>

- CI: replace -DGGML_CPU_RISCV64_SPACEMIT=ON with -DGGML_SPACEMIT=ON (both glibc and OHOS jobs)
- deps: bump spine-runtime (spert) to 0.6.3 (glibc and OHOS packages)
- fix: qwen3-tts talker silently fell back to its scalar kernels on the OHOS toolchain because
  it never received -march/-mabi (ggml-cpu's ARCH_FLAGS and ggml-spacemit's SPACEMIT_MARCH are
  both target-PRIVATE and do not propagate, and the OHOS toolchain leaves -march/-mabi unset)
- docs: document GGML_SPACEMIT=ON and the SPERT_DIR requirement in the RISC-V build docs
- VERSION_NUMBER: 0.1.9 -> 0.2.0