This repository hosts the Hidden Decoding paper, public demonstration checkpoints, and inference patch.

• 2026-07-07: Added the paper PDF, frontier-scale WeLM-HD4-80B/617B results, and the Stream-Factorized Attention framing.
• 2026-04-01: Released Hidden-Decoding-8B-n8-Instruct, our first instruction-tuned demonstration model.
• 2026-03-10: Initial release of base models (n=2, n=4, n=8) and SGLang inference patch.

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

• PDF
• Blog post

The paper studies whether an already-strong LLM can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed. Hidden Decoding expands each token into multiple hidden streams, supervises only the final stream, and retains the intermediate streams' KV cache so their latent computation remains available to later tokens.

Hidden Decoding scales computation along the sequence-length dimension:

1. Each input token is expanded into n streams with independent embedding tables.
2. The expanded length-nL sequence is processed by the same Transformer backbone.
3. Only the final stream predicts the next token.
4. Intermediate streams act as latent computation states and keep their own KV cache.

This differs from recurrent-depth or looped Transformers: the extra computation is a longer sequence in a single forward pass, which fits the pipeline-parallel training stack used for large MoE models.

To make the expanded sequence trainable at scale, the paper introduces Stream-Factorized Attention. Most layers attend only within each stream, and only a subset of layers mix information across streams. This reduces the attention growth from quadratic in n to roughly linear in n.

We train WeLM-HD4-80B and WeLM-HD4-617B with n=4 during continued pretraining. The matched autoregressive and Hidden Decoding models use the same early SFT-only post-training recipe, with no reinforcement learning. Active Transformer parameters per token stay unchanged: 3B for 80B and 23B for 617B.

The 4x expanded sequence costs 5.1x per batch on WeLM-HD4-80B and 4.4x per batch on WeLM-HD4-617B, close to the 4x linear reference and far below the 16x dense-attention baseline.

Increasing the expansion factor improves language modeling loss and downstream accuracy while keeping the Transformer backbone fixed.

The released HuggingFace checkpoints are not the main models of the paper. They are Qwen3-8B based demonstration models provided to show that Hidden Decoding can scale with the expansion factor n: as n increases, the same Transformer backbone receives more latent computation per token and generally improves. The main paper results are the WeLM-HD4-80B and WeLM-HD4-617B studies above.

All released base models share the same 8B Transformer backbone. They differ by the Hidden Decoding expansion factor n.

These released 8B models are included as a public, reproducible demonstration of expansion-factor scaling. They are not intended to be the paper's main model release.

Evaluated on Qwen3-8B-Base with progressive Hidden Decoding scaling:

Instruction-tuned from Hidden-Decoding-8B-n8 and compared with Qwen3-8B-Instruct plus a matched Qwen3-8B SFT baseline trained on the same data:

Sampling: temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5, max_tokens=4096. Judge: GPT-4o.

docker pull aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126

git clone https://github.com/exlaw/sglang.git
cd sglang
pip install -e "python[all]"

Apply the patch on top of SGLang:

git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout 4efe844a2
git apply /path/to/hidden_decoding.patch
pip install -e "python[all]"

Launch the server:

python -m sglang.launch_server \
    --model-path tencent/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128

docker run --gpus all -p 8080:8080 -p 8081:8081 \
  -v /path/to/models:/models \
  aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126 \
  bash -c "python -m sglang.launch_server \
    --model-path /models/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128"

Hidden Decoding models process n times longer sequences internally, so --chunked-prefill-size -1, --attention-backend fa3, and conservative batch sizes are important for stability and performance. Adjust --tp-size for multi-GPU setups.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain hidden decoding in simple terms."},
    ],
    max_tokens=512,
    temperature=0.7,
)
print(response.choices[0].message.content)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
response = client.completions.create(
    model="default",
    prompt="The meaning of life is",
    max_tokens=128,
    temperature=0,
)
print(response.choices[0].text)

We also provide a lightweight, zero-dependency web chat interface for interactive testing:

python chat_ui.py

Then open http://localhost:8081 in your browser. The Chat UI connects to http://localhost:8080/v1 by default.

docker run --gpus all -p 8080:8080 -p 8081:8081 \
  -v /path/to/models:/models \
  -v /path/to/chat_ui.py:/app/chat_ui.py \
  aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126 \
  bash -c "python -m sglang.launch_server \
    --model-path /models/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128 &
  sleep 5 && python /app/chat_ui.py"

Features:

• Streaming responses with real-time token speed display
• Thinking/reasoning block visualization
• Configurable system prompt, temperature, and max tokens
• Pure Python standard library implementation

The patch adds the qwen3_scale_seq model architecture and modifies the scheduler, batch manager, and CUDA graph runner to handle expanded sequence lengths.

If you find this work useful, please cite:

@article{hidden_decoding_at_scale_2026,
  title  = {Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models},
  author = {WeChat AI Team},
  year   = {2026},
  url    = {https://github.com/Tencent/Sequential-Hidden-Decoding/blob/main/paper/hidden_decoding_at_scale.pdf}
}

Sijun Zhang (nepheloturbulence@gmail.com), Aiwei Liu (liuaiwei20@gmail.com)

This project is released under the License Terms of Hidden-Decoding. The dependent open-source models and software components remain licensed under their respective original licenses. See LICENSE for details.