FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Document VLLM_USE_DEEP_GEMM workaround for non-FP8 quantization · FujitsuResearch/OneCompression@dc1f782 · GitHub

/ OneCompression Public

Commit dc1f782

Browse files
FKKimura
committed
Document VLLM_USE_DEEP_GEMM workaround for non-FP8 quantization
vLLM unconditionally runs a DeepGEMM (FP8) kernel warmup at engine startup, which fails when the optional `deep_gemm` package is not installed -- even for GPTQ / DBF / Mixed-GPTQ models that do not need FP8 kernels at all. Document the bypass via `VLLM_USE_DEEP_GEMM=0` and `VLLM_DEEP_GEMM_WARMUP=skip` in the vLLM inference guide.
1 parent 27e6129 commit dc1f782

2 files changed

Lines changed: 25 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,7 @@
146146
- Added `docs/api/quantizers/onebit.md` (OneBit API reference)
147147
- Updated `mkdocs.yml` nav: added AutoBit/JointQ algorithm pages, OneBit API page; renamed Post-Process nav title to include Block-wise PTQ
148148
- Added example script links to `docs/user-guide/pre-process.md`
149+
- Added a Troubleshooting section to `docs/user-guide/vllm-inference.md` describing how to bypass the unconditional DeepGEMM (FP8) kernel warmup for non-FP8 quantization (GPTQ / DBF / Mixed-GPTQ) by setting `VLLM_USE_DEEP_GEMM=0` and `VLLM_DEEP_GEMM_WARMUP=skip`
149150

150151
### Tests
151152

‎docs/user-guide/vllm-inference.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,3 +219,27 @@ Select the model from the dropdown at the top of the chat screen and start a con
219219
| Variable | Default | Description |
220220
|----------|---------|-------------|
221221
| `ONECOMP_DBF_NAIVE_LINEAR` | `0` | Set to `1` to force the naive (non-GemLite) kernel for DBF inference. Useful for debugging or when GemLite is unavailable. |
222+
223+
## Troubleshooting
224+
225+
### `RuntimeError: DeepGEMM backend is not available or outdated`
226+
227+
vLLM unconditionally runs a DeepGEMM (FP8) kernel warmup at engine startup, even for non-FP8 quantization such as GPTQ, DBF, or Mixed-GPTQ. When the optional [`deep_gemm`](https://github.com/deepseek-ai/DeepGEMM) package is not installed, the warmup fails with:
228+
229+
```
230+
RuntimeError: DeepGEMM backend is not available or outdated. Please install or update the `deep_gemm` to a newer version to enable FP8 kernels.
231+
```
232+
233+
OneComp-quantized models do not require DeepGEMM. Disable the FP8 kernel path before launching vLLM:
234+
235+
```bash
236+
export VLLM_USE_DEEP_GEMM=0
237+
export VLLM_DEEP_GEMM_WARMUP=skip
238+
239+
# Then launch vllm as usual
240+
vllm serve ./your-quantized-model
241+
# or
242+
python your_vllm_script.py
243+
```
244+
245+
Both variables are read directly by vLLM; OneComp does not interpret them.

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL