Summary
Two user-facing config reads fall back to the locale encoding and crash with UnicodeDecodeError on Windows machines whose default locale is not UTF-8 (e.g. cp936/GBK):
- fetch_original_config reading the user-supplied original config file (src/diffusers/loaders/single_file_utils.py) — the entry point used by from_single_file when a local YAML config is passed
- the fp16_safetensors conversion command reading model_index.json (src/diffusers/commands/fp16_safetensors.py)
Reproduction
On Windows with a non-UTF-8 system locale, put a non-ASCII comment (e.g. CJK text) in the original config YAML and load a checkpoint with from_single_file(..., original_config=...), or run the fp16_safetensors command on a model folder whose model_index.json contains non-ASCII text. Both fail with UnicodeDecodeError: 'gbk' codec can't decode byte ....
Both files are user-authored/Hub-provided UTF-8 documents, so the reads should pin encoding="utf-8" instead of relying on the locale default.
I have a fix in PR #14819.
Reactions are currently unavailable
Summary
Two user-facing config reads fall back to the locale encoding and crash with UnicodeDecodeError on Windows machines whose default locale is not UTF-8 (e.g. cp936/GBK):
Reproduction
On Windows with a non-UTF-8 system locale, put a non-ASCII comment (e.g. CJK text) in the original config YAML and load a checkpoint with from_single_file(..., original_config=...), or run the fp16_safetensors command on a model folder whose model_index.json contains non-ASCII text. Both fail with UnicodeDecodeError: 'gbk' codec can't decode byte ....
Both files are user-authored/Hub-provided UTF-8 documents, so the reads should pin encoding="utf-8" instead of relying on the locale default.
I have a fix in PR #14819.