Summary

Reject binary responses in fetch and harden HTML sniffing to avoid UTF-8 boundary panics.

Context

The fetch tool is designed for text-based web content. Binary endpoints (for example archive downloads) can produce unreadable output and confusing failures when treated as text. The initial HTML sniffing logic also sliced by byte index, which could panic on multibyte UTF-8 boundaries.

Changes

• Added binary content-type detection before response body parsing
• Returned a clear, actionable error for binary responses with a suggested curl -fLo command
• Made HTML sniffing UTF-8-safe by selecting a valid character boundary before slicing
• Added unit tests for text allowlist, binary detection, and case-insensitive content types
• Updated the net fetch tool description to explicitly document binary-content behavior and fallback guidance

Key Implementation Details

Binary detection is implemented via is_binary_content_type, which allowlists text-oriented content types (text/*, JSON, XML, JavaScript, YAML, TOML, CSV, HTML, SVG, Markdown, and empty content-type) and treats all others as binary.

Use Cases

• Fetching a .tar.gz URL now fails fast with clear guidance instead of trying to parse binary bytes as text
• Responses containing multibyte UTF-8 characters near the sniffing boundary no longer risk panics
• Regular text endpoints (HTML, JSON, XML) continue to work as expected

Testing

cargo test -p forge_services test_is_binary_content_type -- --nocapture

Links

• Related issues: N/A