Filter invisible Unicode after HTML entity normalization - #3110
Merged
SamMorrowDrums merged 3 commits intoAug 19, 2026
Merged
Conversation
Copy link
Copy Markdown
Collaborator
Sorry, something went wrong.
FilterInvisibleCharacters previously ran only before FilterHTMLTags, so numeric HTML entities (e.g. ​ or ​) that bluemonday decodes into invisible or bidirectional control characters could survive sanitization untouched. Sanitize now applies the invisible-character filter both before HTML processing (so raw invisible characters don't interfere with code-fence parsing) and again after, so entity-decoded characters cannot escape the policy. Also expands the removal set to include: - ARABIC LETTER MARK (U+061C), a directional format character in the same family as the already-covered LRM/RLM marks. - Variation selectors (U+FE00-U+FE0F) and the variation selectors supplement (U+E0100-U+E01EF), which can be used to hide payloads after emoji or other base characters. Fixes #3101
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Pull request overview
Adds post-normalization invisible-Unicode filtering to prevent HTML entities bypassing sanitization.
Changes:
- Filters invisible characters before and after HTML processing.
- Expands directional-mark and variation-selector coverage.
- Adds regression and Unicode-range tests.
| File | Description |
|---|---|
| pkg/sanitize/sanitize.go | Updates sanitization order and removal ranges. |
| pkg/sanitize/sanitize_test.go | Adds entity-decoding and Unicode coverage. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 2/2 changed files
- Comments generated: 4
- Review effort level: Balanced
Sorry, something went wrong.
Comment thread
pkg/sanitize/sanitize.go
Outdated
Comment thread
pkg/sanitize/sanitize.go
Outdated
Comment thread
pkg/sanitize/sanitize.go
Outdated
Comment thread
pkg/sanitize/sanitize_test.go
Outdated
Address review feedback on the post-HTML-entity sanitization pass. Entity decoding could still smuggle code-fence metadata past the sanitizer. A first line such as "`​``steal secrets" is not a fence in the raw input, so FilterCodeFenceMetadata left it alone; decoding the entity and stripping the zero width space then produced a real fence with its info string intact. Sanitize now re-runs the fence filter after the input is fully normalized. Filtering every variation selector also corrupted legitimate text: VS15 and VS16 select text or emoji presentation, so "✈️" was reduced to "✈", and the Variation Selectors Supplement encodes registered CJK ideographic variation sequences. Selectors are now filtered contextually. A selector is kept when it can apply to the character it follows, and dropped when it is orphaned, follows a removed or non-graphic character, or continues a run of selectors. Supplement selectors additionally require a CJK ideograph base, matching the Ideographic Variation Database. That keeps the anti-smuggling property, since hidden payloads rely on selector runs, without rewriting valid Unicode. Also corrects a lowercase-hex test case that claimed uppercase digits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…-invisible-unicode-after-html-enti
SamMorrowDrums
deleted the
sammorrowdrums-filter-invisible-unicode-after-html-enti
branch
August 19, 2026 13:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FilterInvisibleCharacters only ran before FilterHTMLTags in Sanitize. FilterHTMLTags (bluemonday) decodes HTML character entities, so a numeric entity like ​ or ​ becomes a literal U+200B (zero width space) after the invisible-character pass already ran — letting encoded invisible/bidi characters survive sanitization untouched.
The removal set also omitted a couple of relevant format/variation-selector ranges.
Fix
These additions target characters commonly used for invisible-payload smuggling while leaving ordinary text, emoji, and CJK content untouched.
Testing
Fixes #3101
Acknowledgments
Thanks @Gal3m for the reports that led to this hardening.