Make triple-quoted strings interpret the escape whitelist (#48)
* Apply the existing escape whitelist to triple-quoted strings
Triple-quoted literals were fully raw while single-delimiter strings processed a
seven-character escape whitelist. That split made the escaping regime depend on
how many quote characters were counted: `\'` is correct in `'…'` and ships
verbatim in `'''…'''`, with nothing else distinguishing them.
LLM-authored Forthic does not survive that. Over a 20,000-program production
week, 93% of every backslash occurrence inside a triple-quoted literal is a
`\'` or `\"` the model wrote expecting it to be an escape, and the corruption
reaches users. Replaying the same week through this change drops apostrophe
corruption inside `'''` from 22.97% to 14.91%, and double-quote corruption
inside `"""` from 49.02% to 1.96%.
No language treats `'''` as raw. Python and Groovy are the only languages that
have it and both process escapes; the raw-triple-quote convention belongs to
`"""` in C-family languages where `'` is a character literal and `'''` does not
exist. Forthic paired Python's spelling with Kotlin's semantics.
The whitelist is what keeps this safe rather than a wholesale move to escaped
strings: only \n \t \r \0 \\ \" \' are interpreted, so `\d`, `\w` and `\U`
still stay literal and regexes and Windows paths remain writable. Across the
committed corpus (2,887 literals) exactly two sites change meaning, both in a
gdrive example that ships a literal `\n` into a Google Doc where a newline was
plainly intended.
Escapes resolve before delimiter detection, so an escaped quote is content and
can never close the literal. Unescaped tripling (`'''today'''s plan'''`) still
closes early — this change moves the silent classes, not the loud one.
Three tests asserting the previous raw behaviour are updated; the regex and
path cases they sat beside are untouched and still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UUMTq8x3R5KN1pxKJa4ua
* Add raw string literals (r'…') ahead of making triple quotes non-raw
Forthic strings carry two jobs that one delimiter cannot serve. Regular strings
interpret a seven-escape whitelist, so an author writing 'today\'s' gets an
apostrophe. Triple-quoted strings carry data, where a backslash has to survive
untouched: JSON payloads and embedded Forthic source parse their own escapes,
and any escape the tokenizer consumes is one the inner layer never sees.
Serving the first job at triple width breaks the second. '''{"a": "x\ny"}'''
JSON> fails on a raw control character, '''{"p": "C:\\Users"}''' JSON> on an
invalid \U, and """ 'a\\nb' """ RUN processes the escape twice. Leaving triple
quotes raw to protect the second leaves the first broken — an LLM writes
'''today\'s plan''' because that is correct one delimiter width narrower, and
the backslash reaches users. Both are paths docs/forthic-prompt.md documents.
An explicit raw form separates the jobs. `r` glued to any opening delimiter
turns escape processing off for that literal: r'…', r"…", r'''…''', r"""…""".
This is a transitional release, not the destination. Triple-quoted strings are
still raw, so the triple-width r forms are aliases today. They exist so data
literals can move before the next release makes '''…''' and """…""" interpret
the whitelist, at which point a bare triple-quoted string stops being a safe
place for a backslash. This is the only window in which both spellings work.
The tokenizer gains is_raw_string_start() and one branch in
transition_from_START, mirroring the existing <<'''…''' marked-redirect
lookahead, plus a raw flag on transition_from_GATHER_STRING that skips its
escape branch. The triple-quote gather does no escape processing, so the
triple-width forms reuse it unchanged. The prompt routes backslash-carrying
strings to the r forms and warns that the bare form will not keep a backslash.
`r` is the strongest prior available (Python, Rust, Nim, D, Scala) and Forthic
already borrowed Python's ''' spelling. Backticks were ruled out despite Go and
JS: generated Forthic is routinely wrapped in markdown fences, where
triple-backtick collides. The prefix is lowercase only and does not compose with
the redirect marker. Unlike Python, a raw Forthic string may end in a backslash
— r'C:\' is C:\ — though the same rule means it cannot contain its own
delimiter.
One narrow break rather than a pure addition: words are gathered without
breaking on quote characters, so a token spelled r + quote (r'don't') used to
lex as a single word and now opens a raw string. No standard-library word is
spelled that way, and definition names cannot contain quotes, so no definable
word is affected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N1Xfa8zk25MrXCZcUEPWY7
* Make triple-quoted strings interpret the escape whitelist
The escaping regime depended on how many quote characters were counted. `\'` was
correct in '...' and shipped verbatim in '''...''', with nothing else in the
source to tell them apart. LLM-authored Forthic does not survive that: over a
20,000-program production week, 93% of every backslash inside a triple-quoted
literal is a \' or \" the model wrote expecting an escape, and the backslash
reaches users.
0.16.2 could not simply fix it, because triple quotes were carrying a second job
— they were the raw, data-carrying form, and JSON> and RUN parse their own
escapes. That release split the jobs by adding an explicit raw prefix and
pre-announced this one. This is the second half: '''...''' and """...""" now
interpret \n \t \r \0 \\ \" \', and r'''...''' is the only raw spelling at
triple width.
The whitelist is what keeps this safe rather than a wholesale move to escaped
strings. \d, \w, \U and \. stay literal, so regexes and Windows paths are still
writable without doubling. Escapes resolve before delimiter detection, so an
escaped quote is content and can never close the literal — '''a \'\'\' b''' is
a ''' b. Unescaped tripling still closes early: this moves the silent failure
classes, not the loud one.
One new loud break: a triple-quoted literal ending in a backslash no longer
parses. '''C:\''' consumes \' as an escaped quote, leaving '' that cannot close.
Write r'''C:\'''.
Fixes a streaming corruption the flip would otherwise arm. streamingRun
re-tokenizes the whole cumulative buffer each chunk and reports the open string's
content-so-far to its redirect sink, which diffs by length. A backslash landing
as the last character of a chunk was reported as content, then re-read as an
escape next chunk — so the value could shrink, or stay the same length while
changing, and the sink received a\b where the stack held a real newline.
get_string_value() now holds back a trailing backslash while a triple-quoted
string is open, exactly as it already held back an unconfirmed closing quote.
Both halves land together; there is no commit where the flip exists without the
hold-back. The test that covers it compares the sink's concatenated deltas
against the string actually left on the stack, with no hardcoded expected value,
because pinning the text would let a future semantics change mask a divergence.
It was confirmed to fail without the hold-back and pass with it. Note that a
doubled backslash still yields a growing prefix at every cut, so it is kept as a
control rather than evidence.
The prompt's Strings section is rewritten around what models were guessing at.
It now states one rule for all four widths, says outright that an escape
resolves before the closing delimiter is looked for, shows \' inside '''...'''
and \" inside """...""", names the unescaped-tripling failure, and puts
"change the delimiter instead" ahead of escaping as the better habit. Eleven of
its examples are written in the `code` -> `expected` form the smoke test
executes, up from three, so the semantics that were being guessed at are now
machine-checked. Prompt docs are generated; the generator was edited.
export_state captures definition source verbatim and import_state re-tokenizes
it, so a 0.16.x state blob whose definitions hold bare triple literals with
whitelisted escapes changes meaning on import. Round-trip tests pin which
spelling survives the boundary, and the changelog says so.
Not included, and tracked separately as urgent: unescape_string() rewrites <
and > across the whole cumulative buffer before tokenizing, so a chunk
boundary inside one of those entities makes the reported value mutate rather
than extend — the sink receives abc&l while the stack holds abc<X, at equal
length, so nothing corrects it. That predates this change and is not caused by
it. The router-side prefix-invariant guard belongs with that fix rather than
here, since it would otherwise throw on a legitimate stream.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012tWcQ3X7VpqUerMEgnZj8o
* Fix the Windows-path claim in the 0.17.0 migration notes
The CHANGELOG listed '''C:\Users\tmp''' among the literals left unchanged by
the escape flip. \t is in the whitelist, so that literal now yields
C:\Users + TAB + mp — the guidance told readers to skip exactly the paths
that break. Replace the example with one that survives, spell out the
overlap, and say what to grep for.
Pin the semantics with a test: the mixed case was present in the redirect
payload table but only checked for sink/stack consistency, so nothing caught
the false claim. Its comment described it as non-whitelist too.
Also trim the comments the escape work added. The hold-back rule was stated
three times (field, docblock, setter) — keep the full version on
get_string_value() and point the other two at it. Drop the paragraph
justifying one loop over two passes down to the clause that carries it, and
the non-escape branch comment that repeated the ESCAPE_MAP docblock verbatim.
Reword release-relative framing in tests ("the flip", "the 0.16.2 window is
closed") to state the rule instead, which is what still reads correctly once
0.17.0 is several releases back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TPasz3i77ev8qk1RCSsgey
* Tighten Forthic string guidance
* Clarify Forthic quote escaping guidance
* Fix executable prompt example notation
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>