unified: Output source code skeletons in AST dump - #22624
Merged
tausbn merged 2 commits intoSep 18, 2026
Merged
tausbn merged 2 commits into
tausbn merged 2 commits into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Skeleton generation currently hides unnamed delimiters and fails to validate malformed zero-width child ranges.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Skip hidden unnamed tokens when building skeletons · New
Adds opt-in source skeleton annotations to Yeast AST dumps and enables them for generated Swift corpus expectations.
Changes:
- Adds source-range skeleton generation, validation, and unit tests.
- Enables skeletons in unified Swift corpus tests.
- Documents the feature and regenerates Swift outputs.
Swift corpus paths below are relative to unified/extractor/tests/corpus/.
| File | Description |
|---|---|
| shared/yeast/src/dump.rs | Implements source skeleton dumping and tests. |
| shared/yeast/doc/yeast.md | Documents source skeleton output. |
| unified/extractor/tests/corpus_tests.rs | Enables skeletons for mapped Swift AST dumps. |
| swift/variables/var-without-initialiser.output | Updates generated variable output. |
| swift/variables/var-binding.output | Updates generated variable output. |
| swift/variables/tuple-destructuring-binding.output | Updates generated variable output. |
| swift/variables/property-with-willset-and-didset-observers.output | Updates generated observer output. |
| swift/variables/multiple-bindings-on-one-line.output | Updates generated binding output. |
| swift/variables/let-with-type-annotation.output | Updates generated variable output. |
| swift/variables/let-binding.output | Updates generated variable output. |
| swift/variables/compound-assignment.output | Updates generated assignment output. |
| swift/variables/binding-modifier-does-not-leak-into-initializer.output | Updates generated binding output. |
| swift/variables/assignment.output | Updates generated assignment output. |
| swift/types/actor-declaration.output | Updates generated type output. |
| swift/types/binding-modifier-does-not-leak-into-accessor-body.output | Updates generated accessor output. |
| swift/types/class-function.output | Updates generated class output. |
| swift/types/class-inheritance.output | Updates generated inheritance output. |
| swift/types/class-with-initializer.output | Updates generated class output. |
| swift/types/class-with-method.output | Updates generated class output. |
| swift/types/class-with-multiple-base-types.output | Updates generated inheritance output. |
| swift/types/class-with-stored-properties.output | Updates generated class output. |
| swift/types/computed-property.output | Updates generated property output. |
| swift/types/conditional-compilation-in-class-body.output | Updates generated conditional output. |
| swift/types/constructor-with-parameters.output | Updates generated constructor output. |
| swift/types/empty-class.output | Updates generated class output. |
| swift/types/enum-with-associated-values.output | Updates generated enum output. |
| swift/types/enum-with-cases.output | Updates generated enum output. |
| swift/types/enum-with-comma-separated-cases-chained-declaration.output | Updates generated enum output. |
| swift/types/extension.output | Updates generated extension output. |
| swift/types/function-type-with-convention-attribute.output | Updates generated function-type output. |
| swift/types/function-type-with-sendable-attribute.output | Updates generated function-type output. |
| swift/types/generic-class-parameters-and-constraints.output | Updates generated generic-type output. |
| swift/types/generic-type-arguments.output | Updates generated generic-type output. |
| swift/types/inline-array-type.output | Updates generated array-type output. |
| swift/types/noncopyable-type.output | Updates generated noncopyable-type output. |
| swift/types/property-with-getter-and-setter.output | Updates generated property output. |
| swift/types/protocol-declaration.output | Updates generated protocol output. |
| swift/types/protocol-with-read-only-and-read-write-property-requirements.output | Updates generated protocol output. |
| swift/types/qualified-type.output | Updates generated qualified-type output. |
| swift/types/static-function.output | Updates generated static-function output. |
| swift/types/struct.output | Updates generated struct output. |
| swift/optionals-and-errors/catch-where-clauses.output | Updates generated error-handling output. |
| swift/optionals-and-errors/do-catch.output | Updates generated error-handling output. |
| swift/optionals-and-errors/force-unwrap.output | Updates generated optional output. |
| swift/optionals-and-errors/nil-coalescing.output | Updates generated optional output. |
| swift/optionals-and-errors/optional-chaining.output | Updates generated optional output. |
| swift/optionals-and-errors/optional-enum-case-binding.output | Updates generated optional-pattern output. |
| swift/optionals-and-errors/optional-type-annotation.output | Updates generated optional-type output. |
| swift/optionals-and-errors/throwing-function.output | Updates generated throwing-function output. |
| swift/optionals-and-errors/try-expression.output | Updates generated try-expression output. |
| swift/optionals-and-errors/try-expression-2.output | Updates generated try-expression output. |
| swift/operators/addition.output | Updates generated operator output. |
| swift/operators/comparison.output | Updates generated operator output. |
| swift/operators/custom-postfix-operator.output | Updates generated operator output. |
| swift/operators/division.output | Updates generated operator output. |
| swift/operators/equality.output | Updates generated operator output. |
| swift/operators/logical-and.output | Updates generated operator output. |
| swift/operators/logical-not.output | Updates generated operator output. |
| swift/operators/logical-or.output | Updates generated operator output. |
| swift/operators/multiplication.output | Updates generated operator output. |
| swift/operators/operator-precedence-addition-and-multiplication.output | Updates generated precedence output. |
| swift/operators/parenthesised-expression.output | Updates generated expression output. |
| swift/operators/partial-range-from.output | Updates generated range output. |
| swift/operators/range-operator.output | Updates generated range output. |
| swift/operators/subtraction.output | Updates generated operator output. |
| swift/operators/unresolved-operator-sequence.output | Updates generated operator output. |
| swift/operators/unresolved-operator-sequence-with-casts.output | Updates generated operator output. |
| swift/operators/unresolved-operator-sequence-with-ternary.output | Updates generated operator output. |
| swift/loops/break-and-continue.output | Updates generated loop output. |
| swift/loops/for-in-over-array-literal.output | Updates generated loop output. |
| swift/loops/for-in-over-range.output | Updates generated loop output. |
| swift/loops/for-in-with-where-clause.output | Updates generated loop output. |
| swift/loops/repeat-while-loop.output | Updates generated loop output. |
| swift/loops/while-loop.output | Updates generated loop output. |
| swift/literals/boolean-literals.output | Updates generated literal output. |
| swift/literals/floating-point-literal.output | Updates generated literal output. |
| swift/literals/integer-literal.output | Updates generated literal output. |
| swift/literals/line-magic-literal.output | Updates generated literal output. |
| swift/literals/negative-integer-literal.output | Updates generated literal output. |
| swift/literals/nil-literal.output | Updates generated literal output. |
| swift/literals/string-literal.output | Updates generated literal output. |
| swift/literals/string-with-interpolation.output | Updates generated interpolation output. |
| swift/functions/call-with-inout-argument.output | Updates generated call output. |
| swift/functions/constructor-call-with-type-arguments.output | Updates generated constructor-call output. |
| swift/functions/function-call.output | Updates generated call output. |
| swift/functions/function-call-with-labelled-arguments.output | Updates generated call output. |
| swift/functions/function-with-default-parameter-value.output | Updates generated function output. |
| swift/functions/function-with-inout-parameter.output | Updates generated function output. |
| swift/functions/function-with-named-parameters.output | Updates generated function output. |
| swift/functions/function-with-no-parameters.output | Updates generated function output. |
| swift/functions/function-with-parameters-and-return-type.output | Updates generated function output. |
| swift/functions/generic-function.output | Updates generated generic-function output. |
| swift/functions/generic-type-alias.output | Updates generated type-alias output. |
| swift/functions/leading-dot-expression-call.output | Updates generated call output. |
| swift/functions/leading-dot-expression-value.output | Updates generated expression output. |
| swift/functions/method-call.output | Updates generated method-call output. |
| swift/functions/nested-function-type.output | Updates generated function-type output. |
| swift/functions/variadic-function.output | Updates generated variadic-function output. |
| swift/expressions/array-type-constructor.output | Updates generated array-constructor output. |
| swift/expressions/array-type-metatype.output | Updates generated metatype output. |
| swift/expressions/consume-expression.output | Updates generated expression output. |
| swift/expressions/copy-expression.output | Updates generated expression output. |
| swift/expressions/generic-specialization-expression.output | Updates generated specialization output. |
| swift/expressions/key-path-expression.output | Updates generated key-path output. |
| swift/expressions/unsafe-expression.output | Updates generated expression output. |
| swift/desugar/additive-expression-is-desugared.output | Updates generated desugaring output. |
| swift/desugar/another-additive-expression-is-desugared.output | Updates generated desugaring output. |
| swift/desugar/import-with-deeply-nested-path-three-parts.output | Updates generated import output. |
| swift/desugar/import-with-dotted-path-two-parts.output | Updates generated import output. |
| swift/desugar/scoped-import-uses-name-pattern.output | Updates generated import output. |
| swift/desugar/simple-import-with-single-name.output | Updates generated import output. |
| swift/control-flow/binding-modifier-does-not-leak-to-sibling.output | Updates generated control-flow output. |
| swift/control-flow/defer-statement.output | Updates generated defer output. |
| swift/control-flow/discard-statement.output | Updates generated discard output. |
| swift/control-flow/fallthrough.output | Updates generated switch output. |
| swift/control-flow/guard-let.output | Updates generated guard output. |
| swift/control-flow/if-case-let-with-shadowing-in-condition-value.output | Updates generated conditional output. |
| swift/control-flow/if-else-if-chain.output | Updates generated conditional output. |
| swift/control-flow/if-else.output | Updates generated conditional output. |
| swift/control-flow/if-let-optional-binding.output | Updates generated conditional output. |
| swift/control-flow/if-statement.output | Updates generated conditional output. |
| swift/control-flow/nested-enum-case-pattern.output | Updates generated pattern output. |
| swift/control-flow/switch-case-item-where-clauses.output | Updates generated switch output. |
| swift/control-flow/switch-expression-pattern.output | Updates generated switch output. |
| swift/control-flow/switch-statement.output | Updates generated switch output. |
| swift/control-flow/switch-with-binding-pattern.output | Updates generated switch output. |
| swift/control-flow/switch-with-labeled-case-pattern-arguments.output | Updates generated switch output. |
| swift/control-flow/ternary-expression.output | Updates generated ternary output. |
| swift/collections/array-literal.output | Updates generated collection output. |
| swift/collections/dictionary-literal.output | Updates generated collection output. |
| swift/collections/dictionary-subscript.output | Updates generated subscript output. |
| swift/collections/empty-array-literal-with-type.output | Updates generated collection output. |
| swift/collections/set-literal.output | Updates generated collection output. |
| swift/collections/subscript-access.output | Updates generated subscript output. |
| swift/collections/tuple-literal.output | Updates generated tuple output. |
| swift/collections/tuple-member-access.output | Updates generated tuple output. |
| swift/closures/closure-with-capture-list.output | Updates generated closure output. |
| swift/closures/closure-with-explicit-parameters.output | Updates generated closure output. |
| swift/closures/closure-with-shorthand-parameters.output | Updates generated closure output. |
| swift/closures/multi-statement-closure.output | Updates generated closure output. |
| swift/closures/trailing-closure.output | Updates generated closure output. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sorry, something went wrong.
tausbn
force-pushed
the
tausbn/unified-corpus-source-skeletons
branch
from
1714793 to
2f0f02b
Compare
September 18, 2026 11:51
With this change, output AST nodes (with locations that span part of the source code) now get an additional `source=...` annotation that compactly expresses how much of the source code is included in the location. For example, a variable declaration might look like this: ``` variable_declaration source="⟨modifier⟩ ⟨pattern⟩ = ⟨value⟩" ``` Here, the bits inside Unicode angle brackets correspond to sub-ranges of the input that are owned by direct children of the node in question. The string inside the brackets gives the name of the corresponding field. (I chose Unicode angle brackets because they are unlikely to occur naturally in source code.) This provides us with (hopefully) an intuitive and compact way of representing locations, without needing to list particular row and column offsets (which are impossible to inspect manually anyway).
Regenerates all of the existing corpus output files to include source skeletons. Note that this already reveals some issues with our current locations: - arguments with trailing commas have a location that includes said comma, - switch_case nodes contain a `block` that encompasses the `case` statement itself. These will be fixed in due course.
tausbn
force-pushed
the
tausbn/unified-corpus-source-skeletons
branch
from
2f0f02b to
a0f9f35
Compare
September 18, 2026 12:06
tausbn
marked this pull request as ready for review
September 18, 2026 12:09
asgerf
approved these changes
Sep 18, 2026
asgerf
left a comment
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Looks great! I really like this way of visualising the source range
Sorry, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes it so that output AST nodes (with locations that span part of the source code) now get an additional source=... annotation that compactly expresses how much of the source code is included in the location.
For example, a variable declaration might look like this:
Here, the bits inside Unicode angle brackets correspond to sub-ranges of the input that are owned by direct children of the node in question. The string inside the brackets gives the name of the corresponding field. (I chose Unicode angle brackets because they are unlikely to occur naturally in source code.)
This provides us with (hopefully) an intuitive and compact way of representing locations, without needing to list particular row and column offsets (which are impossible to inspect manually anyway).
First commit contains the AST dumper changes, the second updates the corpus with all of the new source skeletons.