| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
A step-level EXPR `let` binding was evaluated with the host's path format.
openjd-rs evaluates template-scope expressions with `PathFormat::Posix`
(create_job/instantiate.rs, create_job/mod.rs) so that a create-time result
cannot depend on the host that created the job. This implementation used the
engine default, which is the host's format, so on Windows
`startswith(path("/foo/bar"), "/foo")` evaluated to false where the Rust
implementation gives true, and `string(path("/mnt/out"))` rendered
`\mnt\out` rather than `/mnt/out`.
`evaluate_let_bindings` now takes a `path_format`, and
`StepTemplate._extend_step_symtab` passes `PathFormat.POSIX`. The default
stays `None` (the engine default) so every session-scope caller is unchanged.
The instantiated `StepScript` also records `_template_scope_let_count`, the
number of leading entries in its merged `let` list that came from the step.
The merge itself is unchanged, because consumers that do not pass a resolved
symbol table to `Session.run_task` rely on it. openjd-sessions reads the count
to re-evaluate that prefix in template scope rather than in the host's, and
reads it through `getattr` with a default of 0, so an older openjd-model
degrades to the previous behaviour. It is a `PrivateAttr`, so the serialized
form of the model does not change.
This is the openjd-model half of a two-repo fix. Without the openjd-sessions
half, a step-level binding is still re-evaluated in host scope at session
time, so the 11 Python-on-Windows conformance failures this addresses need
both halves.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
|
|
||
| evaluate_let_bindings(symtab=step_symtab, let_bindings=self.let) | ||
| evaluate_let_bindings( | ||
| symtab=step_symtab, let_bindings=self.let, path_format=PathFormat.POSIX |
There was a problem hiding this comment.
The POSIX pin only covers let bindings; the rest of create-time resolution still renders PATH values in the host format, so the host-independence property the docstring above states is not actually achieved for a step that puts a path expression somewhere other than a let.
The two create-time resolution entry points do not thread a path_format:
So a template with hostRequirements.amounts[].name: "{{ startswith(path(\"/foo/bar\"), \"/foo\") ? ... }}", or a task range built from a path expression, still evaluates against the creating host’s format and yields a different Job on Windows vs Linux — the exact failure the added test test_step_symtab_path_predicate_is_host_independent guards against, just reached through a field expression instead of a binding.
Worth either passing PathFormat.POSIX through those two call sites as well, or narrowing the _extend_step_symtab docstring to say only let bindings are pinned so the remaining gap is not read as closed.
Sorry, something went wrong.
There was a problem hiding this comment.
Correct, and open. Measured: create_job resolved min: '{{ 4 if startswith(path("/foo/bar"),"/foo") else 8 }}' to 4 through the host format, so every non-let template-scope field is still host-dependent, and openjd-rs pins POSIX at every create-time site where this PR pins one. The fix is to thread POSIX through _create_job.py:48 and :283, a behaviour change worth its own PR, so this stays open.
Sorry, something went wrong.
Two defects in the previous commit, both found by local review. The boundary was recorded only by `Step._record_template_scope_let_count`, an after-validator on `Step`. A consumer that parses a `StepTemplate` and runs `resolve_syntax_sugar()` on it never constructs a `Step`, so the validator never ran and the boundary was lost. The Deadline worker agent does exactly that, via BatchGetJobEntity, so the fix reached `openjd-cli` and not the worker. Measured before: `create_job` gave a count of 2 and the resolve_syntax_sugar path gave nothing. The count is now set at the merge site as well, and both paths give 2. The validator also set the count to `len(self.let)` without checking that `script.let` actually starts with `self.let`. It runs for a `Step` built directly, where nothing guarantees that. A mismatch recorded a count that would make a session evaluate a genuinely session-scope binding in template scope. It now verifies the prefix and records 0 when it does not match, which is the previous behaviour. The attribute's comment claimed a session "must skip" the prefix. It reproduces it in template scope and re-tags the result to host format instead, so the comment is corrected to say that. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
The generator script fails on BSD sed, so the two version lines CI reported are bumped by hand: pydantic 2.13.4 to 2.13.5 and pydantic_core 2.46.4 to 2.46.5. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
StepTemplate.resolve_syntax_sugar builds the SimpleAction script with model_construct and never set _template_scope_let_count, while the script: branch does. For the same merged let list the sugar branch yielded 0 and the script: branch 1, so a consumer that parses a StepTemplate and calls resolve_syntax_sugar() directly -- the worker agent, via BatchGetJobEntity -- got no boundary and evaluated the step template's own let bindings in session scope instead of template scope. Set it at the merge, from the same source of truth (len(self.let)), by assignment, as the script: branch does, since model_construct bypasses validators. The existing coverage went through create_job, where Step's validator records the count regardless, so it did not reach this path. The new test resolves the sugar on the StepTemplate itself and compares against the script: branch, over all five interpreters. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
`Step._record_template_scope_let_count` wrote its computed count to `self.script._template_scope_let_count` unconditionally. The script is a shared object: one merged by `StepTemplate.resolve_syntax_sugar` can be reached by more than one `Step`, and the instance is kept rather than revalidated. A sibling `Step` whose own `let` is empty, or does not match the prefix, computed 0 and wrote it over a correct marker of 2 — silently reverting the owning step to re-evaluating template-scope bindings in host scope, the bug the marker exists to prevent. The write could only ever lower a correct value to a wrong one. Record the count only when it is non-zero. Two `Step`s share a script object only when they share its `let` list, so the marker the owning step computed describes that list correctly for both readers, whereas a 0 computed by a sibling describes only that sibling's own `let`. The existing `matches_prefix` verification still guards a genuinely mismatched prefix, so a non-zero count is only recorded after verification, and an unmarked script still reads 0 from the PrivateAttr default. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
A step's template-scope `let` is already resolved at job creation and
travels in the step symbol table returned by
create_job_with_symbol_tables(...).step_symbol_tables[step_name]. For a
step-level let of ["a = 1", "root = path('/foo/bar')",
"txt = string(path('/mnt/out'))"] that table holds `a` int 1, `root` path
'/foo/bar' stored POSIX, and `txt` string '/mnt/out', with the
script-level let correctly absent. The worker agent forwards that table to
openjd-sessions as of 08a5878b and deleted its extra_let_bindings channel
outright.
Merging the step-level let into script.let for the runtime to re-evaluate
is therefore redundant, and harmful: when both run, the session-side
re-evaluation writes last and clobbers the correctly formatted seeded
value.
Removed:
- the merge in the `script:` branch of StepTemplate.resolve_syntax_sugar
- the merge in the SimpleAction sugar branch
- the Step after-validator _record_template_scope_let_count
- the _template_scope_let_count PrivateAttr on StepScript
The create-time step-scope evaluation that populates the step symbol
table is unchanged, and Step.let remains a model field -- it just stops
being merged into the script.
Deleted TestTemplateScopeLetCount, which tested the removed machinery,
and added tests pinning the new contract: resolve_syntax_sugar leaves
script.let holding only the script's own bindings (both the `script:`
branch and all five SimpleAction interpreters), the step-level bindings
arrive in the step symbol table with a PATH value rendering /foo/bar
under POSIX and \foo\bar under WINDOWS, and the script-level let is
absent from that table.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
|
|
||
| evaluate_let_bindings(symtab=step_symtab, let_bindings=self.let) | ||
| evaluate_let_bindings( | ||
| symtab=step_symtab, let_bindings=self.let, path_format=PathFormat.POSIX |
There was a problem hiding this comment.
The POSIX guarantee stops at the let values themselves; the create-time consumers of those values still resolve in host format.
_extend_step_symtab now evaluates the bindings with PathFormat.POSIX, but the fields that consume them at job creation are resolved by instantiate_model -> _instantiate_noncollection_value, which calls value.resolve(symtab=symtab) with no path_format (src/openjd/model/_internal/_create_job.py:283). Those create-time-resolved fields include exactly the ones this docstring cites as the motivation: the step parameterSpace ranges (resolve_fields includes range, _model.py:1435) and hostRequirements (_model.py:3073, 3220).
So for a PATH-typed binding such as root = path("/foo/bar"), used from a task parameter range of "<<root>>/a,<<root>>/b" (double-brace interpolation), the binding is stored POSIX-correct as an ExprValue, but the range format string is rendered via ExprNode._evaluate_raw(path_format=None), so the engine coerces the path with the host separator. The instantiated Job then holds a backslash rendering when created on Windows and a slash rendering on Linux -- the same host-dependence the docstring says this change eliminates.
The new tests do not catch this because they only assert on bindings that coerce to a string inside the expression (string(path(...)), startswith(...)), where POSIX is already baked in at let-evaluation time.
If the intent is to match the openjd-rs hardcoded PathFormat::Posix for the whole of job instantiation, the format-string resolution during instantiation needs the same path_format threaded through it, not just the let evaluation.
Sorry, something went wrong.
There was a problem hiding this comment.
Correct, and a known open item that this PR does not close. Create-time consumers do still resolve in host format via _instantiate_noncollection_value's value.resolve(symtab=symtab) at _create_job.py:283, so parameterSpace ranges and hostRequirements remain host-dependent for a PATH-typed binding. It is tracked as its own change rather than folded in here, because threading POSIX through the whole of instantiation is a behaviour change that deserves its own PR and conformance run; leaving this thread open as the record.
Sorry, something went wrong.
The comment justified running the symbol-table hook before the transform by saying StepTemplate's syntax-sugar transform folds step-level `let` bindings into the script. That fold was deleted in the previous commit, so the stated reason no longer exists -- but the ordering is still load-bearing, and this comment was its only record. Restate the two reasons that survive: * A transform may rebuild the model rather than adjust it -- `resolve_syntax_sugar` returns a `model_construct`ed StepTemplate -- so the fields the hook reads (`name`, `let`) are only guaranteed to be the authored ones on this side of it. The transform carries `let` through deliberately; running the hook first keeps that the transform's choice rather than a requirement on every future one. * `create_job_with_symbol_tables` invokes the same hook on the same untransformed StepTemplate to build the step symbol table it publishes for the runtime. That table is only the scope the step's own fields were instantiated against if both callers hand the hook the same model. Comment only; no behaviour change. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Dropping the step-level `let` merge means the Job returned by plain create_job() is no longer self-contained for an EXPR template whose step declares a `let` its script references: at 6a23815 script.let was ['a = 1'] and the task succeeded; at 260b147 script.let is None and the task fails with `Undefined variable: 'a'`. The removal is correct -- both channels evaluating caused the clobber this branch fixes -- so document the contract instead of adding a shim. Rewrite create_job's docstring to say plainly that the Job does not carry step-level `let` values and that a caller running such a job must use create_job_with_symbol_tables and forward the tables, add a CHANGELOG breaking-change entry, and add the same pointer to the README's "Creating a Job from a Job Template" example. The README's other two create_job() examples only inspect the Job at creation time (StepDependencyGraph, StepParameterSpaceIterator), so plain create_job stays correct there and they are left alone. BREAKING CHANGE: create_job() no longer returns a Job carrying evaluated step-level `let` values. A caller that runs a job whose steps declare a template-scope `let` referenced from the step's script must switch to create_job_with_symbol_tables and forward the step's step_symbol_tables entry to the session. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
| Back | FazBrowse Home | New Git URL |
What changed
A step's template-scope let (RFC 0007 §3.6) is now resolved once, at job creation, with PathFormat.POSIX, and it is no longer merged into the script's own let afterwards.
StepTemplate._extend_step_symtab passes PathFormat.POSIX to evaluate_let_bindings, which gained a path_format parameter defaulting to None. Session-scope callers pass nothing and keep the engine default, which is the host's format. The fixed format matches openjd-rs, whose job instantiation hardcodes POSIX in create_job/instantiate.rs and uses the host's format only inside a session. The reason for a fixed format is that a create-time value must not depend on the host that created the job.
resolve_syntax_sugar no longer folds the step-level bindings into script.let. The merge is gone from both of its branches: the script: branch and the SimpleAction de-sugar branch. Earlier revisions of this PR kept the merge and recorded a boundary marker on the instantiated script so a session could split the merged list at it. That marker is gone as well.
Step.let is retained. It holds the step's source expressions, which a step environment still needs. Their resolved values travel separately, in create_job_with_symbol_tables(...).step_symbol_tables[step_name].
Why the merge had to go
A merged list was re-evaluated at session time in the host's format, which re-rendered its PATH values. On Windows, startswith(path("/foo/bar"), "/foo") flips from true to false between the create-time evaluation and the session-time one.
The re-evaluation was not a harmless second opinion. The seeded create-time value and the re-evaluated one land in the same symbol table, and the re-evaluation writes last, so it overwrote the correct value.
Removing the re-evaluation and forwarding the resolved table instead is what openjd-rs has always done, and what the Deadline worker agent adopted in public commit 08a5878b, "feat: forward the resolved symbol table to the v0 session", where it replaced extra_let_bindings outright. This is a convergent fix that adopts an existing upstream design rather than a new one.
Migration
Removing the merge is a breaking change for any consumer that calls resolve_syntax_sugar() and does not forward a resolved symbol table to its session. Such a consumer silently loses step-level bindings rather than failing. The worker agent forwards; openjd-cli now does too, in OpenJobDescription/openjd-cli#237. A third-party consumer would not, and there is no in-process fallback, because no Python Step carries the resolved table.
Verification
Conformance through the branch CLI: 1172 passed, 2 failed. expr2.2.1--string-conversion and expr2.3.2--path-construction both pass — those are the two fixtures that pull in opposite directions, one wanting a value that has left path-space to stay /mnt/out and the other wanting a path to render in the host's format. The 2 failures are pre-existing range-normalization cases, unrelated to this change and blocked on a spec ruling.
Suites: model 5496, sessions 999, cli 318.
Not verified
Windows. The companion sessions PR (OpenJobDescription/openjd-sessions-for-python#362) has had its Windows CI legs fail-fast cancelled on every round, so Windows has never been exercised by CI there. The behaviour was proven by simulation instead: with the host format forced to Windows in-process, a path-typed binding renders \a\b while a value that has left path-space stays /mnt/out.
The conformance runner concatenates a fixture's output block with its output_<os> block, so on a POSIX host an output_windows block is never asserted. A green POSIX conformance run proves nothing about the Windows expectations.