| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [View Raw Code] [Original HTTPS Page] |
Companion to API_REWORK.md (master design doc) and API_MISSING.md (strawman workshop). This file tracks a verification workstream: an explicit, code-driven audit that the public design promise holds.
The dasSQLITE _sql(...) macro is, by design, a pushdown optimizer on top of plain linq. As of the multi-Q lowering work (F1 resolution), this promise is unconditional for the shipped chain shapes — _sql honors linq's positional chain semantics, lowering divergent orders to nested SELECTs (SELECT ... FROM (subquery) AS t0 ...) instead of silently reordering into canonical SQL eval order. Same chain → same answer in all three modes.
The same chain shape:
src |> _where(_.Price > 100)
|> _group_by(_.City)
|> _select((City=_._0, N=_._1 |> length, Total=_._1 |> select($(u:T) => u.Salary) |> sum))…is meaningful in three execution modes:
Mode 1 and Mode 2 must agree numerically (this is what makes _sql a safe optimizer). Mode 2 and Mode 3 must agree (this is what makes the operator vocabulary genuinely unified, not a SQL-specific dialect).
This invariant is the load-bearing claim behind every tutorial that shows a _sql(...) chain. Today it is asserted by design and by ad-hoc testing, not by mechanical coverage.
In scope: linq chain operators — anything that participates in a src |> op1 |> op2 |> ... |> terminal chain and is meaningful both inside and outside _sql(...). _where, _select, _group_by, _having, _order_by, _join, _left_join, _first, _first_opt, _count, take, skip, distinct, set operations (union/intersect/except), subqueries (_in/_not_in/_any/_none), aggregates (length/sum/average/min/max), text_match (FTS5), _to_array. The parity claim is that these operators have a single semantic, regardless of source.
Out of scope: raw-SQL escape-hatch helpers — query_one, query_scalar, exec, try_exec, and their try_/_opt variants. These are not linq chains — they take a literal SQL string + bind arguments and return a row or scalar. They have no Mode-2 (no chain to run "without _sql") and no Mode-3 (no plain-array analog of "run this SQL string"). The parity story does not apply.
Also out of scope: _sql_text(...) (SQL-text debug emitter, not a runtime operator) and DDL helpers like create_table / _create_view.
For every tutorial example (and ideally every test) that contains a _sql(...) chain over a SQL source:
Every tutorial under tutorials/sql/ whose body contains _sql(...) — roughly tutorials 06 onward. The chains-without-_sql story isn't documented in any tutorial today (it's stated implicitly via the _sql_text debug macro), and the harness is what would let a future tutorial point at "and here, same chain, no macro, plain array" with confidence.
The harness is not a feature; it's an [export]-test pattern. Each case has three runs and two equality assertions:
// fixture: same dataset, two sources
db |> create_table(type<User>) // populated from `users_seed`
let users <- users_seed // plain in-memory copy
// the chain — written ONCE, used three times
def make_chain(src) {
return src |> _where(_.Price > 100)
|> _group_by(_.City)
|> _select((City=_._0, N=_._1 |> length))
}
// runs
let r_sql_pushdown <- _sql(make_chain(db |> select_from(type<User>)))
let r_sql_in_memory <- make_chain(db |> select_from(type<User>))
let r_plain_array <- make_chain(users)
// assertions
assert(r_sql_pushdown == r_sql_in_memory)
assert(r_sql_in_memory == r_plain_array)Open questions before this can be a real harness, not a sketch:
Tutorials currently teaching a thin slice of what the API surface should support. Tracked separately from findings — these are not-yet-shipped capabilities, not divergences in shipped behavior.
Missing: _right_join, _full_outer_join, _cross_join — none implemented in daslib/linq_boost or modules/dasSQLITE/daslib/sqlite_linq. Tutorial 16's note dismisses them as "not shipped" / "raw SQL". Issue #2558 tracks the work: add all three as first-class chain operators in linq, extend _sql emission, expand the tutorial (.das + .rst), extend the parity test. Surfaced 2026-05-02; Boris flagged as "huge gap."
Resolution (2026-05-03): _right_join, _full_outer_join, _cross_join shipped as first-class chain operators in daslib/linq_boost with corresponding runtime in daslib/linq. _sql translator extended to emit native RIGHT JOIN / FULL OUTER JOIN / CROSS JOIN (SQLite ≥3.39 required for FULL/RIGHT; bundled is 3.41.2). is_some / is_none projection probe extended to handle Right (LHS arg) and FullOuter (either arg). Tutorial 16 expanded to cover all five join shapes; tutorial 28's Section 12 mirrors. Parity test extended with three M1/M2/M3 blocks, each with structurally distinct named-tuple projections to dodge #2557. Cross runtime materializes RHS iterator before nested-loop. Branch: dassqlite-2558-all-joins.
Findings accumulate here as the harness walks the tutorials. Each entry names the operator pattern, the modes that disagree, what each mode produces, and the size of the impact (silent miscompute vs. clearly divergent shapes).
Resolution rule (Boris, 2026-05-02): we never just document incorrect behaviour. should work or fail to compile. Every finding ends in either (a) make-it-work code change, or (c) reject-the-shape macro_error that points at a canonical equivalent. (b) "work via different emission" is allowed when (a) and (c) are both worse, but "document and move on" is not a resolution. Tutorials demonstrating a rejected form must be updated to the canonical form in the same PR.
Tutorial 11. The same chain produced different row counts across modes:
| Mode | Chain | Old lowering | Rows |
|---|---|---|---|
| 1 | _sql(... |> take(n) |> skip(m)) | LIMIT n OFFSET m | n |
| 2 | ... |> take(n) |> skip(m) (no _sql, SQL src) | linq positional, in order | max(0, n − m) |
| 3 | arr |> take(n) |> skip(m) | linq positional, in order | max(0, n − m) |
_sql collapsed any combination of take and skip into one LIMIT/OFFSET pair (skip-then-take semantics) regardless of chain order. Plain linq honored chain order: take(n) first narrowed to n rows, then skip(m) dropped m of those. With the tutorial's fixture (take(2) |> skip(1), 5-row table), Mode 1 returned 2 rows and Modes 2/3 returned 1 row.
Decision (Boris, 2026-05-02): initial plan was (c) — reject the divergent shape with a macro_error and a fixit. After discussion, pivoted to (a) + (b) by lowering: the _sql analyzer was rewritten to honor linq's positional chain semantics across the board. Same chain → same answer in all three modes. Matches the EF Core mental model (project_dassqlite_ef_only.md). No fixit, no minefield.
Fix: multi-Q lowering in daslib/sqlite_linq.das. When a chain op's SQL eval phase exceeds q.minPhaseSeen (e.g. take (phase 8) appearing inside a skip (phase 7) or _where (phase 2)), the inner subtree is analyzed into a fresh SqlQuery and snapshotted to SQL string + bind list on the outer Q's innerSql / innerBindExprs fields. The outer SELECT's FROM clause renders as (<innerSql>) AS "t0". Bind collection recurses inner-first via collect_query_binds. Phase divert checks live in 5 peel blocks (TAKE / SKIP / ORDER_BY / ORDER_BY_DESC / DISTINCT — the ops where chain-order divergence is real with default-row inner). WHERE stays commutative (no divert). Set-ops / joins are unchanged for v1.
Side-effects: take(n) |> _where(p), take(n) |> _order_by(k), take(n) |> _order_by(k) |> take(m) (multi-level), and skip(n) |> _where(p) shapes that previously silently miscomputed in _sql now lower to subqueries with the linq-correct semantics.
Tutorial 11 (tutorials/sql/11-take_skip.das + sql_11_take_skip.rst): canonical pagination shape promoted to skip(m) |> take(n) (flat fast SQL). The reverse take(n) |> skip(m) is shown as the "honest, slower subquery" form, with the row-count change explained.
Coverage (tests/dasSQLITE/):
v2.1 RESOLVED: All multi-Q lowering capabilities below now ship; v2.1 closes the outer-WHERE-on-projected-alias gap.
v2 capabilities — shipped:
v2.1 capability — shipped:
Tutorial 13. Same chain produced different types and values:
| Mode | Chain | Returns | Value (Price=[100,200,300,50,999]) |
|---|---|---|---|
| 1 | _sql(... |> _select(_.Price) |> average()) | double | 329.8 (SQL AVG promotes) |
| 2 | ... |> _select(_.Price) |> average() (no _sql, SQL src) | int (input TT) | 329 (integer division) |
| 3 | arr |> _select(_.Price) |> average() | int | 329 (integer division) |
Worse than F1: not only did the chain mean different things across modes, the two results didn't even typecheck (equal(double, int) errors at the == site).
The defect was on the linq side. The old daslib/linq.das:
def average(src : array<auto(TT)>) : TT -const -& {returned the input element type. For array<int> that was int, so (sum / count : int / int) truncated. SQL's AVG always promotes to floating point; C# LINQ's Average(IEnumerable<int>) returns double for the same reason. daslang's linq was the outlier.
Fix: daslib/linq.das average(iterator<TT>) and average(array<TT>) both unconditionally return double and accumulate in double from element zero (uniform path for every numeric TT, no integer overflow hazard up to ~2⁵³, more precise than naive float sum). The private average_impl / average_impl_const helpers were removed — the simple loop is the implementation. The min_max_average family is unchanged (still TT-preserving).
Caller-side fallout: the deliberate "complex average" test case in tests/linq/test_linq_aggregation.das (averaging an array<ComplexType> with custom += and /= uint64 operators) was deleted; users with custom non-numeric types must roll their own averager. All other linq tests had assertion-type updates (equal(query, 2) → equal(query, 2.0lf)). The Mode-1-vs-Mode-2-vs-Mode-3 parity assertion M1 ≡ M2 ≡ M3 now holds against double 329.8 everywhere. parity_check_13_aggregates.das flipped from divergence-pin to equality assertion.
Tutorial 17. The two linq_boost predicate base classes (AstCallMacro_LinqPred2 / AstCallMacro_LinqPredII2) synthesize a $(_ : T) => … lambda for the predicate body. Whenever such a chain nested inside another _-binding macro (canonical case: arr |> _where(_.Id._in(other |> _select(_.UserId)))), the inner lambda's _ shadowed the outer's _ and the typer rejected the chain at linq_boost.das:45 — far from the user's call site.
Fix: set VariableFlags.can_shadow on the synthesized _ parameter. The flag was already used elsewhere in linq_boost.das for macro-generated bindings; the predicate path was missing it. Two-line patch in both base classes' visit overrides.
parity_check_17_subqueries.das covers all four operators across the three modes; the previously-pinned failed_parity_check_17_… file has been removed.
The 1-arg _any() / _none() form (test if the subquery is non-empty, lifts to plain EXISTS (...) / NOT EXISTS (...) under _sql) remains Mode-1-only — LinqAny / LinqNone inherit AstCallMacro_LinqPred2 which macro_verifys on a 2-arg shape, so the zero-predicate form errors with expecting any(iterator, predicate) outside _sql. In plain linq the equivalent is length(...) > 0 or the function-form any(...); this is a different surface, not a parity divergence. Out of scope here.
Tutorial 14. Chains like:
arr |> _group_by(_.City) |> _having(_._1 |> length >= 1) |> _order_by(_._0) |> _select(...)Mode 1 (_sql) emitted SELECT ... GROUP BY "City" HAVING ... ORDER BY "City" and ran. Modes 2/3 used to fail to compile with a deep typer error:
error[30304]: no matching functions or generics: _::clone(tuple<string;array<User>> aka TT&, tuple<string;array<User>> aka TT const&) d:/Work/daScript/daslib/builtin.das:999:15 ... instanced from linq`order_by` ... at d:/Work/daScript/daslib/linq.das:413:8
The trigger was the combination _having |> _order_by. Without _having, linq's fusion of _group_by + _order_by + _select(...) followed a path that didn't materialize the grouped array for sorting; once _having was in the chain, post-aggregate filtering forced a real array<tuple<K, array<T>>> to be produced, then sorted. linq's order_by called clone_to_move on the input (daslib/linq.das:413), which needed a generic _::clone for the per-element type — and the synthesized clone for tuple<string, array<User>> was emitted with the wrong constness, so resolution failed inside builtin.das, far from the user's chain.
Initial decision (Boris, 2026-05-02): (b). Add a static_if guard in linq's order_by to surface a clear error pointing at the user's _order_by call site rather than builtin.das:999.
Actual resolution (2026-05-03): (a) — fixed at the compiler level by PR #2563 ("synthesized tuple/variant clone constness"). makeCloneTuple / makeCloneVariant now keep explicitConst=true; the lowering site picks the flavor-per-constness, so _::clone(tuple<K, array<T>>, ...) resolves cleanly. linq's clone_to_move no longer hits the deep error; no static_if guard is needed.
Coverage (tests/dasSQLITE/):
Every tutorial under tutorials/sql/01..41 has been audited. Tests live under tests/dasSQLITE/parity_check_<NN>_*.das. v2/v2.1 multi-Q parity coverage (14b, 15b, 11b/c/d) added 2026-05-03; F3 pin flipped to positive parity in the same sweep.
| Tutorial | File | Coverage |
|---|---|---|
| 04 select_all | parity_check_04_select_all.das | full row, _where, _select scalar |
| 05 parametrized | parity_check_05_parametrized.das | _first, _first_opt miss/hit |
| 06 error_handling | parity_check_06_error_handling.das | _try_sql variants |
| 07 anatomy | parity_check_07_anatomy.das | symmetric _where, chained AND, starts_with |
| 08 where | parity_check_08_where.das | OR+string, ends_with, contains, to_lower, length, abs |
| 09 select | parity_check_09_select.das | named-tuple, renamed, named-tuple+_first |
| 10 order_by | parity_check_10_order_by.das | single ASC/DESC, tuple-key, where |
| 11 take_skip | parity_check_11_take_skip.das | F1 RESOLVED (multi-Q); + 3 take |
| 11 take | where (multi-Q) | parity_check_11b_take_then_where.das |
| 11 take | order (multi-Q) | parity_check_11c_take_then_order.das |
| 11 skip | where (multi-Q) | parity_check_11d_skip_then_where.das |
| 12 distinct | parity_check_12_distinct.das | full-row, single-col, where |
| 12b set_ops | parity_check_12b_set_ops.das | union, intersect, except |
| 13 aggregates | parity_check_13_aggregates.das | F2 RESOLVED — average promotes to double in all modes |
| 14 group_by | parity_check_14_group_by.das | _group_by, _having, _order_by, multi-key, full reporting chain |
| 14 having|order_by | parity_check_14_order_after_group.das | F3 RESOLVED — inline _order_by after _having works in all modes |
| 14 group_by (multi-Q) | parity_check_14b_aggregate_then_filter.das | aggregate-then-filter (v2 multi-Q lowering) |
| 15 join | parity_check_15_join.das | inner join, left/right filter |
| 15 multi-join (multi-Q) | parity_check_15b_multi_join.das | three-table multi-join with outer WHERE (v2.1) |
| 16 left_join | parity_check_16_left_join.das | _left_join + Option projection |
| 17 subqueries | parity_check_17_subqueries.das | _in/_not_in/_any(pred)/_none(pred) |
| 18 null_handling | parity_check_18_null_handling.das | round-trip, is_some, is_none, unwrap_or |
| 23 foreign_keys | parity_check_23_foreign_keys.das | _where |
| 24 indexes | parity_check_24_indexes.das | _where + _order_by_descending + _to_array (int64) |
| 26 custom_types | parity_check_26_custom_types.das | enum filter (Guid+DateTime+enum), Option |
| 28 json | parity_check_28_json.das | JSON path WHERE/SELECT/count, nested struct path |
| 32 sql_functions | parity_check_32_sql_functions.das | [sql_function] transparency |
| 35 streaming | parity_check_35_streaming.das | _each_sql ≡ _sql ≡ M2 ≡ M3 |
| 36 attach | parity_check_36_attach.das | with_attached, manual attach + with_schema |
| 40 fts5 | parity_check_40_fts5.das | text_match AND/prefix/phrase/OR/NEAR |
| # | File | Why skipped |
|---|---|---|
| 01 | version.das | metadata, no chain |
| 02 | insert_data.das | write-side only |
| 03 | last_row_id.das | write-side only |
| 19 | update.das | write-side only |
| 20 | delete.das | write-side only |
| 21 | upsert.das | write-side only |
| 22 | transactions.das | transaction lifecycle, no chain |
| 25 | defaults_computed.das | DDL-only |
| 27 | blob.das | adapter rail, no new chain |
| 29 | column_names.das | column_info is compile-time, raw PRAGMA otherwise |
| 30 | list_tables.das | raw query("SELECT … FROM sqlite_master") |
| 31 | views.das | DDL chain inside _create_view; no plain-array analog |
| 33 | pragma.das | connection admin, no chain |
| 34 | backup_vacuum.das | DB maintenance, no chain |
| 37 | bulk_operations.das | write-side perf; only chain is `_where |
| 38 | concurrency.das | thread/lock patterns, no new chain |
| 41 | triggers.das | raw exec("CREATE TRIGGER …"), no chain |
| ID | Tutorial | Decision | Status |
|---|---|---|---|
| F1 | 11 take_skip | multi-Q lowering — take | skip and latent siblings (take | where, take | order, skip | where) all preserve linq positional semantics | resolved |
| F2 | 13 aggregates | (a) fix daslib/linq.das average → always double | resolved |
| F3 | 14 group_by | (a) compiler-side fix for synthesized tuple/variant clone constness | resolved (PR #2563) |
| F4 | 17 subqueries | set can_shadow on synthesized _ in AstCallMacro_LinqPred2/LinqPredII2 | resolved (issue #2559) |
| G1 | 16 left_join | add _right_join / _full_outer_join / _cross_join | issue #2558 |
| (lang) | 14 group_by | strongly-named tuple support (recordNames in cache key) | resolved (PR #2565, 2026-05-03) |
| Back | FazBrowse Home | New Git URL |