Fixes deletes, drops, and alter() renames that removed data they shou…

…ld have kept.

A patch release on the 2.3 line, recommended for every 2.3 deployment. There are no API changes.

## Deletes

- A cascade delete removes only the rows it reaches (#1589, #1590, #1591; #1593):
  - `(T & dj.Top(n)).delete()` and `delete_quick()` start from the rows the `Top` selects. Before, they started from every row matching the other conditions.
  - A child whose nullable foreign key shares a primary-key column with the parent keeps the rows whose foreign key is NULL. In a diamond, a child reached through a secondary foreign key is now deleted, instead of the delete failing with an "unloaded schema" error.
  - With `part_integrity="cascade"`, a master joins the delete only if one of its part rows does.

  On earlier releases, restrict the delete by the keys it should remove: `(T & (T & cond).keys()).delete()`.

## Tables

- On PostgreSQL, dropping a table no longer drops columns of other tables that share an enum type (#1630; #1633). Every jobs table has the same `status` enum, so dropping one jobs table used to remove `status` from all the others in the schema.
- `alter()` keeps an attribute's values when it renames a core-type attribute (#1631; #1635). On MySQL the rename is a `CHANGE`. On PostgreSQL, `alter()` refuses the rename before changing anything; renaming there comes in 2.4.
- `alter()` resolves foreign keys in the module that declared the table, not the caller's (#1602; #1618).

## Jobs

- A restricted `populate(reserve_jobs=True)` keeps other workers' reservations and error records (#1594; #1597). Cleanup checks keys against the full `key_source`; the restrictions only limit which jobs are added. Success and error records of deleted rows go on every refresh.
- `completed_time` is recorded to the millisecond, so it can no longer precede `reserved_time` (#1600; #1614).

## Errors

- MySQL error 1143 (SELECT denied on a column) raises `dj.errors.AccessError`, as error 1142 already did (#1586; #1599). Contributed by @Bhanu73120.

## Python

- Python 3.15 is supported: `requires-python` is now `>=3.10,<3.16` (#1598). On 3.15 only, `deepdiff` stays below 9 until its dependencies publish 3.15 wheels.

## Known limitations, fixed in 2.4

- A delete restricted by another query, such as `(Subject & (Session & "quality = 0")).delete()`, raises or deletes nothing, and so does one whose restriction contains `%` (#1610, #1636). Restrict by the keys instead: `(T & (T & cond).keys()).delete()`.
- `Diagram.restrict()` ignores a `dj.Top` seed and shows every matching row. This affects the preview only, not deletes.

A patch release on the 2.3 line, focused on where data came from befo…

…re it reached the pipeline.

A patch release on the 2.3 line, focused on where data came from before it reached the pipeline.

**Rows that enter the pipeline can now record where they came from.** With `provenance.capture` enabled, a `dj.Manual` table carries a hidden `_prov` attribute, declared with the table and filled on insert. It is **off by default**, like `jobs.add_job_metadata` and for the same reason: it changes the DDL of every entry table declared afterwards, so that is a deployment's decision rather than a library default. Inside a pipeline provenance is structural — a computed row cannot exist without its declared upstream, so the foreign-key graph is the lineage — but at the boundary that runs out, and until now each pipeline invented its own record of external origin. `_prov` gives the boundary one shape, so that "which rows have no recorded origin" is a query rather than an audit. **No author writes it:** `insert()` takes no provenance argument, and the content comes from deployment configuration (`provenance.source`), from the connection, and — for an insert inside a `make()` — from the ingesting table and key, which is what makes the fan-out ingestion pattern traceable without a foreign key. A table declared before capture was enabled records nothing until `dj.deploy.add_prov_column` adds the slot. See [Extrinsic Provenance at Entry Tables](https://docs.datajoint.com/reference/specs/boundary-provenance/) and [Record Data Origin](https://docs.datajoint.com/how-to/record-data-origin/).

**Three tiers gain a second name.** `dj.Entry`, `dj.Ingest` and `dj.Compute` join `dj.Manual`, `dj.Imported` and `dj.Computed`. Each pair is one class rather than a subclass, so a table declared either way is identical — same SQL prefix, same tier detection, same `describe()` and diagram — and both names are permanent, with no deprecation and nothing to migrate. `dj.Lookup` and `dj.Part` are unchanged. The documentation keeps the original names for now; adopting the new ones as primary is slated for 2.4 ([#1546](#1546)).

**Codecs receive connection context explicitly.** `encode()` and `decode()` now take a `context` argument carrying `schema`, `table`, `field` and `config`, so `key` means what it means everywhere else — the primary key. Passing the calling connection's `config` is what matters: without it, path and backend resolution fall back to the global `dj.config`, which in a process holding connections for several users belongs to none of them and resolves a different store silently. Codecs written before 2.3.4 keep working unchanged; DataJoint passes `context` only to codecs whose signature declares it.

**Two PostgreSQL defects in the hidden columns are fixed.** Every platform-managed column — `_job_start_time`, `_job_duration`, `_job_version`, `_prov`, `_singleton` — is now declared in DataJoint notation and compiled through the type system, so each backend's mapping comes from one place. `_job_start_time` had been declared as a bare `timestamp` and so took microsecond precision on PostgreSQL where MySQL got `datetime(3)`, and `migrate.add_job_metadata_columns` built its `ALTER` with backtick-quoted identifiers, which is a syntax error there — the retrofit had never run on PostgreSQL at all. Existing tables are untouched; only new declarations and retrofits change.

> **Upgrading from 2.3.3?** No API breaks, and no DDL change: every new setting is off by default, so an unchanged schema declares under 2.3.4 exactly what it declared under 2.3.3. Reading a hidden attribute still requires SQL — `to_arrays('_prov')` and `proj('_prov')` raise, and a supported accessor arrives in 2.4 ([#1562](#1562)).

Full details in [What's New in DataJoint 2.3](https://docs.datajoint.com/about/whats-new-23/).

---

## What's Changed

- Pull MinIO from a registry that still serves it(#1559)@dimitri-yatsenko

## ⚡️ Enhancements

- Provenance capture defaults to off(#1568)@dimitri-yatsenko
- Codec `context` argument, additive to `key`(#1556)@dimitri-yatsenko
- Hidden `_prov` attribute for extrinsic provenance at Entry tables(#1555)@dimitri-yatsenko
- dj.Entry, dj.Ingest and dj.Compute as tier names on one axis(#1558)@dimitri-yatsenko

## 🐛 Bug Fixes

- Declare every platform-managed column through the DataJoint type system(#1565)@dimitri-yatsenko

**Full Changelog**: v2.3.3...v2.3.4

Released: 2026-10-04
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.3.4 (was a lightweight tag)

A patch release on the 2.3 line, focused on storage credentials, diag…

…ram rendering, and type...

A patch release on the 2.3 line, focused on storage credentials, diagram rendering, and type validation.

**S3 stores no longer require static credentials.** `access_key` and `secret_key` are now optional in an `s3` store spec — omit both and the AWS credential chain resolves an ambient identity (EC2 instance profile, IRSA, ECS task role, SSO), matching how the `gcs` and `azure` protocols already behaved. This is what lets a pipeline run under an assumed role with no long-lived keys in its configuration. Setting exactly one of the two is now rejected at validation with a clear message instead of failing later inside the AWS client. See [Configure Storage](https://docs.datajoint.com/how-to/configure-storage/).

**Diagrams were redesigned.** `dj.Diagram` output moves to the DataJoint brand palette — Manual green, Lookup grey, Imported blue, Computed orange, with tier shapes unchanged — and renamed foreign keys are now drawn in amber, distinct from the Computed tier. Line weight now encodes **cardinality and only cardinality**: thick for one-to-one, thin for one-to-many. It previously also tried to signal the master-part relationship, which made the two unreadable together. A new `display.diagram_theme` setting takes `auto` (default), `light`, or `dark`; `auto` emits a single SVG that adapts to the viewer's light or dark mode. See [Read Diagrams](https://docs.datajoint.com/how-to/read-diagrams/) and [Configuration](https://docs.datajoint.com/reference/configuration/).

**Attribute types are validated at declaration and insert.** Four related gaps closed, the most serious being that inserting a NumPy array into a *native* `blob` attribute silently stored the array's text representation, with no error on insert and none on fetch. `decimal(M,D) unsigned` is also accepted again after being rejected in 2.x, and misspelled native types are now caught at declaration rather than producing invalid DDL at the server.

Also: braces in a table or attribute comment no longer crash declaration, `alter()` no longer loses an added attribute's declared type, file-protocol stores are safe on Windows, and non-unique foreign-key columns are indexed on PostgreSQL.

> **Upgrading from 2.3.2?** No API breaks. Two of the fixes above do reject input that previously passed silently — a misspelled native type at declaration, and a NumPy array inserted into a native `blob`. Both cases were already broken rather than working: the first produced invalid DDL at the server, the second corrupted the stored value. Anything affected was losing data.

Full details in [What's New in DataJoint 2.3](https://docs.datajoint.com/about/whats-new-23/).

---

## What's Changed

- docs: fix missing pixi -e flags in CONTRIBUTING.md(#1541)@gtouloumes
- ci: add Windows unit-test job to catch OS-specific path bugs(#1522)@dimitri-yatsenko

## ⚡️ Enhancements

- feat(#<!---->1543): adopt the DataJoint brand tier colors in diagram themes(#1544)@dimitri-yatsenko
- feat(#<!---->1532): modernize dj.Diagram rendering style(#1534)@dimitri-yatsenko
- fix(#<!---->1537): allow s3 stores without static credentials(#1538)@dimitri-yatsenko
- fix(#<!---->1512): index foreign-key columns on PostgreSQL (coverage-aware)(#1526)@dimitri-yatsenko

## 🐛 Bug Fixes

- fix: validate attribute types at declaration and insert (#<!---->1527, #<!---->1528, #<!---->1529, #<!---->1530)(#1531)@dimitri-yatsenko
- fix: braces in comments crash declaration, and alter() drops type metadata(#1548)@MilagrosMarin
- fix(diagram): collapsed-to-collapsed edges carry no per-foreign-key style(#1545)@dimitri-yatsenko
- fix(#<!---->1535): render diagrams with unresolved (raw table-name) nodes on PostgreSQL(#1536)@dimitri-yatsenko
- fix(#<!---->1532): diagram edge weight encodes cardinality, not master-part(#1533)@dimitri-yatsenko
- Use POSIX separators in file-protocol storage URLs and paths(#1521)@lochhh

## 📝 Documentation

- docs(codecs): thread the connection config in the SchemaCodec example(#1549)@MilagrosMarin
- docs(settings): correct the config-precedence docstring(#1542)@dimitri-yatsenko

**Full Changelog**: v2.3.2...v2.3.3

Released: 2026-09-09
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.3.3 (was a lightweight tag)

v2.3.2 is a patch release — internal hardening and bug fixes on top o…

…f v2.3.1, with no breaking...

## Overview

**v2.3.2** is a patch release — internal hardening and bug fixes on top of v2.3.1, with no breaking changes to the public API. The main items: the dependency graph moved to `networkx.MultiDiGraph` (correctly modeling multiple foreign keys between the same pair of tables), a `~jobs`-table fix for UUID foreign-key-derived primary keys, removal of the process-wide `sys.excepthook` (uncaught exceptions now use Python's default handler, with tracebacks rendered by the log formatter), and a garbage-collection fix that avoids acting on partial storage listings. The package version is now derived from the git tag via hatch-vcs.

---

- test: cover parallel foreign-key edges to the same parent (#<!---->1492)(#1514)@dimitri-yatsenko
- tests: add coverage for deferred audit findings + isolate test\_settings from ambient env(#1513)@MilagrosMarin
- chore: normalize stdlib copy imports to `import copy`(#1509)@dimitri-yatsenko
- docs(autopopulate): fix stale finally-comment; document upstream in class Attributes(#1510)@MilagrosMarin
- docs: post-2.3.1 stale-text sweep (3 small drift fixes)(#1511)@MilagrosMarin
- chore(release-flow): stale doc pointers, RELEASE\_MEMO cleanup, and dead badge step(#1504)@MilagrosMarin
- docs(codecs): fix stale storage-path and hash-algorithm docstrings(#1505)@MilagrosMarin
- [github-actions]Update version.py to Release 2.3.1(#1502)@[github-actions[bot]](https://github.com/apps/github-actions)

## ⚡️ Enhancements

- refactor(graph): migrate Dependencies \& Diagram to nx.MultiDiGraph (#<!---->1492)(#1508)@dimitri-yatsenko

## 🐛 Bug Fixes

- Fix jobs table generation for uuid FK-derived primary keys (#<!---->1515)(#1518)@dimitri-yatsenko
- Remove process-wide sys.excepthook; harden log formatter (#<!---->1516)(#1517)@dimitri-yatsenko
- fix(release): derive version from git tag via hatch-vcs (#<!---->1503)(#1507)@dimitri-yatsenko
- fix(gc): don't act on partial storage listings(#1506)@MilagrosMarin

**Full Changelog**: v2.3.1...v2.3.2

Released: 2026-07-21
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.3.2 (was a lightweight tag)

`populate()` performance — `self.upstream` is now built lazily and de…

…pendencies are no longer...

## Highlights

- **`populate()` performance** — `self.upstream` is now built lazily and dependencies are no longer reloaded per key, removing redundant work from `populate()` loops (#1499, #1493).
- **Cascade / part-integrity fix** — a cascaded delete now restricts the master from *every* restricted part and validates part integrity, fixing multi-part-master cascades (#1484).
- **Garbage-collection correctness** — codec-driven reference discovery with file-level, schema-addressed orphan matching (#1479, #1469).
- **`strict_provenance` runtime guardrail retired** — the best-effort runtime guard is removed; provenance remains an intrinsic structural property of the pipeline (#1489, #1488).
- **Python 3.14 support** — supported range extended through 3.14 with min/max exercised in CI; MySQL 8.4 LTS added to the test matrix (#1490, #1498).

## ⚡️ Enhancements

- fix(#<!---->1493): build self.upstream lazily; stop reloading deps per populate key(#1499)@dimitri-yatsenko
- ci: test MySQL 8.0 and 8.4 (LTS) in the matrix; parameterize MySQL image(#1498)@dimitri-yatsenko
- test(cascade): pin part-integrity paths (rollback, empty-materialization, U3 arm)(#1485)@dimitri-yatsenko
- Retire strict\_provenance runtime guardrail (#<!---->1488)(#1489)@dimitri-yatsenko
- Extend Python support to 3.14; test min/max in CI(#1490)@dimitri-yatsenko

## 🐛 Bug Fixes

- fix(cascade): restrict master from every restricted Part; validate part\_integrity(#1484)@dimitri-yatsenko
- ci(pre-commit): pin numpy\<2.2 for mypy stub compatibility(#1491)@dimitri-yatsenko
- fix(#<!---->1469): codec-driven GC reference discovery + file-level orphan matching(#1479)@dimitri-yatsenko

## 📝 Documentation

- docs: clarify \_propagate\_part\_to\_master materialization is delete-order, not just MySQL 1093(#1495)@dimitri-yatsenko
- docs(#<!---->1484): document pair-keyed part-to-master invariant in \_propagate\_restrictions(#1494)@dimitri-yatsenko

**Full Changelog**: v2.3.0...v2.3.1

Released: 2026-07-16
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.3.1 (was a lightweight tag)

DataJoint 2.3 builds on the 2.2.x line with a set of features centere…

…d on provenance — knowing,...

## Highlights

DataJoint 2.3 builds on the 2.2.x line with a set of features centered on **provenance** — knowing, and being able to trust, exactly which upstream data a computed result was derived from.

The headline is the **provenance trinity**. `Diagram.trace` constructs the upstream view of any result; `self.upstream` makes those declared ancestors ergonomic to read inside `make()`; and the opt-in `strict_provenance` flag turns the long-standing "read only from declared dependencies, write only to `self`" convention into something the framework actively checks. Together they move DataJoint's core provenance promise from a convention people hope to follow to one the framework helps construct and — when enabled — enforce.

Around that:

- **SparkAdapter Codec Protocol** — typed codecs can expose their decoded values as Spark-native types, opening lakehouse / Delta-Sharing consumers (e.g. Databricks) to columns that were previously opaque blobs.
- **`dj.deploy.set_replica_identity`** — a new `dj.deploy` module configures PostgreSQL `REPLICA IDENTITY` for change-data-capture pipelines.
- **Cascade fix** — `part_integrity="cascade"` is now correct across Part-of-Part and renamed-foreign-key chains (the same upward-propagation machinery `Diagram.trace` builds on).

**No breaking changes.** `strict_provenance` defaults off and everything else is additive, so existing pipelines are unaffected. See [What's New in 2.3](https://docs.datajoint.com/about/whats-new-23/) for the full narrative.

## 🚀 Features

- feat(#1425): strict\_provenance config flag for runtime enforcement(#1474)@dimitri-yatsenko
- feat(#1424): self.upstream property for pre-restricted ancestor access(#1473)@dimitri-yatsenko
- feat(#1423): Diagram.trace() for upstream restriction propagation(#1471)@dimitri-yatsenko
- feat(#1458): SparkAdapter Codec Protocol(#1472)@dimitri-yatsenko
- feat(deploy): set\_replica\_identity for PostgreSQL CDC (#1447)(#1466)@dimitri-yatsenko

## 🐛 Bug Fixes

- fix(#1429): cascade through FK chain for part\_integrity="cascade"(#1468)@dimitri-yatsenko
- fix(#1454): refresh ~lineage on every `@schema` decoration(#1467)@dimitri-yatsenko
- fix(staged\_insert): converge metadata shape with ObjectCodec.encode(#1465)@dimitri-yatsenko

**Full Changelog**: v2.2.4...v2.3.0

Released: 2026-07-02
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.3.0 (was a lightweight tag)

A small follow-up patch on 2.2.3 cleaning up rough edges discovered d…

…uring the docs review:

## Highlights

A small follow-up patch on 2.2.3 cleaning up rough edges discovered during the docs review:

- **`dj.StorageAdapter` and `dj.get_storage_adapter()` exposed at top level.** The Storage Adapter plugin contract documented in the spec used these names as if they were re-exported from the top-level `datajoint` namespace, but they were not. Plugin authors following the spec literally hit `AttributeError`. Both symbols are now in `datajoint.__all__` and importable as `dj.StorageAdapter` / `dj.get_storage_adapter`.
- **`packaging` declared as an explicit dependency.** `src/datajoint/migrate.py` imports `packaging.version.Version`, but the package was never declared in `pyproject.toml`. With Python 3.12 venvs no longer shipping `setuptools` (which used to pull `packaging` transitively), `pip install datajoint==2.2.3` into a clean Py3.12 venv hit `ModuleNotFoundError: No module named 'packaging'` on `import datajoint`. Fixed.
- **Source-side version markers refreshed** in `settings.py` docstrings and comments to match the public docs.

## ⚡️ Enhancements

- feat: export StorageAdapter and get_storage_adapter at top level (#1463)@dimitri-yatsenko

## 🐛 Bug Fixes

- fix: declare packaging as an explicit dependency (#1462)@dimitri-yatsenko

## 📝 Documentation

- docs: refresh settings.py version markers (#1461)@dimitri-yatsenko

Companion docs: [datajoint/datajoint-docs#172](datajoint/datajoint-docs#172) — env-var configuration of stores + Storage Adapter API spec.

**Full Changelog**: v2.2.3...v2.2.4

Released: 2026-06-05
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.2.4 (was a lightweight tag)

This patch release adds env-var configuration for object stores (clos…

…es the gap for env-var-only...

## Highlights

This patch release adds env-var configuration for object stores (closes the gap for env-var-only deployments such as the DataJoint platform, Kubernetes, and Lambda) and ships several minor fixes and documentation improvements.

- **`DJ_STORES`** — JSON-encoded copy of the `stores` dict, same shape as in `datajoint.json`. Replaces the file's `stores` block when set, with arbitrary adapter-defined attribute names (token, api_key, workspace_url…).
- **`DJ_IGNORE_CONFIG_FILE`** (default `false`) — when `true`, skips `datajoint.json`, the project `.secrets/`, and `/run/secrets/datajoint/` entirely. Hard guarantee that no file on disk leaks into config.
- **`.secrets/stores.<name>.<attr>`** now accepts any attribute, not only `access_key` / `secret_key` — supports plugin-registered adapters (e.g. Databricks Unity Catalog Volumes) with non-S3-style credential fields.

Precedence (high → low): programmatic `dj.config["stores"]` > `DJ_STORES` > `datajoint.json` > `.secrets/stores.<name>.<attr>` (fills missing attrs only).

## ⚡️ Enhancements

- feat(config): DJ_STORES env var + DJ_IGNORE_CONFIG_FILE flag + arbitrary `.secrets/` attrs (#1452)@dimitri-yatsenko
- chore: remove .vscode and .devcontainer (#1459)@dimitri-yatsenko

## 📝 Documentation

- docs: extend datajoint.migrate timeline to 2.4 or 2.5 (#1456)@dimitri-yatsenko
- docs: fix code formatting on rendered codecs API page (#1455)@dimitri-yatsenko
- docs: use relative path for README pipeline image (#1450)@MilagrosMarin
- docs: refresh README pipeline illustration (#1448)@dimitri-yatsenko
- docs: cite DataJoint 2.0 manuscript and add RRID (#1449)@dimitri-yatsenko

Companion docs: [datajoint/datajoint-docs#172](datajoint/datajoint-docs#172)

**Full Changelog**: v2.2.2...v2.2.3

Released: 2026-06-05
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.2.3 (was a lightweight tag)

feat: add StorageAdapter plugin system for third-party storage protoc…

…ols (#1432)

## ⚡️ Enhancements

- feat: add StorageAdapter plugin system for third-party storage protocols (#1432) @kushalbakshi
- refactor: simplify index query, warn on MariaDB connections (#1439) @dimitri-yatsenko
- feat: export AutoPopulate and Job in public API (#1431) @dimitri-yatsenko

## 🐛 Bug Fixes

- fix(#1442): scan_*_references reads raw JSON metadata instead of decoded codec output (#1444) @kushalbakshi
- fix(#1438): preserve json flag for MariaDB longtext-aliased columns (#1443) @dimitri-yatsenko
- fix(#1433): clear error for leading-underscore attribute names (#1441) @dimitri-yatsenko
- fix: remove EXPRESSION column dependency in get_indexes_sql() (#1437) @dimitri-yatsenko
- fix: preserve NOT NULL constraint in migrate_columns() (#1435) @dimitri-yatsenko

**Full Changelog**: v2.2.1...v2.2.2

Released: 2026-05-01
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.2.2 (was a lightweight tag)

feat: add `database.name` setting for PostgreSQL connections (#1426)

**Released: 2026-04-09**

## ⚡️ Enhancements

- feat: add `database.name` setting for PostgreSQL connections (#1426) @kushalbakshi
  - Configurable database name via `dj.config["database.name"]`, `DJ_DATABASE_NAME` env var, or `database_name` kwarg on `Connection`
  - Extracts `_build_connect_kwargs()` helper to eliminate duplicated connection parameter construction
  - `__repr__` and logger show database name when set (e.g., `user@host:5432/my_db`)
  - Warns when `database.name` is set with MySQL backend
  - Deprecates `database_prefix` (warning emitted at `Schema.activate()`)

## 📝 Documentation

- Fix CONTRIBUTING.md link for PyPI (#1422) @dimitri-yatsenko
- Add PostgreSQL configuration to CONTRIBUTING.md
- Document `database.name` setting and `database_prefix` deprecation ([datajoint-docs#159](datajoint/datajoint-docs#159))
- Fix dead links in llms.txt — now auto-generated from mkdocs nav ([datajoint-docs#160](datajoint/datajoint-docs#160))

**Full Changelog**: v2.2.0...v2.2.1

Released: 2026-04-09
Retroactive: re-tagged 2026-10-20 from the GitHub release v2.2.1 (was a lightweight tag)