1. pgBackRest version:

pgBackRest 2.59.0

The issue was originally observed multiple times with pgBackRest 2.56.0 and was reproduced after upgrading Production and DR to pgBackRest 2.59.0.

2. PostgreSQL version:

PostgreSQL 17

3. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each:

Production database server:

• Red Hat Enterprise Linux 9
• PostgreSQL 17
• pgBackRest 2.59.0
• Local POSIX backup repository
• Database size approximately 7.7-8.3 TB

DR standby:

• Red Hat Enterprise Linux 9
• PostgreSQL 17
• pgBackRest 2.59.0

DEV:

• Red Hat Enterprise Linux 9
• PostgreSQL 17
• pgBackRest 2.59.0

4. Did you install pgBackRest from source or from a package?

Package.

pgBackRest 2.59.0 was installed using the official PGDG RHEL 9 RPM:

pgbackrest-2.59.0-1PGDG.rhel9.8.x86_64

5. Please attach the following as applicable:

Relevant pgBackRest configuration used when the failures occurred:

[global]
repo1-path=/backup/<prod-stanza>
repo1-retention-full=1
process-max=8
delta=y
hardlink=y
repo1-bundle=y
repo1-block=y
checksum-page=y
start-fast=y
log-level-console=info
log-level-file=detail
compress-level=5
compress-type=zst
protocol-timeout=600

archive-async=y
archive-timeout=60
spool-path=/var/lib/pgbackrest/spool
archive-push-queue-max=2GB

[<prod-stanza>]
pg1-path=/data/postgres/17

Relevant PostgreSQL settings:

archive_command = pgbackrest --stanza=<prod-stanza> archive-push %p >> /log/17/pgbackrest_archive.log 2>&1
archive_mode = on
listen_addresses = *
max_wal_senders = 10
wal_level = replica
port = 5432

WAL archiving itself is working normally.

I can also provide/attach:

• pgBackRest DEBUG log from the failed pgBackRest 2.59.0 backup
• Relevant PostgreSQL log entries around the failure
• Additional TRACE logging if required

At the timestamps of the failures, we found no corresponding:

• PostgreSQL error indicating corruption
• OOM event
• segmentation fault
• pgBackRest core dump
• filesystem error
• storage/kernel I/O error

6. Describe the issue:

We are experiencing repeatable incremental backup failures with:

ERROR: [042]: raised from local-N protocol: unexpected eof
[FileReadError] on 2 retries: unexpected eof

The failures consistently occur very late in the backup, typically around 99.9%-99.99% complete.

The issue was originally reproduced multiple times with pgBackRest 2.56.0.

We upgraded both Production and DR to pgBackRest 2.59.0, but the same issue continued.

The database is approximately 7.7-8.3 TB.

Failure with pgBackRest 2.59.0 and process-max=3

After upgrading to pgBackRest 2.59.0, we ran the Production incremental backup with:

PGOPTIONS="-c idle_session_timeout=0" \
pgbackrest \
  --stanza=<prod-stanza> \
  --process-max=3 \
  --type=incr \
  backup

The backup progressed normally until approximately 99.99%.

Immediately before the failure, the workers were processing small files using repository bundles.

Example:

2026-08-12 10:31:37.773 P02 DETAIL: backup file .../95219_fsm (bundle 258/456212, 1.4MB, 99.99%)

2026-08-12 10:31:39.155 P03 DETAIL: backup file ... (bundle 268/..., 99.99%)

2026-08-12 10:31:39.307 P01 DETAIL: backup file .../1625043094_fsm (bundle 269/3452763, 1.4MB, 99.99%)

2026-08-12 10:31:53.010 P00 ERROR: [042]: raised from local-2 protocol: unexpected eof
                                    [FileReadError] on 2 retries from 106-15138ms: unexpected eof

2026-08-12 10:31:53.011 P00 INFO: backup command end: aborted with exception [042]

An interesting observation is that local worker P02 stopped producing output while P01 and P03 continued processing bundle files.

Approximately 15 seconds later the parent process reported the unexpected EOF.

Test with process-max=1

To determine whether the issue was related to parallel backup workers, we repeated the Production incremental using only one worker:

pgbackrest \
  --stanza=<prod-stanza> \
  --process-max=1 \
  --type=incr \
  --log-level-file=debug \
  backup

This also failed near completion with the same error.

Relevant output:

2026-08-12 15:54:44.806 P01 DETAIL: backup file .../1934172120_fsm (bundle 262/1026425, 1.3MB, 99.97%)

2026-08-12 15:54:44.806 P01 DETAIL: backup file .../1493220600_vm (bundle 262/1028108, 368KB, 99.97%)

2026-08-12 15:54:59.820 P00 DEBUG:
job: {
    state: done,
    key: {263},
    command: bp-f,
    result: null,
    code: 42,
    message: {
        "raised from local-1 protocol: unexpected eof
        [FileReadError] on 2 retries from 12-15017ms: unexpected eof"
    }
},
bundle: true,
pageSize: 8192,
sizeTotal: 8309323133409,
sizeProgress: 8306825712089,
currentPercentComplete: 9994

2026-08-12 15:54:59.821 P00 ERROR: [042]: raised from local-1 protocol: unexpected eof
                                    [FileReadError] on 2 retries from 12-15017ms: unexpected eof

2026-08-12 15:54:59.822 P00 INFO: backup command end: aborted with exception [042]

The diagnostic block generated by pgBackRest was:

version: 2.59.0
command: backup

options:
--archive-timeout=60
--checksum-page
--compress-level=5
--compress-type=zst
--delta
--log-level-console=info
--log-level-file=debug
--pg1-path=/data/postgres/17
--process-max=1
--protocol-timeout=600
--repo1-block
--repo1-bundle
--repo1-path=/backup/<prod-stanza>
--repo1-retention-full=1
--stanza=<prod-stanza>
--start-fast
--type=incr

stack trace:
command/backup/backup.c:backupJobResult:1611
command/backup/backup.c:backupProcess
command/backup/backup.c:cmdBackup
main.c:main

Since the same issue was reproduced with process-max=1, the number of parallel workers does not appear to be the primary trigger.

DEV test with repository bundling and hardlinking disabled

We then tested pgBackRest 2.59.0 in DEV with repository bundling and hardlinking disabled:

pgbackrest \
  --stanza=<dev-stanza> \
  --process-max=10 \
  --repo1-bundle=n \
  --repo1-hardlink=n \
  --type=incr \
  --log-level-file=debug \
  backup

The backup started with the following effective options:

--process-max=10
--no-repo1-bundle
--no-repo1-hardlink

This backup completed successfully:

2026-08-12 14:29:49.958 P00 INFO: backup command begin 2.59.0
...
2026-08-12 15:17:01.473 P00 INFO: new backup label = 20260811-144526F_20260812-142949I

2026-08-12 15:17:04.892 P00 INFO: incr backup size = 244.6GB, file total = 176915

2026-08-12 15:17:04.926 P00 INFO: backup command end: completed successfully

We subsequently restored/reloaded the DEV PostgreSQL environment using this new unbundled backup.

The restore completed successfully and no issues were observed.

Production test with repository bundling and hardlinking disabled

We then performed the same test in Production with:

repo1-bundle=n
hardlink=n

The Production incremental completed successfully.

Since applying this workaround, Production has completed multiple consecutive backups successfully.

Successful Production backups after workaround

Incremental:

backup: 20260809-170001F_20260812-160109I

timestamp:
2026-08-12 16:01:09 / 2026-08-12 18:34:51

database backup size: 1429.5GB
repo backup size: 265.1GB

SUCCESS

Differential:

backup: 20260809-170001F_20260812-220002D

timestamp:
2026-08-12 22:00:02 / 2026-08-12 23:26:52

database backup size: 1503.2GB
repo backup size: 293.2GB

SUCCESS

Following incremental:

backup: 20260809-170001F_20260813-051501I

timestamp:
2026-08-13 05:15:01 / 2026-08-13 06:55:10

database backup size: 394GB
repo backup size: 89.7GB

SUCCESS

The current stanza status is:

status: ok

WAL archiving and PostgreSQL streaming replication also remain healthy.

Test matrix

OS investigation

At the exact failure timestamps we checked:

journalctl
journalctl -k
coredumpctl

We found:

• No OOM kill
• No pgBackRest process kill reported by the kernel
• No segmentation fault
• No core dump
• No filesystem errors
• No SCSI/storage I/O errors

We also checked process/file descriptor limits and found sufficient headroom.

Current workaround

Our current Production configuration is:

process-max=6
delta=y
hardlink=n
repo1-bundle=n
repo1-block=y
checksum-page=y

Backups are currently completing successfully with this configuration.

The main disadvantage of the workaround is significantly increased repository storage consumption.

For example, before the workaround a comparable incremental backup used approximately:

database backup size: 382.2GB
repository backup size: 25.9GB

With bundling/hardlinking disabled, a recent incremental used:

database backup size: 394GB
repository backup size: 89.7GB

Therefore, we would prefer to understand and resolve the underlying issue rather than permanently operate with bundling disabled.

Questions

Could this behavior be related to repository bundle processing, hardlink interaction, or the bundle/block processing path?

The strongest observations are:

1. The failure occurs repeatedly near 99.9%-99.99%.
2. The failure occurs while bundle processing is active.
3. The failure reproduces with process-max=3.
4. The failure also reproduces with process-max=1.
5. No corresponding OOM, segfault, core dump, or storage/kernel error is recorded.
6. Disabling bundle/hardlink allows the same environment to complete backups successfully.
7. DEV succeeds with process-max=10 when bundle/hardlink are disabled.
8. Multiple subsequent Production backups have completed successfully with the workaround.
9. A DEV restore from the new unbundled backup also completed successfully.

Is there any additional TRACE logging, diagnostic option, or specific test that you would recommend to determine why the local backup worker terminates and the parent process receives [042] unexpected eof during the final portion of the backup?

We can provide the complete DEBUG log from the failed pgBackRest 2.59.0 process-max=1 backup if required.