| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
The tracing JIT this image enables (opcache.jit=1255, 128M buffer) can emit machine code that loops forever, wedging every php-fpm worker at 100% CPU until the container is restarted. See linuxserver#539. PHP ships with the JIT off by default; Nextcloud's own tuning docs recommend tracing with an 8M buffer. 1255 is not a documented preset (tracing is 1254).
There was a problem hiding this comment.
Thanks for opening this pull request! Be sure to follow the pull request template!
Sorry, something went wrong.
Added changelog entry for disabling OPcache JIT due to performance issues.
|
This pull request has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions. |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Description:
Sets opcache.jit=disable and opcache.jit_buffer_size=0 in 00_opcache.ini, instead of
opcache.jit=1255 / opcache.jit_buffer_size=128M. Same two lines in Dockerfile and
Dockerfile.aarch64, plus a changelog entry in readme-vars.yml.
Benefits of this PR and context:
Closes #539 (and very likely #536, which is the same failure with a different trigger).
Under sustained traffic the tracing JIT can emit machine code that never makes forward progress.
I caught an instance in the failed state and attached gdb to the stuck workers before restarting it:
the PHP VM instruction pointer (EG.current_execute_data->opline) is frozen at the same address in
every worker while their CPU time keeps climbing, and the CPU instruction pointer is inside the
shared opcache JIT buffer. The PHP call chain the workers are frozen in contains no loop at all, so
the emitted machine code isn't doing what the PHP says. Full write-up with the captures is
here.
Because the JIT buffer is shared memory, once bad code is emitted every worker reaching that path
hangs, including workers forked afterwards. That's why only a full restart clears it, and why it
comes back at random days later.
Worth noting what we currently ship: 1255 is not a documented preset. The named tracing mode is
1254; 1255 is the same tracing JIT at optimisation level 5 ("optimise whole script"). PHP itself
ships with the JIT off (on 8.4 that is opcache.jit=disable), and Nextcloud's own server tuning
docs recommend tracing with an 8M buffer, noting most instances use under 2 MiB. We allocate
128M. Nextcloud is dominated by database I/O rather than computation, so the JIT buys very little
here. OPcache itself is untouched and keeps doing the useful work.
How Has This Been Tested?
PRs no longer auto-build, so I could not test a built image. Instead I ran the equivalent override
at runtime (/config/php/php-local.ini, read after conf.d, so it wins) across the four instances
I administer. All four run 34.0.3-ls448 and Nextcloud 34.0.3, so the JIT setting is the only
variable between them.
Three patched on 16 Aug, one left unpatched by accident:
The mapping check is the objective part, from /proc/<worker>/maps. Unpatched:
Patched: the r-xs line is simply gone, only the data segments remain. So the failure mode is
removed structurally, not just reset the way a plain restart does.
On the patched instance that used to fail most often, php-fpm has restarted 8 times since 16 Aug,
including the weekly logrotate restarts on 23 and 30 Aug (s6-svc -t in
/etc/logrotate.d/php-fpm, a full pool restart) and the image update to ls448 on 2 Sep. Those
restarts are exactly what used to set the failure off, and there has been no recurrence. Before the
change it was roughly one incident every two or three restarts. The unpatched one failed again on
26 Aug and 2 Sep, the latter about seven hours after Watchtower restarted its pool.
Two other users in #539 report the same result independently after applying the same override.
I set both values explicitly rather than deleting the two lines, because it is the combination that
has been running for three weeks, and because the PHP default has already moved once (8.3 → 8.4
changed from jit_buffer_size=0 to opcache.jit=disable), so an explicit value will not silently
change meaning again.
Source / References:
this comment)
rather than a fix: it stops one runaway worker from taking the whole pool down, it does not
prevent the miscompilation.
for the CRTO values and the named presets
for the recommended JIT settings
Strictly the miscompilation is a PHP bug and the real fix belongs upstream, but nobody in #539 has
managed to reproduce it on demand (it needs the JIT to decide the path is hot, which takes hours of
real traffic). In the meantime this stops opting users into a non-default, undocumented JIT level.