Console output sync stalls permanently if the node's line count goes backwards

Summary

Task.process() asks the processing node for console output starting at an
offset it derives from the console it has already stored. If the node's output
ever becomes shorter than that offset, NodeODM returns an empty list with no
error, the stored console never grows, the offset never changes, and console
sync is stuck permanently — while the task itself keeps processing normally.

The UI gives no indication of this. To the user the task looks frozen: the
console stops at a timestamp minutes or hours in the past, while processing_time
keeps ticking up.

Where

app/models/task.py (still present on master as of 2026-09-21):

if not self.console.output():
    current_lines_count = 0
else:
    current_lines_count = len(self.console.output().split("\n"))

info = self.processing_node.get_task_info(self.uuid, current_lines_count)

The offset is derived purely from local state and is monotonically
non-decreasing. Nothing validates it against the node.

self.console.reset() exists, but is only called on the RESTART pending
action path, i.e. only when WebODM itself initiates the restart.

Why the node can go backwards

NodeODM's /task/<uuid>/output?line=N returns the lines from N onwards of
the current run. Verified on NodeODM (WebODM 2.x desktop, node on
127.0.0.1:29301), on a task with ~7000 lines of output:

line=0       -> 6988 lines
line=5000    -> 1988 lines
line=999999  ->    0 lines      <- no error, empty list

So any event that makes the node's current output shorter than WebODM's stored
console produces the stall. Ways this happens in practice:

• the task is re-run on the node (--rerun-from), which resets its output
• WebODM's own processes restart mid-task and a restart is issued around the
  same time, so the RESTART pending action's console.reset() does not run
  but the node does restart the task
• the node's task data is recreated

Observed

WebODM 2.x desktop on Windows, ODM 3.x, single local NodeODM, one 412-image
task.

node (doing the work)      last line 16:14:52   <- live
WebODM (showing it)        last line 16:06:59   <- frozen, 8 minutes behind

processing_time kept increasing the whole time (51.3 min and rising), and
running_progress kept being written, so only the console was affected — which
makes it harder to notice, because the task does not look dead.

Manually truncating the stored console made sync recover on the next cycle
(~5 s), and the console refilled from the current run's line 0.

Suggested fix

Validate the offset against the node instead of trusting local state. The
cheapest version: if the node returns no lines while the task is RUNNING and
the stored console has not grown for N cycles, re-request from line 0 and, if
the node's total is smaller than the stored console, reset and refill.

Sketch:

info = self.processing_node.get_task_info(self.uuid, current_lines_count)

if len(info.output) == 0 and current_lines_count > 0:
    # The node may have restarted the task: its line numbering is back to
    # zero and our offset now points past the end. Re-sync from scratch.
    full = self.processing_node.get_task_info(self.uuid, 0)
    if len(full.output) < current_lines_count:
        self.console.reset()
        info = full

That costs one full fetch only in the stalled case, and nothing in the normal
path.

An alternative that avoids the extra request entirely would be for NodeODM to
include the current total line count in /task/<uuid>/info, letting WebODM
compare cheaply on every poll.

Workaround for anyone hitting this

Truncate the task's stored console; sync resumes on the next cycle:

from app.models import Task
for t in Task.objects.filter(status=20):
    t.console.reset()