FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix(mavlink): stop sizing the receive thread stack by sizeof(MavlinkReceiver) by gguidone · Pull Request #28124 · PX4/PX4-Autopilot · GitHub

fix(mavlink): stop sizing the receive thread stack by sizeof(MavlinkReceiver) - #28124

Merged
MaEtUgR merged 6 commits into
mainfrom
gg/mavlink-receiver-stack
Aug 5, 2026
Merged

MaEtUgR merged 6 commits into
mainfrom
gg/mavlink-receiver-stack

Conversation

Copy link
Copy Markdown
Contributor

Solved Problem

Fixes #28109.

MavlinkReceiver::start() sizes the receive thread's stack as sizeof(MavlinkReceiver) + 2840, but the object is not on that stack: it is a member of the heap-allocated Mavlink (mavlink_main.h:517), and pthread_create is handed only (void *)this. On NuttX thread stacks come out of the heap pool, so every MAVLink instance has been reserving a second copy of the object as unreachable stack.

The term is a refactor leftover: it was added in 0a0c404a08 (Jun 2019) when the object became a stack local, which was correct, and 4498509426 (Apr 2021) moved it back to a heap member without removing it.

On px4_fmu-v4 that is a 7,432 B request per instance against a 1,908 B measured high-water (26%). The board runs out of heap before MAVLink starts and comes up with no connection at all, which is what the issue reports.

firmware, USB ground station attached result
stock main (eca039bd52) no heartbeat, 5 attempts over 110 s, port completely silent
main + commit 1 heartbeat in 1 s

Both builds identical in flash and static RAM; one constant apart.

Solution

Commit 1 is shared code and applies to every NuttX board. Commits 2-6 are px4_fmu-v4 only, ordered so the free ones come first and the trades last.

commit change effect cost
1 receive stack sizeof + 2840 → flat 4000 3,432 B per MAVLink instance, 6,864 B with two none
2 work queue stacks 3472/2776/2312 → 2400/2300/1900 2,040 B heap none, verified ≥1.71× after trimming
3 four inflated serial buffers → 600 3,328 B static telemetry burst headroom
4 CONFIG_FS_PROCFS_MAX_TASKS 128 → 64 1,536 B static, and NSH builtins can spawn again top/procfs truncate above 64 tasks
5 EKF2_LOG_VERBOSE default 0 3,744 B heap estimator innovation/aid-source topics gone from logs
6 drop UAVCAN, uxrce_dds_client, mag_bias_estimator, SIH 3,652 B static, 207 kB flash DroneCAN and pre-takeoff mag calibration on this board

Combined, clean builds: flash 94.73% → 82.61%, static RAM 39,032 → 30,516 B. On hardware, free with a GCS attached and EKF2 running: free heap 320 B → ~9-11 kB, largest free block 144 B → ~9-11 kB.

Test coverage

px4_fmu-v4, USB GCS attached, EKF2 running, never armed. Stock main vs commit 1 as above; full parameter download and rewrite; 30-item mission upload and readback; MAVFTP listing and 400 kB log download; COMMAND_LONG; gyro and baro calibration; EKF2_LOG_VERBOSE A/B with EKF2 liveness confirmed in both arms; work queue high-water at default sizes and again after trimming, with the parameter autosave path deliberately provoked.

Not covered: MAVFTP write (only read), and the margin is measured on px4_fmu-v4, which has nearly the smallest MavlinkReceiver of the boards above. A reading on a board with more handlers would be welcome.

github-actions Bot added scope:build-system CMake, Kconfig, board config, or build tooling. scope:offboard Offboard mode, external setpoints, companion-computer control, or offboard failsafe behavior. scope:mavlink MAVLink module, streams, commands, or protocol handling. scope:boards Board-specific changes or hardware definitions. labels Jul 31, 2026
gguidone changed the title Gg/mavlink receiver stack fix(mavlink): stop sizing the receive thread stack by sizeof(MavlinkReceiver) Jul 31, 2026
github-actions Bot added the kind:bug Something is broken or behaving incorrectly. label Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Comment on commit 1

fmu-v4 builds with CONFIG_STACK_COLORATION=y, so NuttX fills each new thread stack with a known pattern and anything the thread writes overwrites it. Reading back how much of the pattern survived gives the deepest the thread has ever reached, and that is the figure top and cat /proc/<pid>/stack report.

I exercised the receive thread over everything I could think of, taking a reading after each step so an increase could be attributed: full parameter download and rewrite, a 30-item mission upload and readback, MAVFTP directory listing and a 400 kB log download, COMMAND_LONG, and gyro and baro calibration. The deepest was the MAVFTP log download at 1,908 bytes, against 7,432 reserved.

For the replacement value I went with 4,000 rather than 2,840, which is what the code used before the refactor. The reason is that most of the depth turns out not to be in mavlink at all. Compiling the module with -fstack-usage accounts for only about 1,060 of those 1,908 bytes; the rest is NuttX file I/O below MAVFTP, reading the log off the SD card. I only ever exercised a read, never a write, and inside mavlink the read and write handlers are the same size, so whether a write goes deeper is decided by code I did not test. Writing to FAT does more work than reading, so it plausibly does. 2,840 would leave around 930 bytes spare where 4,000 leaves around 2,090, which is why I preferred the larger number.

Being upfront: this investigation was done with Claude, and while I ran the measurements on my own board I can't personally vouch for every inference in the reasoning above, particularly around the FAT and MAVFTP call paths. So please treat the choice of 4,000 as a proposal rather than a conclusion. If someone with real context there thinks 2,840 is safe, I'm happy to change it, and if anyone can exercise a MAVFTP write or check the high-water on a board with more message handlers compiled in, that would settle it properly.

gguidone requested review from MaEtUgR and julianoes July 31, 2026 12:53
julianoes previously approved these changes Aug 3, 2026

julianoes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

So this is not just about the Mavlink stack size but also v4 fixes to avoid running out of RAM, got it.

…eceiver)

The receive thread stack is requested as sizeof(MavlinkReceiver) + 2840,
but the object is not on that stack. It is a member of the heap-allocated
Mavlink object (mavlink_main.h), and pthread_create is handed only
(void *)this, which start_trampoline casts straight back to a pointer.
Every MAVLink instance therefore reserves a second copy of the object as
stack the thread can never touch.

The term is a leftover from a refactor round trip:

  before 0a0c404  heap (new/delete in start_helper), stack 2840
  0a0c404         object became a stack local, stack sizeof + 2840
  4498509         object became a heap-resident member, term left behind

so the pairing has been wrong since 4498509 (Apr 2021).

On px4_fmu-v4 sizeof(MavlinkReceiver) is 4592 and sizeof(Mavlink) is 5568,
so the receiver is 82% of its heap-allocated parent. PX4_STACK_ADJUSTED is
the identity on NuttX and this board has no networking, so the request was
7432 B per instance. Deepest measured high-water over the receive paths is
1908 B, 26% of that.

Replaced with a flat 4000 B, 2.1x the deepest measurement, returning 3432 B
per MAVLink instance. Measured on px4_fmu-v4 with CONFIG_STACK_COLORATION=y,
taking top once between each of: full parameter download and rewrite, 30-item
mission upload and readback, MAVFTP listing and 400 kB log download, and
COMMAND_LONG handling. MAVFTP was the deepest path.

Effect on hardware: with a USB ground station attached the board goes from no
MAVLink heartbeat at all in 110 s to a heartbeat in 1 s.
Work queue stacks are carved from the same allocator pool as the heap on
NuttX, so an over-reserved queue stack costs the board free memory directly.
This board runs out of heap before MAVLink can start (#28109).

Measured at default sizes with CONFIG_STACK_COLORATION=y, reading StackUsed
from top and /proc/<pid>/stack:

  queue         default   peak   set to   margin
  lp_default       3472   1412     2400     1.70x
  hp_default       2776   1104     2300     2.08x
  I2C1             2312    840     1900     2.26x

Effect: 2040 B returned to the heap.

Verified after trimming, on the resulting firmware rather than by calculation:

  queue         trimmed   peak   margin
  lp_default       2376   1388    1.71x
  hp_default       2272   1104    2.06x
  I2C1             1872    840    2.23x

lp_default's peak appears only after provoking the parameter autosave, which
runs on that queue and writes through to storage. Idle it reads 1100 B, so the
deep path has to be triggered deliberately: change a parameter to a genuinely
different value, since PX4 only saves when the value changed, then re-read
StackUsed. On a board too full to run top this is readable with
cat /proc/<pid>/stack, which reports the same up_check_tcbstack() figure.

The remaining queues are left alone because they have no slack: rate_ctrl
1.28x, SPI1 1.42x, INS0 1.56x, nav_and_controllers 1.69x. SPI2 looks generous
at 3.02x but CONFIG_WQ_SPI_STACKSIZE sizes every SPI queue, so its headroom
cannot be taken without starving SPI1.

INS0 is why these are measured rather than estimated: its peak is 1180 B with
ekf2 stopped and 3828 B once ekf2 runs in that queue.
Four of the seven ports carry hand-inflated buffers while every other port on
the board uses 300. They are static allocations, so they come straight off the
heap arena, on a board that runs out of heap before MAVLink can start (#28109).

  USART1 TX  2500 -> 600   ESP8266 header, usually unpopulated
  USART2 TX  1100 -> 600   TELEM1
  USART3 RX  1200 -> 600   TELEM2
  USART3 TX   900 -> 600

Effect: 3328 B of static RAM returned, flash unchanged. Measured with the
NuttX submodule cleaned, since it builds in-tree and does not rebuild on a
config change, and verified in the ELF with nm.

No functionality is removed. What is given up is burst headroom: MAVLink defers
when a TX buffer is full, so a stream is throttled rather than lost, and 600 B
is still 83 ms at 57600 baud. Both telemetry ports have hardware flow control
and RX DMA, so a full RX ring asserts RTS instead of dropping bytes.

This cannot affect the failure in #28109, which is on USB, a CDCACM link with
its own separate buffers.
CONFIG_FS_PROCFS_MAX_TASKS sizes system_load.tasks[], the per-task reporting
table behind top, /proc and the cpuload accounting. NuttX defaults it to 128
and this board never overrode it, so 128 slots at 24 B each are reserved for
the life of the board. It runs 25 tasks, counted with ls /proc.

Effect: system_load 3096 -> 1560 B, so 1536 B of static RAM returned to the
heap arena.

The second effect matters more on this board. Every NSH builtin needs a 4096 B
task spawn, and before this the largest free block with a ground station
attached was 3760 B, so no console command would run at all: param and top both
failed with "command not found", which is ENOMEM in disguise. With this the
largest block reaches 4416 B and the console works.

64 leaves 2.56x headroom over the live count, enough for the transient tasks a
calibration or parameter save adds. If the limit were ever exceeded, top and
procfs would list fewer tasks and a task's CPU time would go unaccounted;
nothing else is affected.
The verbose estimator uORB nodes cost heap this board does not have. Measured
on hardware with EKF2 running and two MAVLink instances up: 7440 B free at 0
against 3696 B at 1.

Effect: 3744 B returned. On this board that is not optional headroom. With the
receive stack fix alone and this parameter at its upstream default, the board
comes up and streams telemetry but answers no request at all, no PARAM_VALUE
and no COMMAND_ACK, and a ground station hangs requesting capabilities.

Cost: the default logging profile picks up estimator* topics by name prefix at
2 Hz via add_optional_topic_multi, which only logs advertised topics, so turning
it off removes innovations, variances, test ratios, states, the aid_src_* set and
the bias topics, and Flight Review's innovation plots go empty on this board.
estimator_status still carries the filtered test ratios, filter fault flags,
reset counters and pre_flt_fail_innov_*, and replay still works because
SDLOG_PROFILE bit 1 logs EKF2's inputs.

Applied with param set-default so an operator can opt back in. rcS sources
rc.board_defaults at line 280 and starts ekf2 at 437, and EKF2 reads the
parameter in its constructor, so the default lands in time. Note a board with
this parameter explicitly saved keeps its saved value.
…mator and SIH

This board is at 94.7% flash on main and out of heap before MAVLink starts
(#28109), so it needs static RAM back, not only flash.

Measured by building fmu-v4 with and without each:

  UAVCAN               flash -147276 B   static RAM -3520 B
  uxrce_dds_client     flash  -59508 B   static RAM   -64 B

Effect: 3652 B of static RAM and 207 kB of flash returned.

UAVCAN is the only module here that returns meaningful RAM, and it is paid even
though the driver never starts, since UAVCAN_ENABLE defaults to 0. For contrast,
dropping five other modules including attitude_estimator_q and
local_position_estimator returned 128 B, so module removal is otherwise a flash
lever only. The UAVCAN driver costing 3.5 kB of .bss wherever it is compiled in
and not started is worth fixing properly for every constrained board.

This is a trade and is offered as one. A Pixracer with a DroneCAN GPS or CAN ESCs
loses them and must set CONFIG_DRIVERS_UAVCAN=y in a local build.
mag_bias_estimator is likewise a deliberate feature removal: MBE_ENABLE defaults
to 1 and rcS starts it, so pre-takeoff continuous magnetometer calibration goes
away on this board. SIH never runs on hardware and uxrce_dds_client only starts
when UXRCE_DDS_CFG points at a port, so neither holds heap today.

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

🔎 FLASH Analysis

px4_fmu-v5x [Total VM Diff: -8 byte (-0 %)]
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.0%     +55  [ = ]       0    .debug_abbrev
  -0.0%      -2  [ = ]       0    .debug_info
  -0.0%      -5  [ = ]       0    .debug_line
     +67%      +2  [ = ]       0    [Unmapped]
    -0.0%      -7  [ = ]       0    [section .debug_line]
  +0.1%      +8  [ = ]       0    [Unmapped]
  -0.0%      -8  -0.0%      -8    .text
  +0.0%     +48  -0.0%      -8    TOTAL

px4_fmu-v6x [Total VM Diff: 0 byte (0 %)]
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.0%     +55  [ = ]       0    .debug_abbrev
  -0.0%      -2  [ = ]       0    .debug_info
  -0.0%      -5  [ = ]       0    .debug_line
     +67%      +2  [ = ]       0    [Unmapped]
    -0.0%      -7  [ = ]       0    [section .debug_line]
  +0.0%     +48  [ = ]       0    TOTAL

Updated: 2026-08-03T07:36:53

MaEtUgR merged commit 6388739 into main Aug 5, 2026
79 checks passed
MaEtUgR deleted the gg/mavlink-receiver-stack branch August 5, 2026 07:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:bug Something is broken or behaving incorrectly. scope:boards Board-specific changes or hardware definitions. scope:build-system CMake, Kconfig, board config, or build tooling. scope:mavlink MAVLink module, streams, commands, or protocol handling. scope:offboard Offboard mode, external setpoints, companion-computer control, or offboard failsafe behavior.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pixracer goes out of RAM with the latest main

3 participants


Back | FazBrowse Home | New Git URL