Sorry, messed up the fork. Had to re-create branch.

FYI: I added an agenda item for discussing this feature to tomorrow's cranelift meeting

@playX18 can you summarize (in a bit more detail) what your use-case for this is? I ask because it's not a trivial change to our ABI code -- this adds a bunch of parallel paths that are conditional on the new calling convention, +1.2kLoC overall, which is a lot of maintenance burden in critical code that we may need to update for security fixes, optimizations, new features, etc. in the future. If this serves a significant and widespread use-case that benefits many users, it may be worth it; but we'll want to weigh this not as a "nice to have" feature with a ready PR that we can merge and move on from, but rather, something with ongoing cost.

And if, for example, the desire is to pin more registers with args in a tail-call-between-basic-blocks style, as your description suggests: would an extension to an existing ABI, where we "keep counting beyond the limit" (i.e. SysV but we go beyond the 6 existing int regs on x86-64), do as well for you? That kind of thing would be a lot simpler to merge into the existing code. Or do you really need compat with the GHC ABI?

(Looking forward to discussing in the meeting tomorrow; the purpose of this comment is to fact-find beforehand)

@cfallin

@playX18 can you summarize (in a bit more detail) what your use-case for this is? I ask because it's not a trivial change to our ABI code -- this adds a bunch of parallel paths that are conditional on the new calling convention, +1.2kLoC overall, which is a lot of maintenance burden in critical code that we may need to update for security fixes, optimizations, new features, etc. in the future. If this serves a significant and widespread use-case that benefits many users, it may be worth it; but we'll want to weigh this not as a "nice to have" feature with a ready PR that we can merge and move on from, but rather, something with ongoing cost.

And if, for example, the desire is to pin more registers with args in a tail-call-between-basic-blocks style, as your description suggests: would an extension to an existing ABI, where we "keep counting beyond the limit" (i.e. SysV but we go beyond the 6 existing int regs on x86-64), do as well for you? That kind of thing would be a lot simpler to merge into the existing code. Or do you really need compat with the GHC ABI?

(Looking forward to discussing in the meeting tomorrow; the purpose of this comment is to fact-find beforehand)

My use-case is a Scheme implementation whose compiler lowers programs into CPS. Calls between compiled Scheme procedures are effectively jumps carrying a continuation and a set of VM values.

The current calling convention is approximately:

(ctx, callee, argc, arg0, arg1, arg2, arg3)

where ctx contains runtime state such as the allocation pointer, stack/runstack (not machine stack!) pointers for overflow arguments (arg4+ go onto this runstack), exception state, and GC metadata.
Each call is always a tail-call (aka jump) to either procedure or heap allocated continuation.

The important requirements for me are:

• more register arguments than the platform ABI normally provides;
• predictable, fixed argument-register assignment;
• efficient return_call/tail transfers between functions using the convention;
• no callee-saved-register preservation at every Scheme-to-Scheme transfer;

Compatibility with GHC callconv and LLVM's ghccc is not really necessary for me. I simply chose ghccc because it exists, is easy to find how it works in LLVM depths and is being utilized by some FP compilers that are not GHC with similar purposes: no callee saves, efficient tail calls, and a lot of effectively "pinned" registers even if compiler does not guarantee that.

An extended Cranelift-native ABI—roughly “SysV/AAPCS register assignment, but continue allocating from an explicitly defined larger register set”—would likely work for me, but callee-saves are sort of a problem: they increase the code size and somewhat decrease performance (Tail callconv currently requires frame pointer to always be available). On my compiler I have verified overall ~18% compiled code size reduction compared to Tail cc, and most importantly 5% to 15% performance increase (depending on benchmarks).

I am also happy to narrow the initial implementation to x86-64 and aarch64, which are my immediate targets, rather than adding three architectures at once.

So, to summarize: I need a register-heavy, tail-call-oriented internal ABI, but I do not specifically need GHC interoperability. I would be happy to rework this toward a smaller Cranelift-specific convention if that is the direction everyone prefers.