I opened #909 for the design write-up.

Now reflect.h2 doesn't have a dependency on parse.h.
It is compiled as a pure Cpp2 header, taking advantage of #594 (comment).
reflect_impl.h2 has the remaining bits that depend on parse.h.

To regenerate reflect.h2, use

cppfront -p reflect.h2 -o cpp2reflect.h
mv cpp2reflect.h ../include/

I use std::any values to build the compilation firewall.
The implementation, cpp2reflect.hpp, is #included at the end of reflect_impl.h2.
This is why odr-uses of the reflection API requires linking to cppfront.

If depending on Boost.DLL is undesirable, loading and using shared libraries is actually quite simple:

For POSIX:

• dlopen to open the SO
• dlclose to close the SO
• dlsym to get a symbol (function in this case)
• dlerror to get a error string for error checking

For Windows:

• LoadLibraryA to open the DLL
• FreeLibrary to close the DLL
• GetProcAddress to get a symbol (again, a function)
• A combination of GetLastError, FormatMessageA and LocalFree to get a error string for error checking

It doesn't get much more complicated than that if you just want to call C-named functions. Let me know if you'd like me to support on this.

Let me know if you'd like me to support on this.

Thank you.
I think we would all like this.

It would help to not have to depend on Boost.DLL if there is an implementation for the current platform.

For GCC, I can use my system's Boost.DLL,
But for Clang, I have to build its dependencies from source due to ABI (#797 (reply in thread)).

Alright, I'll write the changes and open PR against the branch in your repo, we can discuss further over there once its up 👍🏻

Thanks to the contribution of JohelEGP#1 by @DyXel and @edo9300
now we directly use the OS APIs for loading libraries.
This means that Boost.DLL is no longer required, and
a cppfront compiled as usual will have support for loading metafunctions.
You still need to specify the libraries with metafunctions via an environment variable.
I have updated the opening comment, which should be used as commit message when merging, to reflect this.

I'm thinking of just adding a metafunction to lower a declaration with the CPPFRONTAPI macro.

There are similarities to export declarations of C++ modules
(which will be an access-specifier in Cpp2, see #269 (comment)).
I asked on the Cpplang Slack on how it relates to symbol visibility, but they are orthogonal features
(see https://cpplang.slack.com/archives/C92GZLCSE/p1700658714626929).

Relatedly, we might eventually want to lower extern "C" declarations.
#624 is also interested in using @c_api for something else.

Anyways, I'll try to think of some fitting name to get things moving here.
Suggestions are welcome!

When compiling with the hidden visibility preset, loading the metafunction sfml fails. It works when compiling the module that declares sfml because it imports the module that exports rule_of_zero. But on load, the rule_of_zero in its body is diagnosed as an undefined symbol, as it isn't visible. Manually adding CPPFRONTAPI to the lowered declaration of rule_of_zero makes it work.

I guess technically you could use the C declaration for this case, but you'd need to do the casting to void*, plus namespace handling would be out of the window. Or, since you can detect the signature of a metafunction already, metafunctions detected within a body of another could be lowered to the specific C-magic. Both very ugly hacks but could work for a POC.

Indeed there needs to be a way to mark stuff to use C-linkage and/or declare its symbol visibility (in general, a way of spelling this specific stuff, I am sure there are more out there), but I wonder, should that functionality be covered as you mention with a metafunction (c_api)? I didn't think of using them like that, feels like that is not what metafunctions are intended for, and you are still modifying the signature of a function within cpp2 limits, no? It should error out saying its not valid code.

Edit: Saw the commit, of course adding a flag to modify the lowering behavior, that works! But I still am questioning whether metafunctions are the right tool for this.

I guess technically you could use the C declaration for this case, but you'd need to do the casting to void*, plus namespace handling would be out of the window. Or, since you can detect the signature of a metafunction already, metafunctions detected within a body of another could be lowered to the specific C-magic. Both very ugly hacks but could work for a POC.

The undefined symbol being that of a metafunction is a coincidence.
The general issue is that Cpp2, without @visible, can't be used to author a DLL
(with proper hygiene, i.e., hidden visibility by default, like the Windows default).
A metafunction couldn't use a name declared in another Cpp2 TU.

Now I can use a cppfront compiled with the hidden visibility preset.
With it, my uses of metafunctions in my project work just fine.
So the Visual Studio issue should be solved now (#907 (comment)).

I have a plan to solve the name lookup problem for good using only the current source file.

• Emit a metafunction symbol with its fully qualified name mangled.
• Lookup an @-used metafunction using the source order name lookup
  (since source_order_name_lookup is in to_cpp1.h, and
  sema.h's get_declaration_of isn't of help here,
  I'll have to repeat source_order_name_lookup during parse.h).
  If the lookup isn't locally unambiguous, I can diagnose why and how to fix it (there are many things to consider).
• Emit a static_assert to confirm that Cpp1 lookup finds the name we found.
  If you can @-use it, you should be able to use it (even if not strictly necessary by the implementation).

The only chance for surprise is when a user expects a non-local name to be found.
In the rare case we have a local match, the generated code won't be what the user expects.
That error shouldn't get past the static_assert.
But it might just immediately break evaluating the next metafunction in a chain.

Or maybe I should just go ahead and start emitting the extra semantic information (#909 (comment)).

I think having a specific C function per TO/DLL that is able to tell whether or not it has the symbol, has value on its own, for example:

CPP2_C_API int cpp2_meta_library_has_metafunction(const char* name, size_t size) {
    static std::set<std::string_view> mfs = {"greeter", /*...*/};
    return mfs.count(std::string_view{name, size});
}

(...or alternatively, a function that gives you a list of strings from which you can build a look-up table)

Armed with a function like this you would be able to tell what was exported, but also, you could first check the existence of this same function before proceeding with anything else, granting the opportunity to give the user a good explanatory message, like "cpp2_meta_library_has_metafunction was not found in DLL 'x', are you sure 'x' is a cppfront meta library?".

Just my 2 cents though.

Great idea, thank you!

but I wonder, should that functionality be covered as you mention with a metafunction (c_api)? I didn't think of using them like that, feels like that is not what metafunctions are intended for

I don't disagree.
But it's currently the most fitting place to specify properties for the declaration.
I also want a @deleted instead of having to unsatisfactorily abuse a private overload
(see the thread starting at #468 (comment) and the referencing issues).

And potentially @all_freestanding, @freestanding, @freestanding_deleted, and @hosted
(whose effect depend on __STDC_HOSTED__).
Although those have usability limitations (it might not really be what we want):

• https://eel.is/c++draft/freestanding.item#3.sentence-2
• https://eel.is/c++draft/compliance#2.sentence-2
• https://eel.is/c++draft/compliance#3
• https://eel.is/c++draft/compliance#4.sentence-2

More generally, we still need a replacement for some uses of the preprocessor.
Maybe Cpp1 reflection will help here.

I don't disagree.
But it's currently the most fitting place to specify properties for the declaration.

Yeah, for a POC is fine. I do want people trying this functionality to its maximum and give good feedback to Herb, but I do worry about its future, been thinking about this for a while now so I might as well share my opinion:

A user-defined metafunction right now can do way too much, after all, it is arbitrary Cpp1 code being compiled and executed. It was fine being only used internally by the cppfront compiler, but if we give users total freedom to do whatever they want within a metafunction (e.g. execute arbitrary code and generate side-effects) then it'll become a problem once cpp1 reflection/generation lands, assuming it would be a subset of what cppfront offers, there would be competition between what cppfront can do and what was standardized ("teach a man how to fish...").

Another thing that I don't like is the necessary double-pass introduced by this user-defined meta functionality, in order to use a user-defined metafunction you'd need to write Cpp2, which gets lowered to Cpp1, which then gets compiled, and then used somewhere else in Cpp2, this extends the usage from simple transpiler (cpp2 -> cpp1 -> compile and run!) to something more complicated (cpp2 -> cpp1 -> compile meta -> cpp2 -> cpp1 -> compile and run!), which might drive adoption away.

For the latter, I think in a perfect world, cppfront would be able to interpret and apply the metafunction itself without having to loopback, and then when cpp1 meta lands (hopefully a good implementation with feedback received from this experiment!), we start generating cpp1 code instead of interpreting, and so users would be minimally affected.

then it'll become a problem once cpp1 reflection/generation lands, assuming it would be a subset of what cppfront offers, there would be competition between what cppfront can do and what was standardized

For C++26 (P2996), the feature sets would be disjoint.
In the future, we should have metafunctions (meta classes?) in Cpp1.
That should subsume Cpp2 metafunctions, and we should migrate to using that Cpp1 feature.

Regarding the above compiler flags:

• -rdynamic, --export-dynamic: Seems like one of these is redundant according to the documentation. And it seems unnecessary unless you want to debug your metafunction (Which we should try to cover as well a little bit on the user-guide I am imagining?) It is necessary when compiling cppfront itself, in order for it to export the necessary API for metafunctions to use (thanks Max for clarifying). Shouldn't be a problem on Windows methinks.
• -fPIC: It's just as I mentioned during our call: It's good practice to activate for SOs in order to avoid problems when loading multiple SOs that could be on the same memory layout, but it is not strictly required (ref). Windows' DLLs have the equivalent by default called File Base Relocations, so no problems there.
• -shared: Seems like its necessary since the compiler won't assume you want to produce a SO just from the output filename. Importantly, it says that you should use the same linker flags if you have multiple steps (compile and then link) for predictable results (ref), shouldn't be a concern for us if we aren't doing anything fancy inside a script. MSVC and such have the equivalent flag /LD.
• -soname: If I understood this SO answer properly, seems like the system needs it to know where to find a dependency symbol, but I think we shouldn't really need this unless we start talking about inter-dependencies from one metafunction library to another (To be discussed further?). In short, since we are manually looking up symbols with dlopen, I don't think it affects us yet and can thus be skipped. No equivalent/unnecessary on Windows.

By the way, as a side-note: Could we please change the envvar names a little? I got them mixed up during our call (sorry for that), though the conclusion was correct, I think they are too similar:

CPPFRONT_METAFUNCTION_LIBRARY
CPPFRONT_METAFUNCTION_LIBRARIES

But I don't know what to suggest, maybe CPPFRONT_META_LIB_NAME and CPPFRONT_META_LOAD_LIBS?

I also checked the workflow on my machine.

• -rdynamic is necessary. Without it, the symbols of cppfront are not exported and the libraries can not find the cppfront API.
  The error is:

main.cpp2...Contract violation: failed to load DLL './libmetafunctions.so': ./libmetafunctions.so: undefined symbol: _ZNR4cpp24meta16type_declaration10add_memberERKSt17basic_string_viewIcSt11char_traitsIcEE

• I checked if CPPFRONT_METAFUNCTION_LIBRARY is really required and it is not. Each library can have the same function and it will not collide on the dlopen call. Nevertheless, from your comment in reflect_impl.h2:

    // FIXME Doesn't work for a library with more than one source providing a metafunction
    // See https://github.com/hsutter/cppfront/pull/907#issuecomment-1872644205

I understand that you wanted to fix the situation if you have e.g. two files meta1.cpp2 and meta2.cpp2 both containing a metafunction. Then you need two calls to cppfront and cppfront will inject two times the function cpp2_metafunction_get_symbol_names_. This will create a linker error when the two object files are linked. E.g. g++ -o meta.so meta1.cpp2.o meta2.cpp2.o. Your solution with CPPFRONT_METAFUNCTION_LIBRARY will not solve this case, since in both cases you need to call cppfront with CPPFRONT_METAFUNCTION_LIBRARY=meta.so cppfront meta1.cpp2. This will create the same name for cpp2_metafunction_get_symbol_names_ . If you change the name to CPPFRONT_METAFUNCTION_LIBRARY=meta1.so cppfront meta1.cpp2 then
CPPFRONT_METAFUNCTION_LIBRARIES=meta.so cppfront ... will not work since the symbol cpp2_metafunction_get_symbol_names_meta_so is not defined.

There are two options to address this issue:

1. cpp2_metafunction_get_symbol_names_ is no longer generated automatically. The user has to add it manually. This might be addressed with a metafunction @create_meta_export(func1, func2, func3,...). But this would require to have "free flow" meta functions. The code would be:

greeter: (inout t: cpp2::meta::type_declaration) = {
  t.add_member($R"(say_hi: () = std::cout << "Hello, world!\nFrom (t.name())$\n";)");
}
@create_meta_export(greeter)

2. We use the static initialization to register the metafunction automagically. cppfront could export a symbol e.g. add_metafunction and then for each metafunction a static registration method is created, e.g.

static const int init_greeter = ::cpp2::meta::add_metafunction(greeter, "greeter");

It think I would prefer option 1.

Here is a patch that removes CPPFRONT_METAFUNCTION_LIBRARY:
0001-Remove-CPPFRONT_METAFUNCTION_LIBRARY-requirement.txt

On my machine the reduced layout creating and using a metafunction is now:

# Compiling cppfront
  g++ -rdynamic cppfront.cpp -o cppfront

# Creating the library
  ./cppfront metafunctions.cpp2
  g++ -std=c++20 -fPIC -shared -o libmetafunctions.so metafunctions.cpp

# Using the library
  CPPFRONT_METAFUNCTION_LIBRARIES=./libmetafunctions.so ./cppfront main.cpp2
  g++ -std=c++20 main.cpp -o main
  ./main

I think we should at least attempt to follow the auto-registering mechanism that most test framework out there have (and what I hacked very briefly on my test_metafunction branch), that would be Option 2. In fact, I would go as far as saying that maybe the compiler should have some kind of generic infrastructure to aid this "pattern"? Because this can also be extended not just to test frameworks, but to this metafunction registration/look-up problem, registry of bindings for other languages and probably more. Essentially, you write your code as normal, auto registration is generated on a per-TU basis, and then you need a way to signal the generation of a single and unambiguous function per shared object/program (as opposed to per TU), and we'd want the equivalent as well for tear-down.

EDIT: Note: I completely side-stepped this issue entirely on my test_metafunction branch by piggybacking on the fact that there must be only 1 main function in the program--I simply inject the "run test framework" in that function. Maybe we could have something similar for metafunctions?

I think we should at least attempt to follow the auto-registering mechanism that most test framework out there have (and what I hacked very briefly on my test_metafunction branch), that would be Option 2. In fact, I would go as far as saying that maybe the compiler should have some kind of generic infrastructure to aid this "pattern"?

Is this similar to what I suggested in the Note in this 797 comment?

Is this similar to what I suggested in the Note in this 797 comment?

If you mean

Future metafunction generation capabilities. We do (eventually) want metafunctions to be able to generate new declarations into existing scopes in the parse tree that are outside the type the metafunction is being applied to

Then yeah, it would be exactly that.

Sorry, I meant this part (I didn't repaste it here because I didn't want to lose the link to Johel's followup comments about feasibility).

(for example, as a strawman: and have the first pass script invoke cppfront with a new flag that says to only compile functions that have a meta API declaration as a parameter; or require user-defined metafunctions to have names that start with @ and have the first pass script invoke cppfront with a new flag that says to only compile functions with a name that starts with @; or something else).

Ah, I think I see what you mean. Let's grok this in 2 parts:

On the double-pass for a single file: I don't think this is can be made full solution currently, as soon as you consider multiple files, everything breaks apart;

• You have to deal with definitions present in other files but used within the metafunction you are trying to define, which means being able to process all files into a single program, which is simply not how cppfront works today. Lowering to Cpp1 everything but the metafunctions, compiling the world, and re-running over the same set of files and getting everything right seems like an impossibility to me at the moment.
• Metafunctions themselves have regular code inside, so you'd need to do a dependency graph somehow, because an entity that uses a metafunction, could itself be used within a metafunction's definition (and vice versa?).
  • This is unsolvable if you will have Cpp1 code mixed-in (how do you build such graph for code you can't parse?).
    • I guess you could argue that if you don't find a definition then it means that its Cpp1, which don't use meta, but seems like a big assumption to make I think.

Marking metafunctions: I don't think that marking/annotating them is strictly necessary (after all, currently checked-in code simply detects the signature just fine¹), Personally I would argue in favor of actually marking them because:

• Can be used to enforce constraints, the compiler knows a marked function will need to have a specific signature and can provide better diagnostics as a result.
• Can be used to generate additional, out-of-line code, just like I did for my @test metafunction stuff. This could solve the problem related to needing CPPFRONT_METAFUNCTION_LIBRARY, but note that it is not enough to solve the double-pass thing mentioned above, but it does help if later we want to implement something like that.
• I think it looks and reads nicer :^)
  • For example, greeter: @meta (inout t: cpp2::meta::type_declaration) = {} would read as "greeter is a meta function". Which I think sounds proper.
    • Going even further, because we know that it must be a metafunction, we can apply defaults, such as deducing the type of t and of course lowering with the right DLL visibility.

¹ a bit limited but overall seems decent.

Side note: Maybe just me, but its getting harder and harder to keep track of everything that has been discussed, specially since since stuff is getting so meta 😅 I will try to make a post at a later date collecting all the problems and some potential solutions to them in a single post, because this is getting unruly--jumping to several different places to get all the context is hard.

In this post I compile all current "constraints" that I know of, that this specific solution has, as currently implemented in this PR.

Please let me know of any developments so I can directly update this post, and let us enumerate/name these constraints so we can more easily refer to them later.

Constraints

Constraint Nº1: Distinct compilation step required for metafunctions

Currently, we need to manually identify that we are building a metafunction library as opposed to a regular C++ program.

I think it comes with the design of the solution, but having to remember extra steps is inconvenient from a User Experience (UX) standpoint.

As I mentioned before in a different post: This extends the usage of cppfront from simple transpiler (Cpp2 -> Cpp1 -> compile and run!) to something more involved when authoring metafunctions or using user-defined ones (Cpp2 -> Cpp1 -> compile meta -> Cpp2 -> Cpp1 -> compile and run!), which may cause potential adopters to lose interest.

Constraint Nº2: Inability to apply metafunctions defined in the same TU or "step"

A variation of the classic chicken-and-egg problem, bootstrapping (compilers), etc.

Essentially, in order to have a metafunction available for use, you must have first compiled and loaded it as a DLL; Therefore you can't both define a metafunction and use it in the same "step" in your compilation process. In a similar vein, if the user tangles themselves enough, they could be lead to circular dependencies between regular code and metafunction code, breaking causality. Consider this trivial code for example:

foo: @bar type = { } // Depends on @bar

bar: (inout t: cpp2::meta::type_declaration) = {
	a: foo = (); // Depends on foo
	_ = a;
}

// Which came first: foo or bar?

Currently, by having a strict separation of metafunction code and regular code, the circular dependency can be avoided somewhat, but it is something to keep in mind if we want to implement a workaround.

Constraint Nº3: Multiple symbols with the same name

This one I would say spans three distinct levels:

• a. Name collisions in different files which are #include'd together: Adding for completeness, I don't think this one is a big deal, as it should be caught entirely by Cpp1 during compilation. Is there any potential edge-cases to consider?
• b. Name collisions in distinct TUs, but in a singular program/DLL: These violate ODR and do not require diagnostic, they could be potentially insidious to detect (though most linkers do diagnose it).
• c. Fully-qualified names in distinct DLLs, but loaded in a single invocation of cppfront: There must be a way to identify these in distinct DLLs that are loaded simultaneously and either reject them or disambiguate, as currently the first match is picked silently.

In my opinion, we should treat metafunction definitions just like C++ treats them: It is a violation to have multiple definitions with the same fully-qualified name (and going further for our use-case, even across different loaded DLLs). In cppfront we should actually catch these violations and report them to the user when possible.

Constraint Nº4: Need for an entry-point for the DLL ...

... and the need for CPPFRONT_METAFUNCTION_LIBRARY

The main reason (I believe?) why CPPFRONT_METAFUNCTION_LIBRARY is needed right now. There must be a way to identify exported C names in distinct DLLs for "entry-point"s that are loaded simultaneously.

As pointed out by @/MaxSagebaum, you can have multiple DLLs with the same name and different definitions, they shouldn't collide if you are manually loading a DLL via dlopen or LoadLibrary. Further, with the current approach, you'd need to define CPPFRONT_METAFUNCTION_LIBRARY for each file/TU in order to get a distinct name that won't have collisions when linking the entire DLL, this might not be viable at scale.

Constraint Nº5: Names are "C-namespaced" and name look-up is limited

There's a limitation in name look-up explained here. It is marked as a temporary alpha limitation so I expect this to be solvable with extra effort.

TODO: Test how limited it is/put example code showcasing the limitation.

Constraint Nº6: There's no concept of "construction" or "tear-down" for generation

Currently, there's no specification or guarantees to the user or metafunction author on how their libraries are loaded, therefore, it is impossible to determine exactly when static constructors and destructors are going to be called within the library, and thus, cannot be relied upon for things like setting up resources or closing down out-of-file generation states.

Static local variables can be used as reference starting point. dlopen/LoadLibrary and dlclose/CloseLibrary pairs seem to execute static constructors and destructors properly, so it could be possible to give the users the guarantees that global static objects will be constructed/destructed on a per file (TU) basis.

Later, if cppfront is able to handle entire program compilations by itself, an additional guarantee could be given (no idea if its actually a good idea):

• Global static objects will be constructed/destructed on a per program basis, that is, constructors will run before processing any file, and destructors will run after processing all files but before generating the final program.

Constraint Nº7: The mechanism for out-of-class generation is hard-coded to metafunctions

In order to generate a unique entry-point to load the library among other things, out-of-class/out-of-function generation is done, however, this process is not generic/reusable for metafunction authors. Ideally the whole mechanism used right now should be refactored such that it can both be used by this implementation and by users alike.

I have drafted JohelEGP#2 in order to possibly tackle Constraint Nº3, Constraint Nº4 and Constraint Nº5.

@DyXel
To reply to your email, feel free to take over as you see fit.
I haven't been programming lately, and I don't know when I'll be back.

Hey, thanks for your response.

I haven't been programming lately, and I don't know when I'll be back.

I have been there as well, hope to see you active again later!

feel free to take over as you see fit.

Will ponder about it and see where that leads me. Cheers!

I think I have thought about it for long enough, and I've decided to take over this feature; As said in my mail to Johel, my agenda is to simplify this solution by means of re-implementing and/or re-factoring the current work, and to solve several of the constraints mentioned in my comment above while doing so.

Here's my current plan, re-using some (most?) of the work in this PR:

• Specifically, I want to go with the route explored in my POC (Auto register metafunctions (POC) JohelEGP/cppfront#2), discard the mangling stuff, along other things that might not be necessary.

• Rather than having a hardcoded mechanism to detect and register metafunctions, I want to explicitly mark functions with @meta and be able to emit the out-of-class declaration _ : cpp2::meta::register_function = ("::greeter", std::addressof(::greeter));, for this I should be able to reuse what its implemented in declarations in metafunctions #809.

• In order to tackle Constraint Nº6, I want to give the users the guarantee that their DLLs will be loaded once and unloaded once, per file processed. Which in turn would call static object's constructor and destructors in said DLLs. Later (or in the same bulk if not complicated) we can expose as an API the currently processed unit and such to aid generation.

If there are no objections to this roadmap, I would start working on it next weekend. @MaxSagebaum do I have your permission to reuse your work from #809 if needed?

@DyXel yes sure. You have my permission.

Your plan sound good to me.