Summary

Everything under mruby-regexp's README limitations is a place the engine answers a pattern differently from CRuby, and the list is kept by hand. Nothing checks it, so a limitation that stops being true, or one that appears, is found by reading rather than by failing. tools/difftest runs a corpus through both engines and fails on a disagreement its baseline does not describe.

$ rake regexp:difftest
5175 patterns, 101 known differences, no new ones

Changes

Nothing under src/ or the gem's src/ is touched: this adds a task to run, not a check a build reaches.

The two axes

probe.rb runs under either engine and is handed the same bytes by both.

The pattern axis is a line per pattern: where a match starts in each of a fixed list of subjects, what it captured, and which class it raised. The class is part of the answer whether the pattern was refused at compile time or the search raised against a subject, since refusing with RegexpError and refusing with ArgumentError are different answers and a field spelling neither would read as agreement. The message is not, the two being free to disagree about wording. The patterns are generated from their axes rather than listed, so a case added to an axis is added once: every printable ASCII character after a backslash in each of the four places an escape can stand, every quantifier on every kind of atom, the groups, the anchors, the backreferences, the class forms and the POSIX brackets. Each is asked under //, /i, /m and /x, one flag at a time rather than once per combination of them: with all eight asked once, no pattern differed under a combination that did not already differ under a single flag, and the baseline took 105 more lines that were copies of the 21 differences it already held.

The character axis turns that around: a line per character, a column per way of classifying one. A pattern is worth asking about a handful of subjects; a character is worth asking every way there is to classify one, which is the other shape.

compare.rb runs both and diffs. A disagreement the baseline does not describe fails the check. So does a line that has stopped disagreeing, so a fix prunes the list rather than leaving it to describe an engine that has moved on. rake regexp:difftest:update takes a new baseline.

Which build it asks

The one the loaded config declares. A working tree carries the build of every config it has ever run, so reading build/ finds binaries that have nothing to do with the config in hand, and the baseline metadata only catches the ones that answer by other rules. difftest.rake reads the targets instead, skipping the internal ones, the cross builds, and any built without the gem or without an mruby to run.

A config declaring more than one such build leaves nothing to choose between them, since a baseline describes one build, so the task names them and the choice is made with MRUBY. compare.rb takes the binary as an argument rather than looking for one, so that it cannot make the choice either.

The corpus comes from the database

Second commit. A hand-written character list asks about what its author thought of, and the gap shows: it reaches for Ā/ā to stand for "a letter with a case", and those are U+0100/U+0101, one codepoint above the Latin-1 table where CRuby answers by other rules.

The engine classifies by table, so the test wants a character out of every class the tables tell apart. tools/unicode/corpus_data.rb is that rule; gen_corpus.rb writes corpus.rb; rake unicode:generate regenerates it with the tables and unicode:verify checks it, so a Unicode bump moves the questions with the answers.

A class there is a whole signature and not one property at a time: the general category, the script, which POSIX types hold the character, the shape its case folding takes, and the width the encoding spells it in. Taking the lowest of every category, then the lowest of every script, then one in and one out of every POSIX type asks about each in isolation and leaves their combinations unasked, and a combination is what a table gets wrong: a bracket does not stop holding every letter, it stops holding the letters of one script, or the uncased ones, or the ones whose folding expands. The rule takes the lowest codepoint of each distinct signature, lowest being the member that does not move as a class grows, plus the whole of ASCII, which is what almost every pattern is written against. That comes to 995 characters, where taking each property separately came to 338.

The folding shape is read out of every CaseFolding.txt line a codepoint has rather than the last one. A codepoint that folds into several characters is listed twice, as the full folding and the simple one, and a Turkic line stands beside either; taking the last would spell 33 of the 104 expanding codepoints as something other than expanding, which is not a distinction the engine's table makes. Turkic is dropped, being a rule neither engine applies here.

The age floor, and the CRuby it needs

A representative has to be no newer than the Unicode the CRuby running the corpus carries. A release reaches the tables here before it reaches a shipped Ruby, and a character assigned in between is unassigned there and classified as nothing, which would read as an engine that disagrees when what disagrees is the two databases.

MAX_AGE is 17.0, which is both the release the tables are generated from and the one CRuby 4.0 ships, so today the floor takes nothing out. What it is for is the next bump, when it holds the corpus back until CRuby catches up.

The other side of the same rule is a check rather than a floor: compare.rb asks the host CRuby for \p{Age=17.0} before it runs anything, and refuses one that cannot answer.

$ ruby3.2 mrbgems/mruby-regexp/tools/difftest/compare.rb build/host/bin/mruby
this CRuby (3.2.3) carries an older Unicode than the corpus, which is chosen out of 17.0.
A character it has not heard of is classified as nothing there, which would read as a
disagreement between the two engines rather than between the two databases.
Run this under a CRuby with Unicode 17.0 or later (4.0 is the first).

Checking the check

A differential test can go quiet two ways. The engines can agree, which is the answer it is for, and the check can stop looking, which reads exactly the same: no output, green. So there are four things asserted about the tool itself, all of them under rake regexp:difftest:selftest and all of them running again inside a normal run.

The comparison. NEW, CHANGED and GONE are the whole of what compare.rb reports, and they are read out of two hashes rather than measured. The self-test puts the three, the two cases the tool has to stay quiet about, and the round trip a baseline line is written and read back through, to it with answers made up rather than run. Dropping the GONE arm, stopping CHANGED from firing, and narrowing the split a baseline line is parsed with each fail it.

The pattern corpus. A count would catch an axis that quietly stopped generating and would have to be edited on every deliberate change, so what probe.rb asserts is its shape. The axes that are a product say so: ESCAPE_CONTEXTS, NAMED_ESCAPE_FORMS, QUANT_ATOMS and QUANT_SUFFIXES are named rather than written into their loops, and the check walks the same products the corpus is built from. What that alone cannot catch is one of those lists being shortened, since the check would follow it, so AXIS_CASES names both ends of each list, one kind of atom per entry, and a case out of each axis that is not a product. Seven ways of shrinking an axis were tried against it and each fails it, from a context dropped out of the escape loop (186 patterns missing) to the printable range narrowed (2).

Writing it down found a gap: [\u{41}], the list form of a codepoint escape holding one codepoint inside a class, was in no axis. The corpus held [\u0041], [\u{41 42}] and [\u{41}-\u{43}], and not the simplest of the four.

The probe's own output. What probe.rb writes is a protocol: three tab separated fields per answer, and one #build line naming the build. A line of another shape used to become an entry with a field missing, and a label written twice used to replace the answer before it, both silently. collect now accounts for every line it reads, and the self-test puts a well formed run and each of the six ways to be malformed to it. Against a real run, appending a field to every answer, dropping the capture field, writing a label twice and dropping the #build line each fail it:

$ rake regexp:difftest
cruby: a line has 4 tab separated fields, not 3: "\\!/\t.....................X0..\t;;;...;\tmore"

The character corpus. Generating one that covers less than it says it does leaves no trace: the file is written, unicode:verify regenerates the same file and agrees with it, and the difftest passes by asking fewer questions. So compose ends by reading its answer back, taking the signature of every character it picked and stopping the generator if a signature the database has is left with nothing standing for it. Dropping the ASCII block reports 13 such signatures; starting the walk at U+0100 instead of U+0080 reports 10. What it cannot say is that the signature is the right signature: rewriting it changes both sides of the comparison, and moving the floor changes what "every" means. Those are the deliberate edits, and this is for the accidental ones.

What it found

One difference, and it is Onigmo's: CRuby reads a character under 256 off a Latin-1 word table for \b and off the Unicode tables for [[:word:]], so its boundary sits beside characters its own bracket does not hold.

"²" =~ /\b/          # CRuby: 0, here: nil
"²" =~ /[[:word:]]/  # nil either way

This engine reads \b off [[:word:]] at every codepoint, so it answers both the same way. Neither reading is a rule Ruby states, so the difference is baselined and written into the README rather than chased into the engine.

What bounds it

The Unicode properties are asked about as a sentinel rather than as an axis. The engine refuses \p{...} outright, which the README states, so every property name would answer RegexpError against a CRuby that answers it: one limitation written out a thousand times rather than a check. Four patterns stand for it instead, \p{Alpha}, \P{Alpha}, [\p{Alpha}] and \p{L}, which is what the day the refusal stops needs: the difference goes GONE, and reporting a limitation that has stopped being one is the half of the tool's job a corpus holding no property escape cannot do. A corpus that asks what each property holds belongs with the data it would read.

A baseline also describes the build it was taken against. A build reading its strings as bytes, or classifying them by ASCII, answers differently wherever a table is read, so compare.rb refuses a build that is not its baseline's rather than calling all of those regressions.

That, and the CRuby check above, are why this is a task to run and not a job in the workflows, which use whatever CRuby a runner ships. Wiring it in is a one-job change once a CRuby is pinned for it.

Testing

$ MRUBY_CONFIG=host-debug rake -j8
$ MRUBY_CONFIG=host-debug rake -m test
  Total: 2551
     OK: 2545
     KO: 0
  Crash: 0
Warning: 0
   Skip: 6

$ MRUBY_CONFIG=host-debug rake regexp:difftest
5175 patterns, 101 known differences, no new ones

$ MRUBY_CONFIG=host-debug rake regexp:difftest:selftest

$ MRUBY_CONFIG=host-debug rake unicode:verify
the Unicode 17.0.0 tables are up to date

Both commits check green on their own; the first is the pattern axis alone, at 4180 patterns and 100 known differences.

prettier --check passes on the README. pre-commit passes, markdownlint and actionlint skipped (neither installs on this host).

No C source is touched, so there is nothing to measure for speed or size.

Environment

The build the baseline describes is build_config/host-debug.rb, which is full-core and so carries mruby-encoding: the probe reports it as chars=1 unicode=1. enable_debug puts -g3 -O0 after the toolchain's -g -O3, so this build is -O0.

gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_USE_DEBUG_HOOK -DMRB_NO_BOXING -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK "mrbgems/mruby-regexp/src/re_compile.c"

Summary by CodeRabbit

• New Features

  • Added differential testing for regular expressions against CRuby, including Unicode character coverage and comparison baselines.
  • Added commands to run, update, and self-test the differential test suite.
  • Added automated generation and verification of the Unicode test corpus.

• Documentation

  • Expanded regexp limitation and differential-testing documentation.
  • Documented Unicode compatibility and expected engine differences.