Fun. So we have a failing eslint rule test, e.g., https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcle-ubuntu1404/17266/console

not ok 448 parallel/test-eslint-require-buffer
  ---
  duration_ms: 2.34
  severity: fail
  exitcode: 1
  stack: |-
    /home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:148
            throw err;
            ^
    
    AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:
    + expected - actual
    
    - 'Use const { Buffer } = require(\'buffer\'); at the beginning of this file'
    + 'Use const Buffer = require(\'buffer\').Buffer; at the beginning of this file' ('Use const { Buffer } = require(\'buffer\'); at the beginning of this file' strictEqual 'Use const Buffer = require(\'buffer\').Buffer; at the beginning of this file')
        at assertMessageMatches (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:442:24)
        at testInvalidTemplate (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:494:29)
        at Function.RuleTester.it (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:581:25)
        at Function.itDefaultHandler (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:143:23)
        at test.invalid.forEach.invalid (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:580:32)
        at Array.forEach (<anonymous>)
        at Function.RuleTester.describe (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:579:30)
        at Function.describeDefaultHandler (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:160:19)
        at Function.RuleTester.describe (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:578:24)
        at Function.describeDefaultHandler (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/tools/node_modules/eslint/lib/testers/rule-tester.js:160:19)
  ...

This PR doesn't touch that test, nor the eslint check that it exercises but it does fix common.skipIfEslintMissing() so this test probably failed before but was being skipped over as eslint wasn't detected 😱.

#19701 changed the message but not the corresponding test.

Fixed up the test-eslint-require-buffer test. Aborted the failing in progress CI and rebuilding: https://ci.nodejs.org/job/node-test-pull-request/14555/

We have a chicken-egg problem. The CI machines don't install eslint, so these tests are never run. On the other hand this is meta-meta-meta-testing (writing a test to verify a test to the linter is properly skipped 😵) and might be overkill? Why not just move the test-eslint-* files to their own suite and only run that in special circumstancess (maybe only on the linter CI)

We have a chicken-egg problem. The CI machines don't install eslint, so these tests are never run.

The CI machines don't need to install eslint as it's already in the source tree from a git checkout. It's only when building from the source tarball that eslint is missing. The tests weren't being run because the common.skipIfEslintMissing check was looking in the wrong place.

Why not just move the test-eslint-* files to their own suite and only run that in special circumstancess (maybe only on the linter CI)

I'm not opposed to this idea (if someone else does the work* 😛) since these tests are testing our custom eslint rules and not testing Node.js itself.

* It's now 10pm here -- I started working on this three hours ago thinking this would be a quick fix (inserting the missing common.skipIfEslintMissing calls) then thought I'd implement a custom eslint rule to catch regressions. Then spent an hour trying to work out why the new rule test didn't fail (hint you need to clear the eslint cache before any new rules are picked up). And then the test-eslint-require-buffer failure.

The CI machines don't need to install eslint as it's already in the source tree from a git checkout. It's only when building from the source tarball that eslint is missing. The tests weren't being run because the common.skipIfEslintMissing check was looking in the wrong place.

Ohh right. So it makes me think about wrong assumptions... the skip rule should look for positive indicator (e.g. existence of .git directory) instead of a negative indicator, so that if things change it'll detect it and not silintly skip (i.e. fail)

It's now 10pm here ...

I'll see what I can come up with...

Ohh right. So it makes me think about wrong assumptions... the skip rule should look for positive indicator (e.g. existence of .git directory) instead of a negative indicator, so that if things change it'll detect it and not silintly skip (i.e. fail)

It checks for the presence of the eslint directory:

(Note that at the moment it's looking in a relative directory and hence incorrect (fixed as part of this PR)).

The rule is skip if !exist that's a double negative, not a positive

cc @nodejs/linting

arm CI rerun passes: https://ci.nodejs.org/job/node-test-commit-arm-fanned/829/

I didn't label this author-ready due to concerns raised by @refack. While I agree that the skipIfEslintMissing check is fragile if the eslint directory changes, this PR is no less fragile than before and would actually fix the rules tests so that they are run on the CI plus improve the situation for Debian (#20336) and anyone else building from the source tarball.

I'm on a tablet at the moment so can't land this right now, but unless there are actual objections I'll land early next week (unless someone else lands first).

Landed in 870ae72