This PR looks very promising!!

I ran across some issues however... so it still needs polish (as you already mentioned on slack)...

yarn bazel test //integration:cli_hello_world_test --test_output=streamed

results in

[test_runner.js] running test command '/b/f/w/bazel-out/k8-fastbuild/bin/integration/cli_hello_world_test.sh.runfiles/nodejs_darwin_amd64/bin/yarn install' in /tmp/tmp-194EFNZxTaipL8
/b/f/w/bazel-out/k8-fastbuild/bin/integration/cli_hello_world_test.sh.runfiles/nodejs_darwin_amd64/bin/yarn: line 18: /b/f/w/bazel-out/k8-fastbuild/bin/integration/cli_hello_world_test.sh.runfiles/nodejs_darwin_amd64/bin/node: No such file or directory
[test_runner.js] 0 of 2 test commands successful
[test_runner.js] test command 1 failed with status code 1
[test_runner.js] to run test in debug mode:

    bazel run //integration:cli_hello_world_test.debug

on mac

I also tried bazel run //integration:cli_hello_world_test.debug which fails with:

** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
: Compiled successfully.
[09:23:55] I/launcher - Running 1 instances of WebDriver
[09:23:55] I/direct - Using ChromeDriver directly...
[09:23:57] E/launcher - session not created: This version of ChromeDriver only supports Chrome version 75
  (Driver info: chromedriver=75.0.3770.90 (a6dcaf7e3ec6f70a194cc25e8149475c6590e025-refs/branch-heads/3770@{#1003}),platform=Mac OS X 10.14.6 x86_64)
[09:23:57] E/launcher - SessionNotCreatedError: session not created: This version of ChromeDriver only supports Chrome version 75

which is different from the version I have installed locally (78.0.3904.97) - interestingly this hasn't been a problem I experienced before which is odd. I tried to manually run the webdriver-manager command to install version 78.0.3904.105 and then I got a bit further.

next I did get a build error:

This version of CLI is only compatible with Angular versions ^9.0.0-beta || >=9.0.0 <10.0.0,
but Angular version 0.0.0 was found instead.

This is due to: https://github.com/angular/angular-cli/blob/c7135fae357ffe16bd420b59b80b002632806a45/packages/angular_devkit/build_angular/src/utils/version.ts#L55-L78

I tried to use --config=release as well but that didn't work for me and resulted in the same error.

In any case, build stamping is expensive because it invalidates most caches.

This check comes from: https://github.com/angular/angular-cli/blob/c7135fae357ffe16bd420b59b80b002632806a45/packages/angular_devkit/build_angular/src/utils/version.ts#L55-L58

I worked around this in the past by monkey-patching @angular/cli in node_modules and setting its version to 0.0.0 which disables the check (see cli code link above), but it might be even better to update that CLI version check to be angularCliPkgJson['version'] === '0.0.0' || angularPkgJson['version'] === '0.0.0'. This might be worth requesting from the CLI team or sending them a PR.

One more issue I see lurking is yarn caching issue of file:// dependencies (yarnpkg/yarn#2165) - I believe that I don't see my updates to packages/core/src reflected after rerunning the .debug command and I wonder if that yarn issue is the culprit.

Lastly, I wonder if running yarn test automatically as part of the .debug workflow is ideal. Or maybe .debug should do that but there also should be a .install workflow that just sets up the project with locally built packages built and installed and leaves it up to me to run yarn build or yarn test.

Sorry, I vomited a lot of stuff into this commit :-) I'm just a bit too excited about the prospect of this actually working and us being able to cache the test results of these integration tests, and more easily rerun these tests locally and be able to iterate on core packages while observing the impact of those changes on these integration apps/tests.

Thanks for the taking a look @IgorMinar .

• node node found issue came up in CI as well so will fix that today.

• The config:release issue I think we should fix in the CLI so that it allows version "0.0.0" otherwise we'ld have to have --workspace_status_command="node ./tools/bazel_stamp_vars.js" on for the build which is not desirable talking to @josephperrott as building with "0.0.0" in general prevents accidental publishes. Note: when you add --config:release after the build is complete once it won't re-run the build as changes to bazel-out/volatile-status.txt does not invalidate the build cache.

• Chromedriver issue is interesting. It seems like just running webdriver-manager update --gecko=false --standalone=false installs chromedriver_75.0.3770.140 by default. The test postinstall has webdriver-manager update --gecko=false --standalone=false $CI_CHROMEDRIVER_VERSION_ARG and $CI_CHROMEDRIVER_VERSION_ARG is not set. I also had to run ``webdriver-manager update --gecko=false --standalone=false --versions.chrome 78.0.3904.105` to get the correct version that works with my local chrome.

I was thinking I would upstream the npm_integration_test to rules_nodejs as it will be useful for any project that generates npm_packages and wants to integration test against the generated artifacts. I don't think it would make sense for the angular team to maintain this it is a general solution for integration testing against npm_package artifacts. The angular repo .bzl portion would just be the angular_integration_test macro which sets some defaults.

Alrighty. Based on your comments @IgorMinar I've grossly simplified the rule. It is now a macro that generates a .js config file & then calls a nodejs test with that config file as the first argument.

Can't land this yet as it is waiting on:

1. The next release of angular-cli to make it to angular which includes refactor(@angular-devkit/build-angular): allow @angular/core version 0.0.0 angular-cli#16267 to be able to remove the CLI patch
2. build: update to nodejs rules 0.41.0 #33996 to be able to remove the rules_nodejs patch

Next step is to expand this too all of the integration tests (except for the bazel-in-bazel tests integration/bazel and integration/bazel-schematics) and remove those tests from CI and from the integration.sh script.

Not sure if I should land this here or upstream the npm_integration_test portion and land this PR after that makes it into the next rules_nodejs release.

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

This action has been performed automatically by a bot.