Run tests against nightly CLI bundles - #600
edoardopirovano merged 1 commit into
Conversation
Sorry, something went wrong.
| VERSIONS_JSON='[null, "latest"]' | ||
| fi | ||
| # Use both `tools: null` and `tools: (nightly URL)` in the integration tests. | ||
| VERSIONS_JSON='[null, "${{ steps.get-url.outputs.nightly-url }}"]' |
There was a problem hiding this comment.
I'd like to still be testing against latest because it exercises the hardcoded default configured within the Action repo, and sometimes may be different from both null and nightly. Can we matrix over all three?
Sorry, something went wrong.
There was a problem hiding this comment.
Yep, done!
Sorry, something went wrong.
adityasharad
left a comment
There was a problem hiding this comment.
Looks good, thanks! Hopefully won't make too many matrixed jobs, otherwise we might have to split up the workflow. Merge after the nightly bundle is published.
Sorry, something went wrong.
| # Just use `tools: null` to avoid duplication in the integration tests. | ||
| VERSIONS_JSON='[null]' | ||
| # Skip `tools: latest` since it would be the same as `tools: null` | ||
| VERSIONS_JSON='[null, "${{ steps.get-url.outputs.nightly-url }}"]' |
There was a problem hiding this comment.
Minor: You may want to pull this one into an env var too.
Sorry, something went wrong.
There was a problem hiding this comment.
Thanks, done.
Sorry, something went wrong.
|
There's actually already a nightly release up from the workflow's testing, but the PR check is failing because https://github.com/github/semmle-code/pull/39627 isn't merged yet I think. We'll have to wait for the next nightly build after that's merged before this can go in 🙂 |
Sorry, something went wrong.
|
I'm a little bit confused. This looks like it only runs the init action with the nightly CLI. Should we be running analyze as well? |
Sorry, something went wrong.
The way these tests are setup is a little weird. We use the init action because it gets the CodeQL CLI and puts it on disk so we can print out the versions that we are going to be running with in the later step. Then, these all get put in the VERSIONS_JSON matrix and all the later tests are run with all the versions in there. I'm not a huge fan of this structure, but it didn't seem necessary to change it. |
Sorry, something went wrong.
This PR changes the integration tests of the CodeQL Action to also run against a recent nightly build of the CodeQL bundle. This should help catch compatibility issues arising from changes in the Action earlier.