| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
…s: latest` Always test against both the default and latest CodeQL bundle. This improves test coverage shortly after a CodeQL bundle release, where the latest bundle may not yet be built into the Actions VM image as the default bundle. It also saves a manual step during bundle release testing, since we no longer need to temporarily change the PR checks to `tools: latest`. There is some redundancy when the latest bundle is the same as the default bundle on the VM image, but this can be considered a test for the `tools: latest` configuration.
There was a problem hiding this comment.
So, right now it looks like both latest and ~ are the same. It would be really nice if you could detect this situation and skip one of the tests. I wouldn't say this is necessary, though.
Sorry, something went wrong.
|
That would be good. Outline of how that might work:
|
Sorry, something went wrong.
Create a prerequisite job that runs the init step twice, with `tools: null` and `tools: latest`. Use the outputs of these steps to compare the two CodeQL versions. Pass the list of distinct tool versions for the integration tests to use in their matrix strategy. This avoids redundant test jobs when the default and latest bundles are actually the same version of CodeQL. `~` is accepted by JSON but not by the Actions context language, so we use `null` to indicate the default version.
|
Added a commit that implements the above idea, comparing tools: null and tools: latest, and only running the matrixed test against the distinct versions. This prevents duplication in the PR tests. I haven't yet modified the codeql.yml workflow. |
Sorry, something went wrong.
There was a problem hiding this comment.
This looks good. Is there a way you can force it to run against two versions? It looks like the job name includes the matrixed parameters.
Sorry, something went wrong.
| strategy: | ||
| matrix: | ||
| os: [ubuntu-latest,windows-latest,macos-latest] | ||
| tools: [~, latest] |
There was a problem hiding this comment.
Do you want to call check-codeql-versions here as well?
Sorry, something went wrong.
There was a problem hiding this comment.
Done, just copied the code over. This means we might see many extra init calls from this repo, and might have alerts wobbling if the two bundles give different results, but I think that is both rare and something we can deal with.
Sorry, something went wrong.
Create a prerequisite job that runs the init step twice, with `tools: null` and `tools: latest`. Use the outputs of these steps to compare the two CodeQL versions. Pass the list of distinct tool versions for the analysis job to matrix over. This lets us test the analysis against both versions, while avoiding duplication when they are actually the same version.
There was a problem hiding this comment.
Nice. Code looks good. Have you tried running this when the versions are actually different?
Sorry, something went wrong.
|
Tested in a draft PR that changed the referenced bundle to an older release, and verified that it produces [null, "latest"] as the matrix and starts parallel jobs with the right versions: https://github.com/github/codeql-action/pull/444/checks?check_run_id=2310079094 |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Always test against both the default and latest CodeQL bundle.
This improves test coverage shortly after a CodeQL bundle release, where the latest bundle
may not yet be built into the Actions VM image as the default bundle.
It also saves a manual step during bundle release testing,
since we no longer need to temporarily change the PR checks to tools: latest.
There is some redundancy when the latest bundle is the same as the default bundle on the VM image,
but this can be considered a test for the tools: latest configuration.
Merge / deployment checklist