Enable a controlled switchover between CodeQL releases by henrymercer · Pull Request #1475 · github/codeql-action · GitHub

This PR adds functionality that we will use to enable a controlled switchover between CodeQL releases. Specifically, we would like to ensure that each repository running CodeQL transitions from the previous CodeQL release to the new CodeQL release once. We want to avoid repositories flip-flopping between CodeQL releases during the rollout of a new CodeQL release, specifically during the Actions runner image update process, as this creates alert churn.

To do this, we decide the default version of CodeQL on Dotcom using feature flags rather than by using what's in the toolcache. Specifically, we look at all the default_codeql_version_x_enabled feature flags and pick the latest enabled version.

One niggle with this approach is that if the CLI version we need isn't in the toolcache, there isn't yet a nice way to find the CodeQL Bundle release corresponding to that CLI version. In the medium term, we'll consider tagging each CodeQL bundle release with a CLI version number, for instance codeql-bundle-v2.12.0. In the meantime, we add a special asset to the release cli-version-2.12.0.txt specifying the version number. The contents of this asset don't matter, since we don't want to have to download anything.

GHES behaviour is unchanged: we account for the new toolcache format, but we continue to use the CodeQL bundle release specified within the Action (in defaults.json) by default. We also continue to allow CodeQL bundles that have been baked into Actions runner images to override the version in defaults.json.

Commit-by-commit-review recommended.

TODO before merging

Add a changelog note for the impact on the layout of CodeQL within the Actions runner images.
Adapt the functionality that enables using the toolcache for manually specified bundle URLs to the new runner image structure. This will require a search through toolcache versions or a bundle tag name to CodeQL CLI version lookup.
Add some more unit tests for setting up CodeQL. Also require an owner and a repo in the GitHub Releases API mocking as a regression test.

Merge / deployment checklist

Confirm this change is backwards compatible with existing workflows. — see changelog notes
Confirm the readme has been updated if necessary.
Confirm the changelog has been updated if necessary.

I've added some things we need to do before merging this to the PR description, but I think this is ready for an initial look.

Looked through commit-by-commit and everything makes sense to me. Just a few comments for now, will take another 👀 tomorrow!

In the future we might convert the file to JSON with comments, or add this note to a JSON schema.

This gives us an easy cache hit when requesting the same tools URL.

One niggle with this approach is that if the CLI version we need isn't in the toolcache, there isn't yet a nice way to find the CodeQL Bundle release corresponding to that CLI version. In the medium term, we'll consider tagging each CodeQL bundle release with a CLI version number, for instance codeql-bundle-v2.12.0. In the meantime, we add a special asset to the release cli-version-2.12.0.txt specifying the version number. The contents of this asset don't matter, since we don't want to have to download anything.

Could we add a json file in this repo that has mappings from bundle release number to version number or URL? This would be helpful in general for our users since there's no easy way for them to figure out this mapping right now without downloading and running codeql version.

This could go in defaults.json, which we're already updating for each release.

👍 on new additions. CI failure is due to known Swift autobuild hang and I've rerun.

Could we add a json file in this repo that has mappings from bundle release number to version number or URL? This would be helpful in general for our users since there's no easy way for them to figure out this mapping right now without downloading and running codeql version.

This could go in defaults.json, which we're already updating for each release.

I have a slight preference for keeping the GitHub Releases the source of truth:

This aligns better with our medium term goal of changing the version numbering system of bundle releases to include the CLI version number. Once we've done that a mapping in defaults.json would be redundant.
We can automatically add the CLI version marker file to the bundle release as part of the publishing process, saving a bit of manual work.

Given the complexity of this change and the fact that the actions changelog is not very visible to users, are you considering adding this to a changelog post?

Given the complexity of this change and the fact that the actions changelog is not very visible to users, are you considering adding this to a changelog post?

I think a changelog post would be a good idea. Will discuss with Alona once we've agreed on a good Action changelog note.

✅ from me on the code changes, but I believe there is still ongoing discussion about the inclusion of the section for advanced users in the changelog note.

Approving but looks like merge conflicts need to be resolved 😄

Hmm since we already have merge conflicts here, it might make sense to get #1492 in first and then rebase on that, as I expect the TypeScript update will probably create some more conflicts.

TypeScript PR is failing on Windows, so let's do this PR first instead.

Looks like the PR checks are failing with parsing SARIF on Windows but it should be unrelated to this PR 🤔

angelapwen reviewed Jan 11, 2023

View reviewed changes

Comment thread src/init.ts

henrymercer force-pushed the henrymercer/controlled-switchover branch from fa0e8d6 to a9eb86f Compare January 11, 2023 15:59

Base automatically changed from henrymercer/fix-ghae-setup-test to main January 11, 2023 17:14

henrymercer added 8 commits January 11, 2023 18:40

Support determining Dotcom CLI version from feature flags

cdb9019

Ignore default version flags with invalid version numbers

a6dff04

Enable mapping from CLI version to bundle tag name

a76fe4f

Move CodeQL setup to its own file

bd2f52f

Use new default version to set up CodeQL

c3be36f

Add test for using default version with no requested URL on Dotcom

23d151d

Differentiate setupCodeql.setupCodeQL from codeql.setupCodeQL

2f7b9a1

Add a debug log for the feature flag API response

e8c12e1

henrymercer marked this pull request as ready for review January 11, 2023 19:10

henrymercer requested a review from a team as a code owner January 11, 2023 19:11

henrymercer force-pushed the henrymercer/controlled-switchover branch from a9eb86f to e8c12e1 Compare January 11, 2023 19:11

angelapwen reviewed Jan 11, 2023

View reviewed changes

Comment thread src/util.ts Outdated

Comment thread src/codeql.ts Outdated

Comment thread src/feature-flags.test.ts

Comment thread src/setup-codeql.ts

henrymercer added 8 commits January 12, 2023 15:41

Allow using a x.y.z-yyyymmdd toolcache version for CLI x.y.z.

648838c

Remove dead code

1eeb9df

Expand note about defaults.json compatibility

a89ad76

In the future we might convert the file to JSON with comments, or add this note to a JSON schema.

Add doc describing CLI version marker files

31c7ce1

Merge branch 'main' into henrymercer/controlled-switchover

9578699

Add changelog notes

28f827a

Bump version to 2.2.0

c9b1be5

Cache explicitly requested bundles with their URL if possible

c2e39e0

henrymercer requested a review from angelapwen January 12, 2023 20:44

henrymercer added 2 commits January 12, 2023 20:56

Include the bundle version in the toolcache version number

33206d2

This gives us an easy cache hit when requesting the same tools URL.

Use the CLI version when caching the bundle in telemetry too

0be20e5

aeisenberg reviewed Jan 12, 2023

View reviewed changes

Comment thread CHANGELOG.md Outdated

henrymercer added 2 commits January 13, 2023 16:18

Convert logger call to debug

12998b7

Improve changelog notes

13cdac3

henrymercer requested review from AlonaHlobina and aeisenberg January 13, 2023 16:25

Further improve changelog notes

eca06a5

aeisenberg reviewed Jan 13, 2023

View reviewed changes

Comment thread CHANGELOG.md Outdated

Comment thread CHANGELOG.md Outdated

Comment thread src/setup-codeql.ts Outdated

henrymercer added 3 commits January 16, 2023 09:53

Improve changelog note some more

ac7e4d7

Merge branch 'main' into henrymercer/controlled-switchover

115587a

Address review comments

b660a38

Add a sentence on recommended practices for using the CLI

a5b44c1

henrymercer requested a review from aeisenberg January 16, 2023 10:27

angelapwen previously approved these changes Jan 17, 2023

View reviewed changes

henrymercer added 3 commits January 18, 2023 15:37

Merge branch 'main' into henrymercer/controlled-switchover

5f1362d

Support cli-version-x.y.z-pre.txt marker files

8a4abfd

Update changelog note

3d62f02

henrymercer dismissed angelapwen’s stale review via 3d62f02 January 18, 2023 15:59

angelapwen previously approved these changes Jan 18, 2023

View reviewed changes

Merge branch 'main' into henrymercer/controlled-switchover

9012214

henrymercer dismissed angelapwen’s stale review via 9012214 January 18, 2023 22:39

henrymercer requested a review from angelapwen January 18, 2023 22:39

angelapwen approved these changes Jan 19, 2023

View reviewed changes

henrymercer enabled auto-merge January 19, 2023 09:41

henrymercer merged commit 60e5868 into main Jan 19, 2023

henrymercer deleted the henrymercer/controlled-switchover branch January 19, 2023 09:42

henrymercer mentioned this pull request Jan 23, 2023

Use CodeQL path from init Action in Windows Python dependencies PR checks #1502

Merged

3 tasks

github-actions Bot mentioned this pull request Jan 26, 2023

Merge main into releases/v2 #1511

Merged

6 tasks

Enable a controlled switchover between CodeQL releases by henrymercer · Pull Request #1475 · github/codeql-action · GitHub

Uh oh!

Enable a controlled switchover between CodeQL releases - #1475

Enable a controlled switchover between CodeQL releases#1475henrymercer merged 29 commits intomaingithub/codeql-action:mainfrom henrymercer/controlled-switchovergithub/codeql-action:henrymercer/controlled-switchoverCopy head branch name to clipboard

Conversation

henrymercer commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO before merging

Merge / deployment checklist

Uh oh!

Uh oh!

henrymercer commented Jan 11, 2023

Uh oh!

angelapwen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aeisenberg commented Jan 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angelapwen commented Jan 13, 2023

Uh oh!

henrymercer commented Jan 13, 2023

Uh oh!

aeisenberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henrymercer commented Jan 16, 2023

Uh oh!

angelapwen left a comment

Choose a reason for hiding this comment

Uh oh!

angelapwen left a comment

Choose a reason for hiding this comment

Uh oh!

henrymercer commented Jan 18, 2023

Uh oh!

henrymercer commented Jan 18, 2023

Uh oh!

angelapwen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Enable a controlled switchover between CodeQL releases#1475
henrymercer merged 29 commits into
mainfrom
henrymercer/controlled-switchover

henrymercer commented Jan 10, 2023 •
edited
Loading

aeisenberg commented Jan 12, 2023 •
edited
Loading