| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
There was a problem hiding this comment.
Caching these invocations makes a lot of sense! I have a high level comment and a couple of lower level comments.
The main point is that now that we're caching multiple invocations, it might be a good opportunity to generalise the design. For instance, you could imagine something like:
const versionCache = createPersistedCliCache({ envVar: EnvVar.CODEQL_VERSION_INFO, validate: isVersionInfo });
const resolveLanguagesCache = createPersistedCliCache({ envVar: EnvVar.CODEQL_RESOLVE_LANGUAGES, validate: isResolveLanguagesOutput });where createPersistedCliCache handles memoising in the Action and persisting between Actions steps with an environment variable.
Some smaller things:
Sorry, something went wrong.
There was a problem hiding this comment.
I agree with @henrymercer's comments regarding a more generalised design for this. I am wondering about the use of environment variables here vs using a file on disk. I don't know if you have already considered this, but we store e.g. the Action configuration on disk as a file. Perhaps that would make sense for these cached CLI results as well.
A general point: could we also make sure to add doc comments for new top-level definitions before merging?
Sorry, something went wrong.
Repeated calls to `resolveLanguages()` will only pay the performance penalty of executing `codeql resolve languages` once.
By wrapping `resolveLanguages()`, which is memoized, we can avoid executing `codeql resolve extractor` several times over the course of an analysis.
This commit adds a `number` validator`, an `object` validator, an `isNumber` predicate, and `undefinable()` to test optional-but-not-null properties.
This provides a separation of concerns between the memoization and the execution.
|
I've taken your comments into consideration and overhauled the design to be more comprehensive and unified. The design now backs to a temporary file instead of the environment. I also identified a few opportunities to refactor some duplicated code into helper functions. I kept the use of cmd as a key in the cache, but I question whether it's really necessary. I think it's safe to assume that, in most cases, there will only be one instance of codeql in use per job. And, even in the event that there's more than one instance, how likely is it that init would use a different version than autobuild or analyze? If it's not necessary, I would opt to delete it to simplify the code a bit. |
Sorry, something went wrong.
There was a problem hiding this comment.
Warning
This PR introduces a cross-step cache for selected CodeQL CLI command outputs (notably codeql version and codeql resolve languages) to reduce repeated JVM startups and improve performance across GitHub Actions steps. It also refactors extractor resolution to derive extractor roots from resolve languages (reusing the cached output) and extends the internal JSON validation helpers to support stronger runtime validation of CLI JSON output.
Changes:
| File | Description |
|---|---|
| src/util.ts | Removes the prior in-process/env-var version cache helpers. |
| src/util.test.ts | Removes tests for the old version-caching behavior. |
| src/testing-utils.ts | Updates test setup to reset the new command-output cache between tests. |
| src/status-report.ts | Switches telemetry version lookup to the new cache + isVersionInfo guard. |
| src/json/index.ts | Adds number, object, and undefinable validators to support schema checks. |
| src/json/index.test.ts | Adds tests for undefinable semantics (rejecting null). |
| src/environment.ts | Removes the env var used for the old persisted version cache. |
| src/codeql.ts | Adds caching wrappers/type guards and refactors extractor resolution and JSON parsing. |
| src/cache.ts | New: implements the command-output cache (memo + temp file). |
| src/cache.test.ts | New: tests cache persistence/memo behavior and validation. |
| lib/entry-points.js | Generated output (content excluded by policy; not reviewed). |
Sorry, something went wrong.
…e support The output of `resolveLanguages()` can vary based on whether the flag `--filter-to-languages-with-queries` is included, but not all versions of the CLI support that. This makes caching a single execution problematic, so I opted to cache it based on whether it's supported. If it's supported, it's used; otherwise, it's not.
There was a problem hiding this comment.
Thanks for implementing the generic cache mechanism, that's a nice simplification! To make this PR easier to review and get merged, what do you think about splitting this PR into two: the first would implement the generic cache mechanism just for codeql version, and the second would use it to cache codeql resolve languages?
Sorry, something went wrong.
There was a problem hiding this comment.
I haven't finished reviewing everything (particularly the new tests as well as codeql.ts), but I already have a bunch of comments that I think should be addressed. So to not end up with too much noise in one go, I am submitting my review without reviewing those last bits.
The main thing to look at is that the on-disk cache is repeatedly written to and read from, which obviously has unnecessary overheads. It would be better if the on-disk cache was only read once (if it exists) to initialise the in-memory cache. It should then only be written to when the Action is about to exit.
Secondly, it would be nice (but is not essential) if we could avoid having the in-memory cache as global state and instead propagate it between the relevant use sites. Since we already have a CodeQL object, we could probably make the command cache part of it. That would make the data flow more explicit and simplify the tests, which then don't have to clear the global cache and can additionally run in parallel.
Sorry, something went wrong.
| /** | ||
| * Clears the in-process memo (tier 1). Only for use in tests, which exercise | ||
| * multiple "steps" within a single process. | ||
| */ | ||
| export function resetCachedCommandOutputs(): void { | ||
| inMemoryCache.clear(); | ||
| } |
There was a problem hiding this comment.
In some other files, we export these test-only functions in a nested object to make it clearer that they are for test-use only.
That said, my general preference for global state in the CodeQL Action is to try and make it explicit in the way I have done recently in e.g. #3963
Sorry, something went wrong.
There was a problem hiding this comment.
I like the idea of not using global state as a cache, but I wonder if that might be a big shift from the current implementation. And, in the spirit of keeping this PR "small" and tightly-scoped, I think that should be a separate PR.
I'd like to keep this PR focused on generalizing the caching implementation into memory and file backings.
Sorry, something went wrong.
…ated schemas to use `optionalOrNull`
This key-check is unnecessary because this function is only ever called if the key does NOT exist in the cache already. In that case, why check again?
This implementation saves CPU and I/O by not trying to write to the file on every set-cache.
This change should hopefully make the purpose of this module clear.
This move simplifies the consumer-side of the module. And, also, the validators should be an internal implementation detail—not the concern of those who wish to "get" a cached output.
The code reads better/simpler as an `enum`.
These have been spun off into #3980.
This is secondary work that will follow from the main purpose of this PR.
It's been forked off into #3981
# Conflicts: # lib/entry-points.js # src/codeql.ts # src/environment.ts # src/status-report.ts
These hunks were related to another hunk elsewhere, that has since now been reverted in favor of a separate/distinct PR.
This CLI command was cached, at one point, but that's been saved for another PR, so this change will too.
There was a problem hiding this comment.
Thanks for continuing to work on this, and addressing earlier feedback.
Unfortunately, I still don't think this is ready. There are a few things going on which suggest that the changes aren't finished yet, such as that the caching is not used for resolveLanguages and that only the init action stores the on-disk cache.
Further, there continue to be two high-level design points that I am concerned about:
A concern is also self-hosted runners, where the on-disk cache may persist between workflows. We should make sure that this either doesn't happen (by explicitly deleting the file in e.g. init-post) or storing e.g. the analysis UUID in the file and deleting it if it doesn't match.
I'd also like us to be careful with the new validators you've added. These shouldn't accidentally end up in the path of direct CLI output, because that would make this a high-risk change.
To get on with things and have something that can be merged sooner-than-later, perhaps it would be good to break this down into at least two parts (each as separate PR):
Sorry, something went wrong.
| export interface VersionInfo { | ||
| version: string; | ||
| features?: { [name: string]: boolean }; | ||
| /** | ||
| * The overlay version helps deal with backward incompatible changes for | ||
| * overlay analysis. When a precompiled query pack reports the same overlay | ||
| * version as the CodeQL CLI, we can use the CodeQL CLI to perform overlay | ||
| * analysis with that pack. Otherwise, if the overlay versions are different, | ||
| * or if either the pack or the CLI does not report an overlay version, | ||
| * we need to revert to non-overlay analysis. | ||
| */ | ||
| overlayVersion?: number; | ||
| } |
There was a problem hiding this comment.
This type doesn't seem like it really belongs here; i.e. it has nothing to do with the output caching specifically. I assume that you've moved it to avoid a circular dependency. There are two ways of addressing this:
Sorry, something went wrong.
| features: { | ||
| validate: isBooleanRecord, | ||
| check: (obj) => ({ | ||
| unknownKeys: [], | ||
| invalidKeys: [], | ||
| valid: isBooleanRecord(obj), | ||
| }), | ||
| required: false, | ||
| }, |
There was a problem hiding this comment.
This could probably be turned into a reusable record validator in json/index.ts. It would also be good to populate invalidKeys correctly.
Sorry, something went wrong.
| /** | ||
| * Clears the in-process memo (tier 1). Only for use in tests, which exercise | ||
| * multiple "steps" within a single process. | ||
| */ | ||
| export function resetCachedCommandOutputs(): void { | ||
| inMemoryCache.clear(); | ||
| } |
There was a problem hiding this comment.
See my earlier comment about exporting this via an object named to indicate that this is for testing only.
Sorry, something went wrong.
| /** | ||
| * Returns the cached output for `key`, or `undefined` if it isn't cached. | ||
| * | ||
| * Resolves tier 1 (in-memory memo) first, then tier 2 (temporary file). A value |
There was a problem hiding this comment.
Minor: The phrase "in-memory memo" is used a few times. This seems odd to me for two reasons:
Sorry, something went wrong.
There was a problem hiding this comment.
Yeah... that's Copilot 😞
Sorry, something went wrong.
| key: K, | ||
| cmd?: string, | ||
| ): CommandCacheKeyOutputMap[K] | undefined { | ||
| // Tier 1: the in-memory variable. |
There was a problem hiding this comment.
| // Tier 1: the in-memory variable. | |
| // Try to retrieve the result from memory first. |
Sorry, something went wrong.
| core.setOutput("codeql-version", (await codeql.getVersion()).version); | ||
|
|
||
| // Persist the command cache to disk at the end of a successful init. | ||
| writeCommandCacheFile(); |
There was a problem hiding this comment.
Why is this only done for the init action? Consider hooking this into runInActions in action-common.ts.
Sorry, something went wrong.
There was a problem hiding this comment.
My thinking here is that many (most?) of the CLI commands that codeql-action invokes are not worth caching—either because we won't need the output again or because the value of the command is in the side-effect, not the output (e.g. ,codeql database create).
Therefore, I think it would make sense to invoke all of the cache-able CLI commands in init, either as-needed or pre-emptively. Once that's done, the cache would presumably be set for the life of the workflow run. There would be no need to write the cache to file in autobuild or analyze or upload-sarif.
I could be wrong here, of course, but I cannot think of a CLI command that we would invoke for the first time in those steps that we'd want to cache.
Sorry, something went wrong.
There was a problem hiding this comment.
Therefore, I think it would make sense to invoke all of the cache-able CLI commands in init, either as-needed or pre-emptively. Once that's done, the cache would presumably be set for the life of the workflow run. There would be no need to write the cache to file in autobuild or analyze or upload-sarif.
I agree with your assessment that probably not all actions run CLI commands that are worth caching. If you don't have this already, it would be good to put together an overview of which CLI commands are used by which action. That can then guide us.
That said, I'd also like to avoid an unintuitive implementation where restoring/storing the on-disk cache is handled in a fairly ad-hoc manner that may not be transparent when making changes down the line.
Sorry, something went wrong.
| // Populate the in-memory command cache with CLI version. | ||
| // This result will be persisted to disk at the end of the init action. | ||
| await codeql.getVersion(); |
There was a problem hiding this comment.
Does this have any benefit over just waiting for the first, existing getVersion call?
Sorry, something went wrong.
There was a problem hiding this comment.
Right now, no.
I was setting this up for a subsequent PR in which the cache is pre-loaded concurrently at the start of init, to avoid some of the "waiting" penalty of executing these commands.
Sorry, something went wrong.
There was a problem hiding this comment.
Should resolveLanguages use getCachedOrRun as well?
Sorry, something went wrong.
There was a problem hiding this comment.
Yes, but that's for a follow-up PR, per @henrymercer's comment.
Sorry, something went wrong.
There was a problem hiding this comment.
But this PR still includes other resolveLanguages-related changes (like moving the output type and adding the validator). Either leave them all for a follow-up PR as per Henry's suggestion, or do it all in this PR.
Sorry, something went wrong.
| export interface ResolveLanguagesOutput { | ||
| aliases?: { | ||
| [alias: string]: string; | ||
| }; | ||
| extractors: { | ||
| [language: string]: Array<{ | ||
| extractor_root: string; | ||
| extractor_options?: any; | ||
| }>; | ||
| }; | ||
| } |
There was a problem hiding this comment.
Same comment as for VersionInfo.
Sorry, something went wrong.
| interface StoredCommandCacheEntry { | ||
| cmd: string; | ||
| output: unknown; | ||
| } | ||
|
|
||
| /** A single cached command output together with the CLI path it came from. */ | ||
| export interface CommandCacheEntry<K extends CommandCacheKey> { | ||
| /** | ||
| * The path to the CodeQL CLI that produced `output`. Persisted so that a | ||
| * different step using a different CodeQL bundle doesn't pick up a stale | ||
| * value. | ||
| */ | ||
| cmd: string; | ||
| output: CommandCacheKeyOutputMap[K]; | ||
| } |
There was a problem hiding this comment.
These two types are identical except for the type of output. Could you simplify this down by making the type of output polymorphic in a type variable?
If the K extends CommandCacheKey constraint is hard to lift, then perhaps just the following might work:
| interface StoredCommandCacheEntry { | |
| cmd: string; | |
| output: unknown; | |
| } | |
| /** A single cached command output together with the CLI path it came from. */ | |
| export interface CommandCacheEntry<K extends CommandCacheKey> { | |
| /** | |
| * The path to the CodeQL CLI that produced `output`. Persisted so that a | |
| * different step using a different CodeQL bundle doesn't pick up a stale | |
| * value. | |
| */ | |
| cmd: string; | |
| output: CommandCacheKeyOutputMap[K]; | |
| } | |
| /** A single cached command output together with the CLI path it came from. */ | |
| interface BaseCommandCacheEntry<O = unknown> { | |
| /** | |
| * The path to the CodeQL CLI that produced `output`. Persisted so that a | |
| * different step using a different CodeQL bundle doesn't pick up a stale | |
| * value. | |
| */ | |
| cmd: string; | |
| output: O; | |
| } | |
| export type CommandCacheEntry<K extends CommandCacheKey> = BaseCommandCacheEntry<CommandCacheKeyOutputMap[K]>; |
Sorry, something went wrong.
|
This PR is too big and vast. Per the recommendations, I've been broken it up into several smaller PRs: |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Similar to #3943, this PR caches the output of codeql resolve languages, which contains the paths to the various extractors so that repeated calls to resolveLanguages() are idempotent. Additionally, re-implement resolveExtractor() as a wrapper over resolveLanguages() (to re-use the cached output) rather than shell out to codeql resolve extractor.
In one experiment, I counted seven instances of shelling out to codeql resolve extractor. When you dig into the code, you can see why: resolveExtractor() is not called often or from many places; But one caller is isTracedLanguage(), which is wrapped by isScannedLanguage(). And these functions are often used in a loop/map over all/some languages. This can explain why we see consecutive executions of codeql resolve extractor.
In support of the above goals, this PR also adds some additional functions to the json module, to enable validation of the codeql version output.
Risk assessment
For internal use only. Please select the risk level of this change:
Which use cases does this change impact?
Workflow types:
Products:
Environments:
How did/will you validate this change?
If something goes wrong after this change is released, what are the mitigation and rollback strategies?
How will you know if something goes wrong after this change is released?
Are there any special considerations for merging or releasing this change?
Merge / deployment checklist