FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add RAM and threads options to init action by cklin · Pull Request #738 · github/codeql-action · GitHub

/ codeql-action Public

Add RAM and threads options to init action #738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cklin merged 1 commit into from
Oct 28, 2021
Conversation 57 Commits 1 Checks 0 Files changed
Merged

Add RAM and threads options to init action #738

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension .js  (6) .map  (6) .md  (1) .ql  (1) .ts  (6) .yml  (5) All 6 file types selected
Only manifest files
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
63 changes: 63 additions & 0 deletions .github/workflows/__extractor-ram-threads.yml
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

39 changes: 39 additions & 0 deletions .github/workflows/pr-checks.yml
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -393,3 +393,42 @@ jobs:
# Deliberately don't use TEST_MODE here. This is specifically testing
# the compatibility with the API.
runner/dist/codeql-runner-linux upload --sarif-file src/testdata/empty-sarif.sarif --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

runner-extractor-ram-threads-options:
name: Runner ubuntu extractor RAM and threads options
needs: [check-js, check-node-modules]
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2

- name: Build runner
run: |
cd runner
npm install
npm run build-runner

- name: Run init
run: |
runner/dist/codeql-runner-linux init --ram=230 --threads=1 --repository $GITHUB_REPOSITORY --languages java --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

edoardopirovano Sep 15, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think I'd slightly prefer this to use something other than 1 for the threads as 1 could very easily be a default in which case we wouldn't really be checking anything. Maybe 3 as that seems very unlikely to ever be the default value.

cklin Sep 15, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is a decision that I struggled with.

The main challenge here is that the number of threads used defaults to (and is capped at) the number of CPU cores available on the machine. GitHub action runners have 2 CPUs, so both the default and the cap would be 2. Passing any value larger than 1 means that the test will break if it runs on a system with different number of CPUs.

I would love to hear your thoughts. For example, maybe we could assume that the runner will always have at least 2 CPUs and have two tests with threads set to 1 and 2?

edoardopirovano Sep 15, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Aha, this is a good point. As I mentioned below, I'm actually not sure that capping this is a good idea, but for now let's leave this test as is 🙂

- name: Assert Results
shell: bash
run: |
. ./codeql-runner/codeql-env.sh
if [ "${CODEQL_RAM}" != "230" ]; then
echo "CODEQL_RAM is '${CODEQL_RAM}' instead of 230"
exit 1
fi
if [ "${CODEQL_EXTRACTOR_JAVA_RAM}" != "230" ]; then
echo "CODEQL_EXTRACTOR_JAVA_RAM is '${CODEQL_EXTRACTOR_JAVA_RAM}' instead of 230"
exit 1
fi
if [ "${CODEQL_THREADS}" != "1" ]; then
echo "CODEQL_THREADS is '${CODEQL_THREADS}' instead of 1"
exit 1
fi
if [ "${CODEQL_EXTRACTOR_JAVA_THREADS}" != "1" ]; then
echo "CODEQL_EXTRACTOR_JAVA_THREADS is '${CODEQL_EXTRACTOR_JAVA_THREADS}' instead of 1"
exit 1
fi
2 changes: 1 addition & 1 deletion CHANGELOG.md
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## [UNRELEASED]

No user facing changes.
- The `init` step of the Action now supports `ram` and `threads` inputs to limit resource use of CodeQL extractors. These inputs also serve as defaults to the subsequent `analyze` step, which finalizes the database and executes queries. [#738](https://github.com/github/codeql-action/pull/738)

## 1.0.21 - 28 Oct 2021

Expand Down
14 changes: 12 additions & 2 deletions analyze/action.yml
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@ inputs:
required: false
default: "brutal"
ram:
description: Override the amount of memory in MB to be used by CodeQL. By default, almost all the memory of the machine is used.
description: >-

adityasharad Oct 21, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Since the input descriptions here are going to vary between the two actions, we should add an exception to the custom query that's producing the code scanning warning on this file.

cklin Oct 21, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I edited queries/inconsistent-action-input.ql to exclude inputs named ram or threads from the query.
The amount of memory in MB that can be used by CodeQL for database finalization and query execution.
By default, this action will use the same amount of memory as previously set in the "init" action.
Comment thread
adityasharad marked this conversation as resolved.
If the "init" action also does not have an explicit "ram" input, this action will use most of the
memory available in the system (which for GitHub-hosted runners is 6GB for Linux, 5.5GB for Windows,
and 13GB for macOS).
required: false
add-snippets:
description: Specify whether or not to add code snippets to the output sarif file.
Expand All @@ -29,7 +34,12 @@ inputs:
required: false
default: "false"
threads:
description: The number of threads to be used by CodeQL.
description: >-
The number of threads that can be used by CodeQL for database finalization and query execution.
By default, this action will use the same number of threads as previously set in the "init" action.
Comment thread
adityasharad marked this conversation as resolved.
If the "init" action also does not have an explicit "threads" input, this action will use all the
hardware threads available in the system (which for GitHub-hosted runners is 2 for Linux and Windows
and 3 for macOS).
required: false
checkout_path:
description: "The path at which the analyzed repository was checked out. Used to relativize any absolute paths in the uploaded SARIF file."
Expand Down
14 changes: 14 additions & 0 deletions init/action.yml
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@ inputs:
source-root:
description: Path of the root source code directory, relative to $GITHUB_WORKSPACE.
required: false
ram:
description: >-
The amount of memory in MB that can be used by CodeQL extractors.
By default, CodeQL extractors will use most of the memory available in the system
(which for GitHub-hosted runners is 6GB for Linux, 5.5GB for Windows, and 13GB for macOS).
This input also sets the amount of memory that can later be used by the "analyze" action.
required: false
threads:
description: >-
The number of threads that can be used by CodeQL extractors.
By default, CodeQL extractors will use all the hardware threads available in the system
(which for GitHub-hosted runners is 2 for Linux and Windows and 3 for macOS).
This input also sets the number of threads that can later be used by the "analyze" action.
required: false
outputs:
codeql-path:
description: The path of the CodeQL binary used for analysis
Expand Down
75 changes: 75 additions & 0 deletions lib/analyze-action-env.test.js
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions lib/analyze-action-env.test.js.map
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

75 changes: 75 additions & 0 deletions lib/analyze-action-input.test.js
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions lib/analyze-action-input.test.js.map
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Back | FazBrowse Home | New Git URL