Bundle codeql-bundle package data into the frozen binary - #39
Draft
nicolaswill wants to merge 3 commits into
Draft
Bundle codeql-bundle package data into the frozen binary#39nicolaswill wants to merge 3 commits into
nicolaswill wants to merge 3 commits into
Conversation
codeql-bundle's cache-aware bundle support reads two package data files at runtime via importlib.resources: codeql_bundle/supported-codeql-bundles.json codeql_bundle/supported-codeql-bundles.schema.json PyInstaller only freezes Python modules, so these were missing from the frozen binary QLT ships and every invocation aborted with an unhandled FileNotFoundError, breaking 'qlt codeql run install --custom-bundle' and '--quick-bundle'. Pass both files to PyInstaller with --add-data. The separator expected by --add-data is os.pathsep, which is ';' on Windows and ':' elsewhere, so it is resolved from [System.IO.Path]::PathSeparator rather than hardcoded -- this script runs on all three platforms. The Test-Path guard keeps the script working for older codeql-bundle releases that do not ship these files, and the final argument list is echoed before the build. Also bump the pinned codeql-bundle version to 0.5.0, which is the release that introduces these files.
Expose the codeql-bundle cache directory through QLT, either as a 'CacheDir' key in qlt.conf.json or as 'qlt codeql run install --cache-dir' on the command line, with the command line taking precedence. The directory holds downloaded bundles and published compilation caches, so restoring it across CI runs (e.g. with actions/cache) avoids re-downloading and lets the cache-aware bundle support reuse prebuilt compilation caches. Relative paths are resolved against the caller's working directory before being handed to the bundle tool, which runs with its working directory set to the QLT base path.
The example pinned codeql-bundle-v2.15.5. The compilation cache catalog only covers v2.18.0 and later, so the example got a warning and no cache benefit. Move the CLI, standard library and bundle pins to 2.26.1, the current release.
There was a problem hiding this comment.
Pull request overview
This PR updates the CodeQL Toolkit (QLT) packaging and install flow to better support codeql-bundle by bundling required package data into the frozen binary, and by exposing a cache directory knob to make bundle downloads/compilation caches reusable across runs (especially CI).
Changes:
- Add
CacheDirsupport in config and pass it through to the bundle tool via--cache-dir, with CLI override support. - Update the
codeql-bundlePyInstaller build script to include runtime package data files when present. - Bump referenced
codeql-bundleversion to0.5.0in docs/workflows and update the example config to CodeQL2.26.1.
Show a summary per file
| File | Description |
|---|---|
| src/CodeQLToolkit.Shared/Utils/QLTConfig.cs | Adds CacheDir to configuration model for bundle-tool cache pass-through. |
| src/CodeQLToolkit.Shared/CodeQL/CodeQLInstallation.cs | Threads CacheDir into installation and prepends --cache-dir to bundle tool args. |
| src/CodeQLToolkit.Features/CodeQL/Commands/Targets/InstallCommand.cs | Allows command-line --cache-dir to override config-loaded cache directory. |
| src/CodeQLToolkit.Features/CodeQL/Commands/CodeQLCommandFeature.cs | Adds --cache-dir option wiring for the install command. |
| scripts/build_codeql_bundle_dist.ps1 | Adds PyInstaller --add-data entries for codeql-bundle package data files. |
| example/qlt.conf.json | Updates example CodeQL versions/identifiers to 2.26.1. |
| developer_guide.md | Updates documented build command to use codeql-bundle 0.5.0. |
| .github/workflows/internal-build-release-win64.yml | Updates workflow to build/package codeql-bundle 0.5.0. |
| .github/workflows/internal-build-release-macos64.yml | Updates workflow to build/package codeql-bundle 0.5.0. |
| .github/workflows/internal-build-release-linux64.yml | Updates workflow to build/package codeql-bundle 0.5.0. |
| .github/actions/install-qlt-local/action.yml | Updates local install action to build/package codeql-bundle 0.5.0. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 11/11 changed files
- Comments generated: 1
- Review effort level: Low
Comment on lines
+274
to
+278
| if (CacheDir != null && CacheDir.Length > 0) | ||
| { | ||
| Log<CodeQLInstallation>.G().LogInformation($"Note: Using bundle cache directory `{CacheDir}` ..."); | ||
| bundleArgs = $"--cache-dir \"{Path.GetFullPath(CacheDir)}\" {bundleArgs}"; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements cache-aware bundle support in QLT: adds the codeql-bundle package data files to the PyInstaller build, pins codeql-bundle 0.5.0, adds a
--cache-dirpass-through, and bumps the example to CodeQL 2.26.1. Blocked on advanced-security/codeql-bundle#23 and thev0.5.0release.