Implement cache-aware bundle optimization - #23
Merged
Conversation
Add an authoritative supported-bundle catalog, local cache tooling, verified per-language cache downloads, and release automation with cross-platform validation.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a cache-aware optimization layer to codeql-bundle by introducing a supported-bundle catalog and per-language compilation-cache assets, along with a new codeql-bundle-cache CLI and GitHub Actions automation to build/verify/publish those caches.
Changes:
- Add catalog + cache resolution/download/verification plumbing and integrate it into the custom bundle build path.
- Introduce
codeql-bundle-cacheCLI for planning, building, verifying caches, and updating the catalog. - Add unit tests plus CI workflows to run tests and to automate compilation-cache releases.
Show a summary per file
| File | Description |
|---|---|
| tests/workspace/cpp/foo-customizations/qlpack.yml | Relax test workspace dependency constraints for codeql/cpp-all. |
| tests/workspace/cpp/aa/qlpack.yml | Relax test workspace dependency constraints for codeql/cpp-all. |
| tests/test_codeql.py | Adds tests asserting explicit compilation-cache flags are used for pack create and query compile. |
| tests/test_cache.py | Adds coverage for catalog loading, bundle source resolution, safe extraction, and cache download/install behaviors. |
| tests/test_cache_targets.py | Adds tests for determining compilation-cache targets from pack dependency graphs. |
| tests/test_cache_cli.py | Adds tests for cache CLI commands (prune, build plan validation, catalog updates). |
| README.md | Documents new bundle source forms, compilation-cache behavior, cache CLI usage, and automation workflow. |
| pyproject.toml | Bumps version to 0.5.0 and registers the new codeql-bundle-cache entrypoint. |
| CONTRIBUTING.md | Adds local unit test invocation guidance and notes on local cache-release development. |
| codeql_bundle/supported-codeql-bundles.schema.json | Adds JSON schema for the supported bundle catalog format. |
| codeql_bundle/supported-codeql-bundles.json | Adds the initial (empty) supported bundle catalog as packaged data. |
| codeql_bundle/helpers/codeql.py | Adds streaming execution, compilation-cache options for pack create, and a new query_compile helper. |
| codeql_bundle/helpers/bundle.py | Integrates cache-aware behaviors into bundling flow and adds pack fingerprinting + target selection utilities. |
| codeql_bundle/cli.py | Extends --bundle to accept path/URL/release tag and wires in catalog/cache resolution flags. |
| codeql_bundle/cache.py | Introduces catalog, source resolution, download/verification, safe extraction, and cache installation primitives. |
| codeql_bundle/cache_cli.py | Implements codeql-bundle-cache CLI for planning/building/verifying/publishing caches and catalog updates. |
| .github/workflows/test.yml | Adds a unit test workflow (Python 3.11/3.12). |
| .github/workflows/codeql-compilation-caches.yml | Adds automation to build/verify/publish compilation caches and open catalog update PRs. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 18/18 changed files
- Comments generated: 2
- Review effort level: Low
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (1)
codeql_bundle/cache.py:906
- Same issue as above: touching
install_path(a directory) won’t update the entry’s timestamp for pruning/retention logic. Touch the marker file instead so cache usage extends retention.
_touch(install_path)
- Files reviewed: 18/18 changed files
- Comments generated: 3
- Review effort level: Low
Comment on lines
+117
to
+119
| for line in process.stdout: | ||
| output.append(line) | ||
| logger.info(line.rstrip()) |
Comment on lines
+120
to
+124
| return subprocess.CompletedProcess( | ||
| command_args, | ||
| process.wait(), | ||
| stdout="", | ||
| stderr="".join(output), |
| if self._is_installed( | ||
| marker_path, installed_cache_path, target, cache.asset.sha256 | ||
| ): | ||
| _touch(install_path) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
codeql pack create --compilation-cache=.cache.py,cache_cli.py(codeql-bundle-cache),codeql-compilation-caches.yml,supported-codeql-bundles.json+ schema as package data.--cache-dir,--cache-manifest,--no-compilation-cache.--threadsrespected on every compile path:pack createviacodeql.threads(default 0), andcodeql-bundle-cache build --threadsfor both the cache build and its verification. Standaloneverify*pins 0 (all cores) — one representative query.java-queries28m13s → 19m57s, wholeqlt ... install --custom-bundle43m55s → 29m33s. Partial reuse expected, that workspace extendscodeql/java-all.v0.5.0after merge — the toolkit pins it, and it unblockscodeql-compilation-caches.yml.