Skip to content

STAC-25461 report release-branch build failures to Slack via Cerberus - #449

Open
LouisParkin wants to merge 3 commits into
STAC-25142-agent-lint-unitfrom
STAC-25461-cerberus-notify
Open

STAC-25461 report release-branch build failures to Slack via Cerberus#449
LouisParkin wants to merge 3 commits into
STAC-25142-agent-lint-unitfrom
STAC-25461-cerberus-notify

Conversation

@LouisParkin

Copy link
Copy Markdown

Replaces GitLab's notify-on-master-fail with a Cerberus call from GitHub Actions.

What

  • New reusable workflow cerberus-notify.yml — one curl to the Cerberus Lambda with platform: github in the context, so it builds GitHub pipeline/commit URLs. Calling convention copied from cerberus-block-on-master-fail in StackVista/stackstate's ci.yml.
  • push: on stackstate-7.78.2 added to the three pipeline workflows. They were pull_request + workflow_dispatch only, so there was no release-branch lane for a notification to hang off.
  • A terminal cerberus-notify job in each, passing its own suite label (lint-and-unit-tests, binary-builds, deb-package) so the Slack message names what broke.

notify, not block

Per the policy for migrated repos. action: block locks the branch via lock_branch, needs the Cerberus GitHub App installed on the repo, and mutates Pulumi-managed branch protection out from under pulumi-infra. None of that is wanted here — the GitLab job it replaces only notified.

Details worth a look

  • The if gate is just github.event_name == 'push'. The push: filter already restricts to the release branch, so re-checking github.ref would duplicate the literal in two places that must agree. PRs cannot reach the job.
  • needs lists every job, not only the leaves. If build-deb fails, test-deb-renaming and build-agent-image are skipped, not failed — contains(needs.*.result, 'failure') over the leaves alone would miss it.
  • .cerberus/cerberus_notify_failure.sh is not reused. It is built around GitLab's CI_* variables and predates the platform field. It stays in the tree until GitLab is retired (STAC-25464).
  • docker-public, not xlarge-public — it is a curl. It runs the same sts-ci-images runner image as the small scale set that StackVista/stackstate's Cerberus job uses, so curl and jq are present.
  • permissions: {} on the reusable workflow. The payload comes entirely from the github context; there is no checkout.

Prerequisite, not yet satisfied

secrets.CERBERUS_LAMBDA_URL and vars.SLACK_CI_REPORT_CHANNEL need to reach this repo at repo level through pulumi-infra. Org secrets default to visibility=private, which excludes this PUBLIC repo — the same constraint STAC-25350 hit for the REGISTRY_* credentials.

Until they exist the step emits a ::warning:: and exits 0. It deliberately does not fail: the run is already red, and a second red job would read as "Cerberus is broken" rather than "Cerberus is not configured yet". The annotation is the signal.

Validation

  • zizmor --collect=workflows,actions,dependabot . — no findings.
  • All four workflows parse; job graphs are as intended.
  • End-to-end cannot be proven from a PR: the job only runs on a failed push to the release branch, which does not exist until this integration branch merges and the secret is provisioned. First real exercise is the first red release-branch build after cutover.

Jira: STAC-25461

@LouisParkin

Copy link
Copy Markdown
Author

Follow-up pushed (46a089ea97) plus the infra half:

SLACK_CI_REPORT_CHANNEL is a secret, not a variable. pulumi-infra declares it type: secret in github/orgVariables/resources_gitlab.yaml, so the original vars.SLACK_CI_REPORT_CHANNEL resolved to the empty string. Now read from secrets and threaded through the reusable workflow's secrets: block from all three callers. Worth noting StackVista/stackstate's cerberus-block-on-master-fail has the same bug (.github/workflows/ci.yml:773) — its payload has been carrying a blank channel and relying on the Lambda default.

Visibility prerequisite: StackVista/pulumi-infra#257 flips both CERBERUS_LAMBDA_URL and SLACK_CI_REPORT_CHANNEL to visibility: all. Org secrets default to private, which excludes this public repo — same wall STAC-25350 hit for REGISTRY_*. Went org-level rather than repo-level because selected visibility would mean enumerating every current private-repo consumer by ID, and repo-level entries would need the secret values re-set in the repoVariables stack.

Until #257 lands the notify job annotates and exits 0, so this PR is safe to merge first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant