ci: run core's acceptance suite against the published image - #677
ci: run core's acceptance suite against the published image#677DeepDiver1975 wants to merge 4 commits into
Conversation
The build pipeline only proves the image builds, passes Trivy and answers /status.php. There is no functional test of the shipped artifact anywhere, so regressions in the image's own layers - entrypoint scripts, config template, app bundle - are only found by hand. Add a reusable acceptance workflow that runs core's Behat suite against a published owncloud/server tag. The suite talks to the container over HTTP: run.sh --remote proxies every occ call through the testing app's OCS endpoint, so no shell access to the server is needed. Test code comes from the release QA tarball, which is the same build as the shipped code and carries a version-matched testing app. This is a port of the pipeline the retired .drone.star already ran, so the shape is proven; the notable adaptations for Actions are: - Host networking throughout. The suite hands federated addresses such as localhost:8081 to the *server*, so they must resolve to the peer from inside the container. Actions service containers get a bridge network, where they would resolve to the container itself. - The testing app is bind-mounted from the QA tarball rather than installed from its GitHub release asset, which was last rebuilt in 2022. The mount must be writable because 25-chown.sh chowns the app tree at every start. - The enabled-app set is pinned explicitly to core CI's, since the Complete bundle default-enables a different set and a mismatch produces failures that look like server regressions. Wired into main.yml on master only, after build, defaulting to @smoketest so the ~4 min PR loop is untouched; the full set is available via the dispatch inputs. 10.16.3 is left out because 10.x ships no QA tarball. Verified against owncloud/server:11.0.0-rc3 locally: a smoke shard with both servers up reports 76 scenarios, 76 passed, 0 failed, including apiFederationToRoot1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
0d5ec20 to
16f78ee
Compare
|
https://github.com/owncloud-docker/server/actions/runs/30525799046 |
editorconfig-checker rejected the continuation lines of the SQL passed to mariadb: they were aligned to the opening quote, which is not a multiple of the 2-space indent width. Move the statements onto stdin via a heredoc. Repeated -e flags are not an option - mariadb concatenates their values into a single statement, so the GRANT would have been parsed as part of the CREATE DATABASE and the fed database would silently never have been created. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
phil-davis
left a comment
There was a problem hiding this comment.
This might work. I am glad that I didn't spend days trying to convert the drone starlark code into this! How do we actually run it to see a result?
As a first approach I want to run the tests in this PR with the 11.0.0-rc3 image - quick win for the upcoming release. In the long run a minimal smoke test should run upon every build - maybe a bigger suite - no idea yet ..... exploring .... |
workflow_dispatch cannot reach acceptance.yml yet: GitHub only registers dispatchable workflows from the default branch, and the file exists only on this branch. The acceptance job is also gated to master, so it showed up as "acceptance in 0s" on the PR. Let it run on pull_request and drop the @smoketest filter so this PR produces authoritative 11.0.0-rc3 numbers. Shards go to 20 api / 9 cli: run.sh slices by suite count distributing whole suites, so 20 and 9 are the maximum useful splits for 61 and 9 suites, and no shard ends up empty. The cli set carries only 3 @smoketest scenarios out of 386, which is why filtering it is pointless. Temporary - the PR loop is slow while this is in place. Revisit before merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
|
https://github.com/owncloud-docker/server/actions/runs/30528690247/job/90827472789?pr=677 The code underneath calls to the testing app to do: Is there anything special about trying to adjust the log level inside the container? |
|
The owncloud log for https://github.com/owncloud-docker/server/actions/runs/30528690247/job/90827472789?pr=677 has a lot of complaints about background jobs, for example: I suppose that many apps are in the disabled list, but their background jobs are still registered. Is there something that can be done to de-register background jobs for apps that are disabled, or? |
The first full-suite run against 11.0.0-rc3 came back with 49 failures out of 5040 scenarios. 40 of them were caused by this workflow, not the image. server.env set OWNCLOUD_LOG_LEVEL and four OWNCLOUD_MAIL_* keys. Those land in overwrite.config.php, which Config::readData() merges after config.php, so they beat anything config:system:set writes. Scenarios that have to *change* one of those settings therefore could not: - apiWebdavUpload1, apiWebdavUpload2 and cliBackground/backgroundQueue set the log level to debug and assert it took effect. All 39 reported "The expected log level is 0 but the log level has been set to 2". - cliProvisioning/resetUserPassword switches mail_smtpmode to sendmail to assert the send fails; with smtp pinned the mail went out and occ exited 0. The mail keys were redundant on top of that: run.sh already sets mail_domain, mail_from_address, mail_smtpmode, mail_smtphost and mail_smtpport itself via occ, and defaults EMAIL_SMTP_PORT to inbucket's 2500. Dropping both groups lets the suite own the settings it needs to mutate. This is the same overwrite.config.php precedence trap the file already documents for the database and redis keys - those must be set via env, these must not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
What
Adds
.github/workflows/acceptance.yml— a reusable + dispatchable workflow thatruns core's Behat acceptance suite against a published
owncloud/serverimage — and wires it into
main.ymlas anacceptancejob onmaster.Why
Today's CI (
docker-build.yml) proves the image builds, passes Trivy and answers/status.php. That's it. There is no functional test of the shipped artifact,so defects in the image's own layers — entrypoint scripts, the
config.phptemplate, the app bundle — are only ever found by hand.
The retired
.drone.starin this repo used to run exactly this pipeline, sothis is a port of something proven rather than a new invention.
How it works
…-qa.tar.bz2), which is thesame build as the shipped code and ships a version-matched
testingapp. Thebehat manifest, which the tarball omits, is fetched from the matching core tag.
run.sh --remoteproxies everyocccall through thetestingapp's OCSendpoint, so the suite needs no shell access to the container.
run.sh --part N M, mapped onto a matrix (api× 10,cli× 3 — kept at or below the suite count so no shard boots a full stackjust to find it has nothing to run).
Notable adaptations for Actions
docker run --network hostfor everything, notservices:localhost:8081to the server, so they must resolve to the peer from inside the container. On a bridge network they resolve to the container itself.testingapp bind-mounted from the QA tarball, writable25-chown.shchowns the whole app tree at every start — the entrypoint aborts with exit 123 if that fails, so:rokills the server.Installer::getShippedApps()enables everydefault_enableapp. The Complete bundle default-enables a different set; a mismatch shows up as failures that look like server regressions (e.g. disablingfiles_externalmakesapiAuthreport OCS 999 instead of 997).OWNCLOUD_REDIS_ENABLEDonly, noOWNCLOUD_MEMCACHE_LOCKINGarray_merge_recursive; setting both rendersmemcache.lockingas a two-element array instead of a class name.Trigger and scope
masteronly, afterbuild. Deliberately off PRs — those are mostlyrenovate digest bumps and the ~4 min loop should stay that fast.
@smokeTest. The full set is available viaworkflow_dispatch(clearfilter-tags, raiseshards).behat.ymlrather thanbehat.php, sorun.shwould need a differentinvocation. Noted in a comment on the matrix entry.
Verification
Run locally against
owncloud/server:11.0.0-rc3with the exact stack theworkflow builds (mariadb 10.6 + redis 8 + inbucket + two rc3 containers on
8080/8081):
status.phpgreen on both ports;occtransport returnscode=0on bothapp:listshows exactly the intended 13 enabled appsmemcache.lockingrenders as the\OC\Memcache\RedisscalarTotal 76 scenarios (76 passed, 0 failed), exit 0,"There were no unexpected failures." — including
apiFederationToRoot14/4,which is what proves the
--network hostfederation addressing works.actionlint(incl. its shellcheck pass) is clean on both files.Out of scope
webUIsuites (need Selenium; core's own CI never enables them).docker-build.ymlpushes to a job-localregistry and exposes no outputs, so that needs an upstream change in
owncloud-docker/ubuntu. Worth doing later — it would catch defects beforepublish rather than after.
apps_pathsis written unconditionally bythe config template so
AppManagementContext::setAppsPaths()can't mutate it,and
convert/ffmpegare absent (hence theadditional-packagesinput).Separate PRs.
🤖 Generated with Claude Code