[BUG] Elemental 7.x.x allows eXist-db EXPath Packages to be updated to unstable versions

[adamretter]

This bug occurs due to the eXist-db bugs eXist-db/eXide#822 and eXist-db/existdb-openapi#57. Background, full details, and a proposed solution follow

---

EXPath Packages can declare a dependency on a processor. For example, a dependency on the eXist-db processor would look like the following in the EXPath packages expath-pkg.xml:

 <dependency processor="http://exist-db.org" version="7.0.0-SNAPSHOT"/>

A dependency on the Elemental processor instead looks like:

<dependency processor="http://elemental.xyz" semver-min="7.6.0"/>

A package can declare multiple processor dependencies if it is compatible with more than one processor.

As there is a large body of existing EXPath packages for the eXist-db processor, we declare a compatibility level with the eXist-db processor. This is defined in elemental-parent/pom.xml:

<exist-db.expath.pkg.compatible.version>6.4.0</exist-db.expath.pkg.compatible.version>

This is stating that this particular version of Elemental is compatible with EXPath Packages that declare a dependency on an eXist-db processor version 6.4.0 or later.

This value is templated into exist-core/src/main/resources-filtered/org/exist/system.properties:

exist-db-expath-pkg-compatible-version=${exist-db.expath.pkg.compatible.version}

It is then available from Java by reading the System Property: exist-db-expath-pkg-compatible-version.

This is used in two places:

1. org.exist.repo.Deployment#checkExistDbProcessorVersion(PackageLoader.Version) - it is used here to stop packages that are not compatible from being installed. If you try and install a package that is not compatible, an exception will be raised:

   throw new PackageException("Package requires eXist-db version: " + version + ". " +
                   "Installed version is Elemental: " + procVersion + ", with compatibility for eXist-db version: " + eXistCompatibleProcVersion);

2. org.exist.repo.ClasspathHelper#isCompatible(Package) - it is used here to prevent loading Java classes from an EXPath package that are not compatible. If it detects an incompatibility it logs a warning in the file `logs/expath-repo.log:

   LOG.warn("Package {} is not compatible with this version of Elemental. To avoid conflicts, Java libraries shipping with this package are not loaded.", pkg.getName());

When looking at the Package Manager app on the eXist-db Dashboard. That app uses a module called Package Service to retieve information about available updates to installed packages from eXist-db's EXPath Public Repo.

In modules/packages.xqm of the Package Service, it requests the version of the processor it is running on:

declare function packages:get-version() {
    (util:system-property("product-semver"), util:system-property("product-version"))[1]
};

When running on the latest versions of eXist-db this will return 6.4.0 or 7.0.0-beta3, when running on the latest versions of Elemental this will return 6.10.0 or 7.6.0.

The Package Service then uses that information to access a URL like this: https://exist-db.org/exist/apps/public-repo/public/apps.xml?version=7.6.0

Unfortunately due to bugs like eXist-db/eXide#822 and eXist-db/existdb-openapi#57 in the way that eXist-db has chosen to incorrectly publish development versions of their EXPath packages, Elemental will see newer versions of packages than it should. This fundamentally comes from the simple fact that eXist-db are incorrectly publishing packages that depend on 7.0.0-SNAPSHOT that themselves are not declared as -SNAPSHOT versions.

Let me be clear here, Elemental is correctly following the standards. Sadly eXist-db appears to have recently chosen to break with the standard for its EXPath Packages (eXist-db/eXide#822) and (eXist-db/existdb-openapi#57).

We will need to work around the problems with the broken versions of these eXist-db packages. When we build an Elemental distribution, we already apply several patches to the pre-shipped Dashboard and Monex apps. We could consider also patching the Package Service so that instead of calling:

declare function packages:get-version() {
    (util:system-property("product-semver"), util:system-property("product-version"))[1]
};

We instead call something like:

declare function packages:get-existdb-compatible-version() {
    util:system-property("exist-db-expath-pkg-compatible-version")
};

This would have to be used only when connecting to the eXist-db EXPath Public Repo, and not other repos, as the eXist-db public repo (understandably) only thinks about eXist-db processors; sadly it is not a general purpose repo for EXPath packages.

Finally, we might consider how we apply these app patches. Perhaps we can find a better method. At the moment they are only applied at build-time, unfortunately this means that if a user validly updates an app, then the patches are not applied to the new version. We should likely move and/or copy such patches into the org.exist.repo.Deployment code pipeline, so that any install of a package that needs patches has them applied.