Maintenance needs operational visibility

While deploying mod-oa, we noticed that a module was not reaching the startup state we expected. Following that anomaly through the pod state, application logs and database led back to a class of failures we had been dealing with operationally for years.

This time, we could investigate the running system ourselves. We could inspect restart behaviour, compare what the application reported with what PostgreSQL contained, and reproduce the failure using the exact deployed image in a disposable local environment.

That access changed what we could learn. It took us from a familiar stale-lock symptom to a reproduced failure mechanism and ownership races in shared infrastructure. Correcting those problems then took us through consuming-module upgrades, release qualification and our first live in-place Foundry tenant upgrade.

There was no new button at the end of this work. There was a considerable amount of maintenance.

We had seen this before

We had previously encountered similar startup and upgrade failures in externally hosted systems, including installations hosted by EBSCO and GBV. In those situations, we generally lacked sufficient direct access to the runtime, pod lifecycle, logs and database state to trace the failure properly. The eventual intervention often involved a DBA removing a stale lock so that the system could run again.

We maintained the code, but could not always observe the conditions in which it was failing. A recovered service was a useful outcome for the operator; it did not necessarily explain how the lock had become stranded.

What was already public

The Apache mod-agreements README already documents this problem under Locks and failure to upgrade. It describes failures associated with container termination during startup or upgrade, distinguishes Liquibase migration locks from application federation locks, and explains that the module may fail to recover by itself. It also records that attempts to fix the problem were ongoing.

The guidance covers CPU allocation, inspecting the relevant lock tables, allowing uninterrupted startup time and, as a last resort, clearing federation locks that have not recovered. It warns against killing the application or container during sensitive startup and upgrade periods.

The failure mode, diagnostic information and operational mitigations were already public. What we had been missing was a sufficiently reproducible environment and enough visibility to establish the underlying software defect. Finding that explanation was new work; discovering that locks could get stuck was not.

Following the lock back to its owner

mod-oa and mod-agreements both depend directly on grails-okapi, the shared Grails infrastructure that provides their FOLIO tenant integration and federation coordination. The lock in this incident was an application federation lock, separate from Liquibase’s migration lock.

Our local reproduction used the deployed mod-oa 3.0.0 image, containing grails-okapi 7.1.0. Its startup sequence could commit a federation promotion lock in one transaction while the owning application instance’s registration remained uncommitted in another.

If the process died in that interval, PostgreSQL rolled back the registration but retained the separately committed lock row. The next process started with a different instance identifier, saw a lock belonging to someone else, and kept waiting. Closing the dead process’s database connection could not release an ordinary persistent table row. Recovery also depended on federation leadership and cleanup that startup could not reach while blocked on the promotion lock.

We reproduced that sequence by deliberately holding up leader selection locally, killing the disposable module and restarting it. The resulting sleeping startup thread, open registration transaction and failing health checks matched the observed condition. Repeated termination could keep restarting the wait without removing its cause.

This explains why the operational symptom looked like a stale lock: that was exactly what remained visible after the failed transaction. The row did not preserve the sequence that created it. We could reproduce the mechanism, but the retained logs did not establish which original interruption had triggered the hosted incident. Nor does this prove that every earlier externally hosted failure had the same cause.

Finding the race

Investigating the shared locking code also exposed a concrete race between threads in the same module process. The ownership correction in grails-okapi shows the problem clearly.

The old code acquired database ownership before obtaining the local thread lock. Database ownership identified the application process, not the individual thread. A second thread could therefore obtain the same process-owned row, time out waiting for the local lock, and release the database lock while the first thread was still executing protected work. Another module process could then enter what was supposed to be an exclusive section.

There was a corresponding handover hazard: releasing the local lock before relinquishing database ownership could let a new local holder enter before the previous holder deleted its protection.

The correction takes the local lock first and releases database ownership before handing the local lock to another thread. Nested acquisition by the same thread retains ownership until the outermost release. The commit also confines legacy orphan cleanup to the correct application family and fixes shutdown cleanup to match the lock’s owner identifier rather than its own row identifier.

These are related ownership defects, but the thread race should not be mistaken for proof of how the historical orphan was created. The commit itself explicitly records that automatic orphan recovery tests still failed at that stage.

The completed repair, subsequently released in grails-okapi 9.2.0, goes further. It uses PostgreSQL session advisory locks by default behind the existing named-lock API. The recorded ownership decision explains the scope: PostgreSQL arbitrates ownership and releases it when the owning session ends, without needing an application leader to clear an abandoned row. Schema, application family and lock name remain part of its identity.

That has a defined boundary. Session-loss detection is not instantaneous in every network failure, and losing a database lock cannot revoke arbitrary Java work already running. Old row-lock and new advisory-lock processes must not share the same schema and application family during a rolling protocol switch. The deprecated legacy mode retains its orphan limitation. These constraints belong in the qualification and operating guidance, alongside the fix.

Proving it through consuming modules

A related manifestation was subsequently identified in mod-agreements, reinforcing the need to test this as shared infrastructure. Both Open Access and Agreements were upgraded to use the corrected library.

The repair was not complete merely because a JAR built. We exercised real PostgreSQL contention, process death, nested locking, startup, leader turnover and heartbeats, then checked the consuming applications and their dependency combinations. The final released library passed its federation and tenant-barrier regressions; Open Access passed all 42 integration tests. Agreements’ final CI reported 719 successful tests, 29 skips and no failures. Published image checks verified that both contained the same released grails-okapi 9.2.0 bytes.

Those are distinct delivery outcomes. Open Access 4.0.0 became a tagged release and the hosted pilot candidate. Agreements adopted the library in its development line and a verified snapshot image; its hosted rollout was still pending. The work gave us evidence across two real consumers without requiring an application-specific rewrite of their protected job logic.

Correcting the locking problem also gave us a meaningful candidate for something Foundry itself has to make routine: moving a running tenant from one coherent working set to another.

Turning corrected source into a working release

Publishing an individual module artifact does not identify everything that should run alongside it. Foundry’s reviewed release definition selects that working set. An immutable Foundry Release Manifest identifies the exact capability bills of materials, which bind backend artifacts, frontend artifacts, dependencies and transition requirements. The release machinery signs and publishes that content; publication makes it available for selection without upgrading tenants.

Loom holds the Workspace’s desired and ready state and plans the lifecycle needed to reconcile them. Its executor creates or verifies runtimes, performs tenant lifecycle calls and publishes routing. Release definition and operational execution meet here, but they remain separate responsibilities.

The Open Access Release Frontier page provides the component history behind this work: the Apache baseline and the newer K-Int module release. That source-release history is distinct from the Foundry manifest selecting the complete deployment in which we tested it.

For this pilot, the path included:

1. Publishing the qualified grails-okapi 9.2.0 and mod-oa 4.0.0 releases, and verifying the actual library and image artifacts.
2. Producing and verifying the signed, immutable Foundry 2026.2 Open Access 4 Pilot manifest.
3. Testing populated Open Access 3.0.0 → 4.0.0 migration locally, including an injected failure, successful retry, fresh-process readback and an isolated restore to the old version.
4. Importing verified release availability, selecting the pilot and checking Loom’s exact upgrade plan through the existing Workspace lifecycle APIs and foundryctl.
5. Creating the target runtime, putting the Workspace’s routes into maintenance and invoking the normal FOLIO POST /_/tenant migration with the unchanged tenant key and exact module_from and module_to versions.
6. Publishing routes, verifying actual application requests and federation coordination, then promoting that exact tested manifest to the Workspace’s ready release.

The pilot Workspace had no Console Service binding, so this exercised Loom’s lifecycle directly. It did not test the customer-facing Console upgrade journey. Promotion advanced this Workspace’s ready manifest; it did not move the wider Stable stream or upgrade other Workspaces.

Recovery needed its own proof. After migration, the old module process rejected the upgraded tenant; switching the container image back was insufficient. The qualified transition therefore used maintenance and roll-forward, with restore tested separately. Minting a coherent working set means recording these transition consequences as well as the component versions.

Our first live in-place tenant upgrade

The existing Open Access pilot moved from mod-oa 3.0.0 to 4.0.0, including Grails 6 to Grails 7 and the corrected federation-locking implementation. It retained its public tenant identity and URL. Its frontend and the other four capability versions remained unchanged.

This changed one significant backend component within an existing deployment. The normal lifecycle rechecked the complete selection and placed the Workspace into maintenance during cutover.

The application migration succeeded. Running the process for real then exposed two routing problems.

First, the installed executor was behind the qualified source. Its older routing compiler used a shared module-name cluster where distinct backend runtimes needed distinct clusters. The pilot could answer correctly while a neighbouring tenant was directed towards the wrong version.

After deploying the selected kernel release, a second problem appeared. Cached runtime information for a neighbouring Workspace lacked exact backend coordinates and still compiled to Kubernetes Service names that had been retired. The narrow correction resolved those missing coordinates from Loom while preserving that Workspace’s existing tenant, versions and routing state.

Both corrections went through the normal release and deployment process. We then requested fresh reconciliation under the corrected executor. The final run passed all 15 steps. No manual workload scaling was required.

Request checks through the internal Envoy listener returned HTTP 200 for Open Access. Inspection of the consumed routes confirmed the intended runtime destinations. Federation leader handover and subsequent heartbeats worked. The tenant retained its existing public identity and URL, and its exact ready manifest was promoted after those checks.

There are limits to that result. This was an almost-empty pilot tenant, with backup explicitly waived for its Open Access data. Populated migration and recovery had been tested locally; the live pilot was not a test of a large customer dataset. Tenant deletion or recreation was never required. Browser-level acceptance remained pending, as did wider rollout.

Nor did starting the new application family prove that the old family’s orphan had been reclaimed. That old state remained with a runtime still referenced by another tenant. Same-family crash recovery was established by the separate local tests.

What the upgrade exposed

The weakest part of this exercise was observability. Unnecessary 30-second waits accumulated between successful lifecycle steps. Progress reporting did not make it sufficiently clear what had completed, what was still happening or what the operator should expect next.

More seriously, the platform could report reconciliation complete without establishing that user requests reached the correct backend. A completed handler timeline, a ready pod and a working application request are different pieces of evidence. We had to inspect the routes consumed by Envoy and make requests ourselves before promoting the result.

These findings now give us concrete improvements to make in readiness checks, routing verification and lifecycle progress reporting. The upgrade path worked, and running it for real exposed the work needed to make future upgrades routine.

A sustainable release process has to answer more than whether a new image exists. Which exact versions belong together? Can we identify that set immutably? Can a running tenant move to it and recover when migration fails? Can we prove that the intended backend is serving requests? Can we promote the exact tested release, rather than reconstructing something approximately equivalent?

Maintenance that produces no new button

In Making the value beyond Apache visible, we described why analysis, architecture, testing, maintenance and release engineering need to be visible alongside features. This case makes that argument concrete.

Someone had to operate the modules, notice the anomalous state, preserve evidence, reproduce the failure and trace it across application, framework, orchestration and database boundaries. Someone then had to repair the common library, upgrade its consumers, qualify the artifacts, assemble the release, migrate a tenant and repair the deployment machinery exposed by that exercise.

Observe → diagnose → repair shared infrastructure → upgrade consumers → qualify artifacts → mint a coherent release → reconcile a live tenant → learn from its operation.

Measuring that work only through feature tickets or lines of application code would miss most of its value. These are the foundations that keep the visible functionality usable.

What stays open?

This incident also helps articulate a principle behind our Sustainable Release Lifecycle:

Operational knowledge stays open; independently funded engineering follows the Sustainable Release Lifecycle.

Operators of the current Apache software should continue to have the information they need to recognise this failure and operate around it safely. Its existence, diagnosis and recovery guidance were already public. Identifying the root cause should add to that understanding, not be deliberately obscured to make our implementation commercially valuable.

We should not monetise ignorance of a problem. We do need a sustainable way to fund the engineering required to solve problems.

K-Int funded the investigation, common-infrastructure changes, consumer upgrades, qualification and verification at its own risk. The resulting implementation follows our normal lifecycle: initially available through the FSL development stream, usable immediately under its terms, with organisations or the community able to fund accelerated Apache publication. Otherwise, qualifying work follows the normal FSL → Apache transition described in our sustainability proposal.

We do not want operators to have to pay us to learn that their existing Apache software can fail or how to recover it. That does not require every new piece of independently funded engineering to be donated immediately. The Release Frontier makes that development and its route towards Apache publication open to inspection.

Security vulnerabilities require appropriate responsible or coordinated disclosure. They cannot simply be assumed to follow the same publication rules as this operational defect.

Reconnecting maintenance and operation

Operating the modules ourselves improved our ability to maintain them. We could follow an unhealthy startup through restart behaviour, logs, transactions, locks and timing until it became an actionable software defect. A remote report that “the module won’t start” rarely provides all of that evidence.

Taking operational responsibility also improved our understanding of the delivery system. Moving the correction into a live tenant forced release definition, runtime creation, migration, routing, reconciliation and promotion to work together. Their rough edges became observable engineering tasks.

The old arrangement could leave us responsible for the code without necessarily giving us responsibility for, or visibility into, the running system. Operating the software ourselves is beginning to reconnect those things.

Sustainable stewardship has to span that whole path: source code, shared libraries, consuming modules, qualified releases, coherent working sets, live tenant migration and runtime observation. Keeping software useful requires maintaining the means by which it reaches people, and learning from what happens once it does.