The last few days of Foundry work began with a fairly modest question: can a self-hosted installation provide its own S3 storage, and can modules receive the configuration they need when we provision a tenant?
Following that question through took us into release procedures, routing capacity, rebuilding test environments and cleaning up after ourselves. There are few eye-catching features in that list. There is a great deal that matters to anyone who will have to install, run, test or maintain the system.
We now have a working Snapshot environment with five modules, automatic storage bootstrap and a verified nightly rebuild. It is a useful milestone in Foundry’s transition from demonstrating interesting capabilities to doing the practical work needed to operate them.
The original question: storage we can run ourselves
The storage work was about making self-hosting practical. An institution should be able to provide S3-compatible storage on its own infrastructure instead of depending on an external object-storage provider. Our installation now uses Garage for that purpose.
The boundary was equally important. Foundry configures modules through the interfaces they already expose: container startup parameters and supported tenant configuration APIs. It does not redesign their storage implementations or require them to adopt a shared internal credential mechanism.
Agreements, Licenses and ILL now receive their default storage configuration through normal provisioning. That is a bootstrap default, not an attempt to take ownership of all subsequent module settings. For example, ILL retains its own interface through which a tenant can choose its own bucket credentials.
Provisioning is only half the job. A disposable environment also needs a reliable way to remove what it created. We completed the corresponding lifecycle work: purge the module’s tenant through its existing API, remove the exact Foundry-owned storage allocation, verify its absence, then bootstrap the next generation. Shared resources and externally supplied storage retain their ownership boundaries. A failed cleanup stays visible as a failed operation rather than being reported as a successful reset.
Why that became release work
Once the storage features were ready, the next question was how to make them available for testing without inventing a special release process just for S3.
That led to a consolidated Foundry release procedure. A Foundry Release Manifest is the system-level bill of materials: the exact, coherent set of component selections we intend to run. Installation selections also record the kernel, executor and substrate inputs needed to operate that set.
The intended flow is straightforward. Module owners publish their modules. Foundry consumes those published releases, assembles and qualifies a coherent set, and makes it available to Snapshot. When we want a formal release, we take the exact set from a successful Snapshot run as a fixed candidate and curate it from there. Snapshot can continue moving while that candidate stays fixed.
We have settled the procedure and delivered supporting selection and qualification tooling. The complete automatic publication-to-Snapshot chain is still unfinished. Today, the nightly rebuild resolves the latest coherent definition already imported into Loom, Foundry’s control plane. Automatically producing and importing those new definitions remains separate work. A module release does not yet travel that whole path without intervention.
The routing limit we met along the way
Trying to create Snapshot through the normal Console workflow exposed a genuine capacity problem: the old routing configuration had grown beyond its ConfigMap size limit, even with a small installation.
Our intended scale is hundreds of tenants and more than fifty modules. Continuing to expand one large configuration document was not a satisfactory answer.
The routing work now publishes incremental configuration to Envoy using Delta xDS, within the existing executor component. Envoy continues to route application requests; Java services maintain and deliver its configuration. The hosted migration is complete, the old routing resources have been retired, and the original Snapshot service was recovered rather than replaced with another test installation.
We qualified configuration delivery for 500 tenants × 50 modules, alongside smaller real-runtime migration and recovery tests. That is evidence about routing configuration capacity and delivery, not a claim that we have load-tested a complete production application estate at that scale.
What Snapshot does now
Snapshot runs Agreements, ILL, Licenses, Open Access and Serials Management. Its public workspace address stays stable while its disposable tenant state is rebuilt every night at 02:00 Europe/London.
Here, “Snapshot” means a continually refreshed test workspace. It does not mean a database backup. Each run resolves one coherent release definition, fixes that selection for the whole run, clears the old tenant state and recreates the configured baseline. It still rebuilds when the selected release has not changed.
Operators manage this through the existing Service in KI Console. The State Management page shows the policy, schedule, next occurrence and recorded run results.
We exercised both a manual reconstruction and a real scheduled occurrence. The manual run completed thirty steps; the scheduled run completed thirty-six, including removal of the storage allocations created by the first run.
The checks went beyond a successful job status. Agreements, Licenses and ILL each uploaded and downloaded small and 6 MiB files, with checksums also verified directly against Garage. The scheduled reset removed the old file records, buckets, keys and credential files. Fresh browser access and module requests worked afterwards. Neighbouring workspaces continued to respond, and shared backend pods were not restarted by the second reconstruction.
What is closed, and what remains
For readers following the backlog, the current position is:
| Work | Position |
|---|---|
| 000092 — Tenant-owned binary storage | Complete and accepted: provisioning, configuration and hosted S3 verification. |
| 000103 — Owned S3 destructive lifecycle | Complete: repeated reconstruction, exact cleanup and fresh allocations verified locally and on the hosted installation. |
| 000088 — Scalable routing | Complete: implementation, capacity qualification, hosted migration and legacy retirement. |
| 000107 — Consolidated release workflow | Nightly Snapshot milestone complete. Automatic release production/import and the full candidate-to-promotion workflow remain open. |
| 000091 — State provisioning and nightly Snapshot | Nightly operation is delivered. The broader lifecycle record still needs its remaining qualification and closure review reconciled with that evidence. |
There is further operational work around health reporting, administrative alarms, retirement of unused runtimes, browser cache coherence and consistent management screens. These retain their own backlog items. They do not reopen the storage and routing work that has been completed.
Availability: running, but not generally open yet
Snapshot is running at snapshot.foundry.thelibraryplatform.com. It is not yet generally open to the public. We still have work to do on permissions and the rules for automatically approving new access requests before opening it more widely. Having a reachable address or being able to authenticate does not by itself grant workspace access.
For those with approved access, it is a development and testing environment. Its data is disposable and is reset nightly; it is not a place to keep work that needs to survive. We are not announcing unrestricted self-service access with this update.
The next release-workflow milestone is to connect owner-published releases to coherent assembly, qualification and import, so that new releases reach Snapshot through the normal process. From there, we can complete the proof of cutting and promoting a fixed release while Snapshot continues to move forward.
This work is less immediately visible than a new application screen. It gives us a repeatable environment in which to test those screens, a record of exactly what is running, and a way to rebuild it without leaving yesterday’s resources behind. Those are practical foundations for people to depend on Foundry beyond a demonstration.