Zero-Downtime Updates: How Ops Hot-Swaps Modules While You're Running

The Cost of Downtime

Every time a traditional ERP requires a maintenance window, the business stops. Payments can't be processed. Orders can't be placed. Inventory can't be received. For a restaurant or retail operation, even a 15-minute window during peak hours can cost thousands of dollars in lost sales.

And for what? To apply a configuration change. To update a module version. To add a new feature that one department requested. The cost-benefit doesn't add up — but it's been accepted as "just how enterprise software works."

Ops rejects this assumption entirely.

How Hot-Swap Works

The key insight behind hot-swap is module isolation. When each addon runs in its own WASM sandbox with its own memory space, the core system can replace a sandbox without affecting anything else.

The process:

• New binary is downloaded: The new version of an addon is downloaded in the background while the current version continues serving requests.

• Migration dry-run: The new addon's database migration scripts are executed against a transactional savepoint. If any migration fails, the savepoint is rolled back and the new version is rejected — the old version keeps running.

• Migration commit: If the dry-run succeeds, the migrations are committed. The database schema is now updated, but the new addon hasn't started serving requests yet.

• Traffic cutover: The router begins sending new requests to the new WASM sandbox. In-flight requests on the old sandbox are allowed to complete.

• Old sandbox drain: Once all in-flight requests on the old sandbox have completed (typically within seconds), the old sandbox is terminated.

The entire process takes 15–45 seconds. During that window, users experience zero interruption.

Schema Migrations: The Hard Part

The trickiest part of zero-downtime updates is database schema migrations. If the new code expects a column that doesn't exist yet, it crashes. If the new migration drops a column that the old code still reads, the old code crashes.

Ops handles this with a two-phase migration model:

Phase 1 (Expand): Add new columns, tables, and indexes. Never drop or rename anything. The old code can still run against the expanded schema.

Phase 2 (Contract): Once the new code is fully deployed and the old sandbox has drained, run the contract migration — dropping deprecated columns, renaming ambiguous fields.

Addon developers declare which migrations are expand vs. contract in the manifest, and Ops orchestrates the two phases automatically across the cutover.

Rollback: The Safety Net

What if something goes wrong after the new version is deployed? Ops maintains the previous addon version for a configurable grace period. If error rates spike or a health check fails, an automatic rollback is triggered:

• Traffic is re-routed to the previous sandbox (restarted from the cached binary).
• The contract migration (if it ran) is reverted.
• The tenant admin is notified.

Rollback completes in under 30 seconds. The database is consistent. No data is lost.

The Operational Impact

For the businesses running on Ops, zero-downtime deployment changes the relationship with software:

• Updates happen continuously, not in quarterly big-bang releases.
• New features ship when they're ready, not when a maintenance window is available.
• Bug fixes reach production in minutes, not the next maintenance cycle.
• The fear of updates disappears, because every update is reversible.

This is what modern software operations look like. Ops brings this standard to ERP.