Skip to main content

Docker HA/DR Implementation Plan

Status: proposed design, not functionality delivered by the current installer. Reviewed against installer, core and updater source on 2026-10-06. See Deployment Topology for the current single-host deployment.

For component owners, dependencies, implementation tasks and acceptance gates, see the DR Implementation Work Breakdown.

Use two Docker hosts in separate failure domains: an active primary and a passive secondary. Only the active host may accept writes, consume queues, execute workflows, poll integrations or issue telephony commands. Each host retains its own uniconnect_internal bridge network; these networks do not span hosts.

Start with operator-approved failover. Automatic failover is a later milestone requiring independent quorum/leases and reliable fencing, not just two servers pinging each other.

The diagram shows two upload alternatives, not two simultaneous sources of truth. Prefer shared, independently resilient object storage when feasible; otherwise use local-file replication below. The traffic endpoint must itself be redundant or managed externally. Management access remains private/authenticated.

Recovery Objectives​

Agree targets before implementation. Initial planning targets are 15-30 minutes RTO (time to restore service) and up to 5 minutes RPO (potential acknowledged-data loss) for asynchronous DR. These are acceptance targets, not guarantees; measure database, Redis and file lag independently under load.

If every acknowledged write must survive loss of the primary, asynchronous replication is insufficient. Database synchronous commits and durable file replication must both be part of the acknowledgement path. This adds latency and can stop writes when the secondary is unavailable. Redis and external side effects require separate durability/reconciliation guarantees.

2. Current Implementation and Gaps​

AreaVerified current behaviorRequired HA/DR change
InstallerCreates seven single-instance containers, including managerGenerate active/passive configurations and retain a reproducible deployment manifest
PostgreSQLPostgreSQL 16 with a host data mount; no replication configuredStandby bootstrap, secure replication, WAL archive, backups and promotion procedures
UploadsAPI mounts host Data at /assets; file management supports local and S3 storageInventory actual paths/metadata; replicate local files or use resilient shared object storage
RedisRedis Stack, AOF and host /data mountReplica authentication, persistence and controlled promotion; retain matching modules
API startupRuns migrate-seed-and-db-seed when /app/migration_done is absentRemove migration/seed side effects from normal API startup on both hosts
UpdaterResolves release metadata and promotes temporary API/Cube/Web containers locallyPair-wide coordination, exact release targets, passive validation and durable deployment status
Version recordUpdater writes release version to PostgreSQL after local promotionReport actual image digests/config per host; DB version alone cannot prove parity
CubeStoreInstaller sets /cube/data but declares no persistent mountDecide rebuild versus durable storage and measure rebuild delay
Event busInstaller does not deploy NATS; checked core config uses transporter: nullDo not assume a cross-host NATS cluster is a current dependency

Installer and updater API launch commands both perform startup migration/seed work. The script includes Prisma deployment migrations, database seeding and system scripts. A container-local marker is neither a pair-wide lock nor a durable migration record.

3. Data and File Replication​

PostgreSQL​

  1. Keep both hosts on the same PostgreSQL major version and tested pinned images. Treat major upgrades as a separate procedure.
  2. Bootstrap the secondary with a base backup and configure physical streaming replication over private authenticated/TLS connections. Never rsync a live PostgreSQL data directory.
  3. Configure WAL retention and replication slots where appropriate. Alert on retained WAL disk usage so an offline standby cannot fill the primary disk indefinitely.
  4. Keep the secondary read-only. DDL, Prisma migration history, application rows and DB release records replicate through WAL; never migrate it independently.
  5. Archive WAL and base backups with pgBackRest or WAL-G to independent storage. Test point-in-time restores, not only backup creation.
  6. Monitor receive/replay positions, replication state and storage. Report byte lag/replay position as well as time; an idle DB can make time-based lag misleading.

Use asynchronous replication for initial cross-site DR. For planned switchover, stop all writers and wait for replay of the final primary WAL position. For unplanned failover, record last known replication positions and any possible data-loss window before promotion.

Upload Storage Options​

Inventory every file written by API, workers and integrations: attachments, thumbnails, imports, exports, recordings stored by UniConnect, generated Cube schemas and non-regenerable artifacts. The Data mount does not prove every writer uses /assets; verify resolved paths in deployed images.

Preferred: shared object storage. Both hosts use the same stable endpoint/object keys, with versioning, access control and independent backups. Verify the endpoint survives the selected site failure. For separate regional buckets, configure replication and a stable download resolution strategy; stored absolute URLs may otherwise point at the failed site.

S3 support exists, but the checked file service includes a background upload path that returns a URL before completion. Audit all paths and acknowledge completed uploads only after durable storage. Cross-region object replication is asynchronous unless a stronger application protocol is implemented.

Local alternative: one-way lsyncd/rsync over private SSH from active to passive, plus periodic full reconciliation. Complete the initial copy before declaring the secondary eligible. Preserve paths, permissions, checksums and deletion semantics; transfer finalized files atomically, never exposing partial uploads.

  • Write to a temporary name, close/durably finalize, then publish metadata. Track pending/ready replication state where asynchronous copies are used.
  • Maintain a journal of finalized files and tombstones; do not blindly use rsync --delete against the only recovery copy.
  • Monitor pending transfer age, failures, reconciliation timestamps and checksums, not just daemon uptime.
  • Planned switchover drains pending uploads and checks all required DB-referenced files on the secondary before routing traffic.
  • Unplanned failover validates references and reports missing files: a current DB replica can still reference an upload that has not replicated.
  • After promotion, stop the old copy direction and explicitly reconfigure the new active as source. Never enable uncontrolled bidirectional sync.

Local asynchronous sync cannot guarantee zero upload loss. For that requirement, acknowledgement must wait for durable secondary storage, with writes blocked/rejected when unavailable. File publication and DB commits are not inherently atomic; use durable pending state and reconciliation for missing references and orphaned objects.

Redis and Queue State​

Use native Redis replication, not live AOF/RDB file copying. Pin identical Redis Stack versions/modules, restrict replication to private authenticated channels and configure persistence on the secondary.

Classify keys: cache may be rebuilt; BullMQ jobs, locks, sessions and integration state are not necessarily disposable. Redis replication is asynchronous and not transactionally aligned with PostgreSQL. Queue writes can be lost and in-flight work can replay.

Persist critical job intent in PostgreSQL using an outbox/reconciliation mechanism and stable idempotency keys. Reconcile queues after promotion. Email/SMS sends, workflows and call commands must not blindly replay: an external effect may already have occurred before the failure.

Cube, Configuration and External Dependencies​

Treat CubeStore as rebuildable only after proving PostgreSQL and model definitions can reconstruct it. Document report unavailability during rebuild; reporting RTO can differ from transactional RTO. Do not assume a live directory copy is a consistent CubeStore backup. Version generated models and verify freshness after promotion; disable refresh workers on passive hosts.

Unify installer/updater differences in Cube mode, mounts, ports and environment as part of reproducible manifests. Distribute config through an encrypted, versioned bundle. Share logical settings, stable public URLs and required JWT/encryption keys; keep role, IPs and replication credentials host-specific.

Do not blindly mirror /etc/uniconnect or the entire /var/lib/uniconnect tree. DB directories, logs, manager recovery budgets and machine identity have different ownership. Both hosts need valid TLS, license/release service access and connectivity to 3CX. API mounts machine identity files: approve standby licensing with the licensing owner rather than clone primary identity. PBX availability and recordings stored on 3CX need a separate recovery plan.

4. Versions and Migrations​

Pair-wide Release Contract​

Maintain one immutable desired release manifest containing:

  • Release ID and image digests for API, Web, Cube, Manager, PostgreSQL, Redis and CubeStore; eliminate mutable latest tags.
  • Config revision, migration set/checksum and supported schema compatibility range.
  • Host roles/identities, promotion epoch, actual running digests, deployment phase and promotion eligibility.
  • Rollback image digests and explicit compatibility with the resulting database schema/data.

Standby eligibility requires approved image/config parity plus DB, Redis and file readiness. Both hosts must report actual Docker digests; replicated DB release metadata cannot prove local image parity.

Use one operator-controlled coordinator initially. Local updater leases do not coordinate the pair. Persist pair-wide intent/status outside the active host and block independent local updates. Both hosts must deploy the same explicit target, not independently resolve their own "next version".

Upgrade Procedure​

  1. Acquire a pair deployment lock and record exact targets. Disable promotion during incompatible/partial phases; verify backups and retained rollback images.
  2. Pull/verify images on both hosts. Test candidates with isolated restored data in staging. Never start temporary full APIs against production if they can seed, schedule or send external requests.
  3. Execute an audited migration job once on the current DB writer, not on each API boot. Use prisma migrate deploy, not prisma migrate dev. Seed/system scripts require separate approval and idempotency.
  4. Apply expand-only schema changes while old code serves traffic. Record migration success and wait for standby WAL replay at the required position.
  5. Prepare the new standby release with migrations, workers, refresh and external consumers disabled. Until passive boot is proven, keep its application stopped and validate in isolated staging.
  6. Update the active API/Cube/Web and verify login, CRM writes, uploads/downloads, queues, reports and 3CX events. Roll back images only when old code remains compatible with migrated data.
  7. Check actual digests/config and replication readiness on both hosts; mark complete and re-enable promotion. Manager upgrades require an external controller/service mechanism, not Resource Guard self-restart.
  8. Apply destructive contract migrations in a later approved release after both hosts and rollback paths no longer need the old schema. Incompatible upgrades require a maintenance window and explicit temporary loss of failover readiness.

Do not automatically reverse migrations or restore an old DB to roll back images: restores can discard accepted writes. Prefer forward fixes for irreversible changes. Dynamic CRM schema changes replicate as DB writes and also need compatible generated Cube models on the secondary.

5. Failover Safety​

Implement a durable role and promotion epoch checked at boot and before side-effecting work. New or uncertain hosts default to passive, including after Docker restart or host reboot. Blocking HTTP alone does not block background jobs.

  • Stop/gate queue consumers, workflows, campaigns, email pollers, 3CX connections, writers and Cube refresh on passive hosts.
  • Manager can monitor but cannot independently upgrade, promote or auto-restart applications into active mode.
  • Active readiness must check role/epoch, writable DB and schema compatibility. Container health or pg_isready alone is not sufficient.
  • Restrict PostgreSQL 5432 to private/authorized sources despite the host mapping. Docker socket access is privileged; restrict it to trusted management paths.
  • Switch API, WebSocket, report and webhook endpoints together; sockets reconnect and resynchronize. DNS changes are subject to cached answers.
  • Monitor externally: the local manager cannot recover its own failed host.

Later automation can use a proven control plane such as Patroni with a three-member distributed configuration store for PostgreSQL, integrated with application/file fencing. PostgreSQL leadership alone does not fence workers. A third witness must be independent; automatic two-host promotion based solely on timeout is unsafe.

6. Recovery Runbooks​

Planned Switchover​

  1. Validate release parity, backups, standby licensing/connectivity, replication and promotion eligibility; acquire the pair lock.
  2. Enter maintenance, drain requests/uploads and stop all application/background writers and externally triggered work on the old active.
  3. Record final WAL position and wait for replay. Reconcile Redis/job intent, finalize file sync and check file references/checksums.
  4. Fence old active writes and external effects. Stop replication paths that might resume in the old direction.
  5. Promote PostgreSQL/Redis under one recorded epoch. Enable writable storage and active application role only after dependencies pass checks.
  6. Verify privately, switch traffic and test login, CRM write, upload/download, queue execution, 3CX state, socket reconnect and reports.
  7. Record measured interruption/data consistency; keep old active fenced until rebuilt as passive.

Unplanned Failover​

  1. Confirm failure through independent monitoring and operator approval; distinguish host failure from partition.
  2. Fence the old active first, using cloud/hypervisor power-off or equivalent enforceable isolation. DNS is not fencing: cached clients and background jobs can continue. Without proven fencing, do not automatically promote.
  3. Check standby eligibility and last known WAL/file/Redis lag; approve any loss window. If unsafe, invoke restore-to-clean-host recovery instead of promotion.
  4. Promote data services, validate available files/references and reconcile in-flight jobs. Flag uncertain external effects for review rather than blind replay.
  5. Assign the new epoch, start active applications and run private checks. Switch traffic only after verification.
  6. Keep the returned old host isolated; Docker restart policies must not resurrect stale writers.

Failback​

Failback is a new planned switchover, not a DNS reversal. Rebuild old primary from the current writer with a fresh base backup or pg_rewind only when prerequisites hold. Re-seed Redis and rebuild/reconcile files from new active; never automatically merge diverged databases or stale file trees.

Deploy approved images/config, prove replication integrity, then follow planned switchover. Do not promote an old data directory merely because the host is reachable again.

7. Implementation Work Packages​

OrderOwnerDeliverable and gate
1Product and operationsApproved RTO/RPO, failure domains, file storage option, licensing and fence mechanism
2Core developersWriter/file inventory, passive role gates, migration-free startup and idempotent queue reconciliation
3InfrastructureDB/Redis replicas, private channels, file/object durability, independent backups and measured lag alerts
4Installer/updater developersPinned manifests, explicit release target, pair lock/status, parity and eligibility checks
5Operations and QATested switchover, unplanned failover, stale-primary reboot and failback
6Infrastructure/core developersOptional quorum/fencing automation after manual recovery passes

Acceptance Tests​

  • Host failure/partition: one writable DB and one active worker set; an unfenced partition cannot trigger automatic promotion.
  • Upload interrupted by failure: no partial file reported complete; missing files detected and loss bounded by approved RPO.
  • Queue job interrupted: duplicate external effects prevented or flagged; lost intents recovered from durable application state.
  • Migration/update failure: incompatible standby promotion blocked; approved recovery does not seed twice.
  • Version drift: promotion blocked even if replicated DB version appears correct.
  • Standby/old-primary reboot: passive defaults survive restart policies.
  • Independent restore: chosen DB recovery point plus matching files/config/secrets work without primary disks.
  • Loaded switchover/failback: measure transactional/reporting RTO, DB/queue/file loss and 3CX reconnection.

Replication is not backup: deletes/corruption can replicate. Use versioned/immutable off-host backups with approved retention and tested restores. DB/upload backups need a consistent checkpoint or retained file versions so references resolve after point-in-time recovery.

8. Licensing, Optional DR and Partner Portal​

Current License Identity Behavior​

Installer generateInstanceId() and core utils/generateInstanceId.js both compute SHA-256 from /etc/machine-id, /sys/class/dmi/id/product_uuid and INSTANCE_UUID in the host system.env. Moving to different hardware or regenerating that UUID changes the resulting ID. Copying only INSTANCE_UUID does not preserve the ID on another host.

Branch verification: the license server's dev branch was checked on 2026-10-06. It mounts the installer's v2 routes, so the earlier missing-v2-handler finding no longer applies to this branch. This confirms source behavior, not which image is running in production.

Existing v2 APIObserved behaviorDR implication
/api/v2/installer/metaReturns initial license metadata/tokensCan support provisioning, but does not enroll a passive member
/api/v2/installer/init and /api/v2/instance/initShare activation service; require instance ID/origin and reject an existing Manager-side instance for non-DEV licensesSecondary must use a new join/enrollment flow, not ordinary activation
/api/v2/installer/bindAuthenticated domain binding through domain controllerKeep one stable public domain; do not bind both hosts independently
/api/v2/instance/authRequires license key and instance ID; shared token service rejects a differing bound IDDR's distinct hardware ID cannot authenticate as primary
/api/v2/instance/renewShared token service validates a supplied ID; missing ID currently logs a warning and skips binding validationLegacy compatibility is not standby authorization; new member protocol must require identity
/api/v2/instance/syncLicense replacement checks the old instance ID and the new license's binding/stateLicense replacement is not a server-role switchover mechanism

The license model still stores one instanceId. validateAndBindInstanceId() binds an empty record on first use, then rejects a different ID. Reusing replicated data/tokens must not be treated as authorization for a second server.

Activation writes the incoming instance ID, activation date, calculated expiry and domain binding state; it requires a successful Manager license update before updating the local license cache. Saving instance telemetry afterwards is best-effort. Therefore a DR enrollment endpoint must not reuse activation's expiry/binding side effects, and successful activation alone is not proof of a fully enrolled/ready pair.

Manager synchronization already saves or updates telemetry by matching instance ID, but getCustomerInstace() selects the first instance returned for a license. Existing per-license telemetry updates and lookups must become deployment/member-aware. The external CRM-backed Manager service here is distinct from the local Docker uniconnect-manager updater agent; the planned partner portal should integrate or migrate those records without conflating the two.

The branch provides per-license Redis operation locks and configurable grace/refresh periods. Reuse these foundations where appropriate, but neither currently expresses DR membership, active-role authority or fencing. IP-range validation currently targets DEV/QA license types, with Manager-specific handling; register/authorize both DR members explicitly rather than treating source-IP checks as role control.

Do not solve DR by copying host identity files, disabling identity validation or rebinding the license on every failover. Those approaches can invalidate primary renewal, permit unintended duplicate production use, or depend on licensing availability during an outage.

Proposed Identity and Entitlement Model​

Separate the customer deployment from its physical hosts:

Identifier or fieldMeaningLifetime
licenseIdCommercial entitlement: modules, capacity, expiry and DR allowanceExisting license lifecycle
deploymentIdStable logical customer deployment, public domain and data lineageSurvives promotion and approved host replacement
memberIdServer-generated enrollment identity for one authorized hostSurvives container upgrades; replacement creates a new member
hostFingerprintCurrent hardware/install-derived ID from existing generation codeMay change on rebuild; requires explicit re-enrollment/rebind
activeMemberIdMember currently authorized to run productionChanges through a controlled promotion
promotionEpochMonotonically increasing authority generationChanges on accepted role transfer; never rolled back
drEnabled / drStateRequested optional mode and actual readinessDisabled, enrolling, syncing, ready or degraded

These fields are proposed contracts, not existing env variables or APIs. Store authoritative membership/entitlements in the license control plane and transition state in a durable recovery controller. Link them through deploymentId and epoch; do not create two independently writable sources of active-role truth.

Allow one active production member. When DR is enabled and entitled, permit one passive member initially. Seats/modules belong to the logical deployment, not two independent installations; count active usage once and prevent the passive host's zero/stale counts from overwriting production usage. Independent active-active use is outside this plan and requires a different commercial/concurrency policy.

Both hosts retain their own local enrollment credentials and installation UUID. Do not overwrite these by copying primary system.env or by restoring replicated license/token rows. Shared product configuration can replicate, but host-bound authentication must be stored outside replicated application state or explicitly partitioned by member. On promotion, the new active obtains credentials for its own member, not the former primary's identity.

Maintain two distinct signed authorizations:

  • Entitlement: deployment membership, modules, capacity, DR allowance, expiry and credential audience. A passive member can authenticate for monitoring/updates without permission to execute production work.
  • Production authority: active member, promotion epoch, operation ID and bounded lease/expiry. Core/Manager verify member identity, signature and role before enabling side effects. A fingerprint alone is not authentication: use per-member key pairs or equivalent protected credentials.

Revoking a DB record does not instantly invalidate cached/offline signed tokens. Expiry/renewal policy plus enforceable fencing controls stale authority. Do not claim the license server alone prevents split brain.

Three Supported Installation Paths​

ScenarioInstaller and portal behaviorCompletion gate
Install without DRDefault standalone; activate one member normally, no secondary/replication requirementOne licensed working server, backups configured
Choose DR at installationCreate primary deployment, verify DR entitlement, issue short-lived secondary enrollment invitation; run secondary installer in join modePrimary functional; DR only marked ready after enrollment, version parity, replication and fence tests
Add DR laterPortal action on the existing deployment issues invitation; secondary installer joins without resetting domain/license activationSeed from existing writer, preserve release/data lineage, validate readiness without reinstalling primary

Choosing DR is not proof that it is ready. Enrollment failure leaves primary usable and shows DR not ready; rollback only the failed member enrollment, not the active license. Report backup-only and degraded states distinctly from a promotable standby.

Secondary join must not call ordinary first-install activation, initialize an empty production DB, run seeds, create a second public DNS identity or consume another independent production entitlement. Install the approved pair release and seed DB/files using the replication procedure. Keep the secondary passive throughout provisioning.

When DR is disabled later, require that the retained host is the confirmed active member. Revoke/detach passive membership, stop replication safely and apply a documented retention policy; never automatically wipe the detached host's data. If the originally designated secondary is now active, retain it as standalone rather than silently switching back.

Replacing a failed host is a separate audited enrollment/replacement workflow. Preserve deploymentId, register a new fingerprint/member and revoke the old member only through approved transition rules. Restored or cloned hosts default to passive until membership is reconciled.

Where Switching Happens​

Partner portal is the management UI; a recovery controller performs the operation. Place the controller outside both application hosts, with durable state, RBAC, audit and a queue. License service approves membership/entitlement; it must not be overloaded with Docker, file replication or cloud power-control responsibilities.

Managers execute scoped local commands received through authenticated outbound polling or a secure agent channel. Do not expose Docker sockets, unauthenticated manager ports or host secrets to the browser. Managers need least-privilege command authorization, nonce/expiry checks, member binding and replay protection; independent fencing must work even when the primary agent is unreachable.

Portal actions:

  • Enable/Add DR, enroll/replace/remove a member and view readiness/version/replication status.
  • Planned switchover to the current standby; emergency failover with explicit loss acknowledgement; failback only after re-seeding.
  • Coordinate release upgrades and show migration status without letting either host independently resolve a new release.
  • View immutable operation audit including requester, approval, fencing evidence, epoch, lag, target versions and smoke-test results.

Use tenant/partner-scoped permissions, MFA and explicit confirmation for disruptive operations; allow a configurable second approver for emergency promotion. A portal dashboard cannot treat stale telemetry as current readiness.

Promotion Transaction and Outage Policy​

The UI submits a target member, expected epoch and idempotency key. Controller acquires the pair operation lock and persists phases: requested, preflight, quiescing (planned only), fenced, promoting, verifying, routing, completed or needs-attention.

  1. Validate tenancy/permissions, membership, DR entitlement, expected epoch, release/schema parity, data readiness and freshness. Coordinate locking with updates/enrollment; refuse concurrent operations.
  2. Drain/quiesce for planned switching; obtain independent fencing proof before either kind of promotion.
  3. Reserve the next epoch atomically through the authoritative role store and issue target-bound promotion authority. Promotion of DB/Redis, file-role reversal and application activation follow the runbooks above.
  4. Verify the target privately, then change traffic. Persist each step and observed outcome; retry using the same operation ID rather than issuing a second promotion.
  5. Publish the confirmed active member and complete the operation. If interrupted after fencing/promotion, remain blocked or needs-attention; do not reactivate old primary automatically.

Control-plane role updates, DB promotion and DNS changes cannot be one distributed ACID transaction. Use a durable, resumable state machine with explicit unknown-outcome handling, not an assumed all-or-nothing request. Agents verify current authority before resuming after reboot.

Define separately what happens if portal, controller or license service is unavailable:

  • A UI outage must not stop a healthy active server; provide a secured operations API/CLI using the same controller and audit workflow.
  • Single-server installations must not acquire an unintended permanent online dependency from the DR feature. Preserve an explicitly approved entitlement grace policy.
  • Cached entitlement grace is not a second active lease. Any DR offline grace must prevent stale writers through fencing and bounded authority rules; specify renewal timeouts and safety behavior.
  • Initially require reachable control plane for promotion. If business requires failover when it is down, implement a tested break-glass path with pre-provisioned authorization, independent fencing, epoch reconciliation and audit upload. Do not invent a local "force active" bypass.
  • The external role store/license service/controller must meet the recovery availability target and have backups. Loss or restoration of control state must not reset the promotion epoch or reauthorize a revoked member.

Required Changes by Component​

ComponentChanges required before DR is supported
Installer UI/backendDefault standalone option; primary/join modes; DR readiness distinct from install success; secure invitation enrollment; separate migration runner; retained pinned manifests; fingerprint/config separation
License service (existing v2 APIs)Extend with deployment/member records, DR entitlement, identity-required member tokens, enrollment/replacement/revocation, active-authority integration, member-aware Manager sync, usage deduplication, audit and scoped partner APIs
CoreEnrollment-aware validation/refresh, separation of replicated license data from local credentials, passive boot gates, epoch/authority checks and explicit offline policy
Updater/ManagerReport actual member/release/readiness; authenticated command channel; shared operation lock integration; role-aware update/restart behavior; authorized idempotent local recovery commands
Recovery controllerDurable state machine, authoritative epoch ownership, pair-wide locking, independent fencing, storage/replication checks, traffic-provider integration and resumable audit
Partner portalTenant-scoped license/deployment/member inventory, enrollment lifecycle, DR health, guarded switching/upgrading actions and audit views; no direct Docker control

For existing customers, migrate each legacy license binding into one deployment and active member without changing its activation, expiry, commercial limits or customer data. Version token/API schemas and support old standalone clients during rollout. Add passive membership only after entitlement/enrollment approval; never change a single instanceId field back and forth to implement DR.

Additional Acceptance Tests​

  • Standalone install/renew/upgrade remains operational with DR disabled; enabling DR later does not reset existing license or data.
  • Primary and secondary have distinct fingerprints/member credentials but one deployment entitlement; passive enrollment does not overwrite primary binding.
  • Replicated license records cannot impersonate another member; revoked/replaced identities cannot renew or rejoin silently.
  • Switchover keeps the stable public domain and commercial limits, while active authority changes exactly once under retries/concurrent requests.
  • License/control-plane outage, stale tokens, cloned disks and reboot after promotion follow documented grace/fencing rules without two active writers.
  • Partner tenant isolation, approval gates, replay protection and audit integrity hold for all management operations.

9. Source References and Decisions​

Repository-relative implementation locations (not wiki routes):

  • Installer: uc-enterprise-installer/backend/services/dockerConfigService.js and dockerService.js for container startup/deployment.
  • Updater: uc-enterprise-updater/internal/services/deployment_service.go and container_service.go for release resolution, temporary containers and promotion.
  • Core: uc-enterprise-core/package.json for migration scripts; services/fileUpload/file-management.service.js for local/S3 storage; moleculer.config.js for the in-process broker.
  • Identity: uc-enterprise-installer/backend/utils/helpers.js, backend/services/licenseService.js and uc-enterprise-core/utils/generateInstanceId.js.
  • Checked license-server behavior (dev branch): uc-enterprise-license-manager/src/models/License.ts, routes/api/index.ts, routes/installerRoutesV2.ts, routes/instanceRoutesV2.ts, controllers/TokenController.ts, services/token/TokenServiceImpl.ts, services/validate/LicenseValidateServiceImpl.ts, services/Manager/ManagerSyncService.ts and helper/ipValidationHelper.ts (paths after src/). Verify deployed image revision before applying the plan to production.

Before implementation, approve upload storage and acknowledgement guarantees, independent fencing, DR commercial allowance, member replacement policy, existing Manager-to-partner-portal record ownership and offline promotion/grace policy. These decisions determine achievable RPO, licensing continuity, partition behavior and exact Docker/network configuration.