Skip to main content

UniConnect DR Implementation Work Breakdown

Status: draft backlog for review, not an approved schedule or implemented functionality. Date: 2026-10-06. Design input: Docker HA/DR Plan. Runtime baseline: Deployment Topology. License analysis uses the license server's dev branch; confirm deployed revisions before execution.

This checklist covers the complete Docker-only active/passive DR feature: standalone operation, DR chosen during installation, DR added later, enrollment/licensing, configuration, data/files, releases, migrations, partner portal and recovery operations.

Scope and Delivery Rules​

  • First production release: operator-approved switchover/failover/failback, one active server and one passive member, fencing before promotion.
  • Standalone remains supported and is the installation default. No secondary or permanent DR-controller dependency is introduced for customers who do not select DR.
  • Primary/secondary are deployment roles, not immutable server identities. The original secondary can remain active or become standalone after promotion.
  • Automatic failover and offline emergency promotion are separately approved follow-on capabilities, not shortcuts required to complete the first release.
  • No Kubernetes, active-active database/file merging, copied hardware identities, unconditional migration-on-boot or direct portal-to-Docker access.
  • Existing module/repository patterns remain authoritative. Use Prisma for core data, existing installer services/UI, Go manager boundaries and existing license v2 services. No framework migration is part of this work.

Task IDs are stable issue references. Each checkbox is one implementable work package, not necessarily one PR. Owner is inherited from its phase unless noted. Existing code locations below are repository-relative implementation anchors, not wiki routes. Locations explicitly marked proposed do not exist yet and must be approved before implementation.

This is a planning artifact based on the existing design page, not a formal modernization-pipeline sign-off. Detailed API/data contracts, approved policy and estimates are outputs of Phase 1; dependent implementation stays blocked until they are accepted. No new architecture is silently approved by this checklist.

Requirement Catalog​

IDRequired outcome
REQ-001Standalone install/renew/update remains supported without DR
REQ-002Choose DR at install or add it later without resetting active license/data
REQ-003Stable deployment identity, separate authenticated members and safe replacement/detachment
REQ-004One commercial entitlement, active usage counting and legacy customer compatibility
REQ-005Validated, versioned shared/host-local DR configuration and secrets
REQ-006PostgreSQL replication, promotion and independent PITR recovery
REQ-007Upload/file durability, integrity, deletions and recovery-point alignment
REQ-008Redis/queue replication and reconciliation without blind external-effect replay
REQ-009Passive application gates, restart safety and one active authority
REQ-010Cube/model recovery and 3CX/integration/WebSocket continuity
REQ-011Exact pair-wide release versions, rollback compatibility and manager upgrade handling
REQ-012Once-per-writer migrations, controlled seeds and schema compatibility
REQ-013Secure outbound agents, external controller, durable operations and pair locks
REQ-014Fencing, promotion epochs and stable coordinated public traffic switching
REQ-015Partner-scoped license/deployment management, guarded actions and audit
REQ-016Fresh readiness/lag/backup metrics, alerts and notifications
REQ-017Planned switch, emergency failover, resumption, failback and host replacement
REQ-018Approved outage/grace behavior and optional offline emergency policy
REQ-019Restore/failure drills, backward compatibility, staged rollout and support handover
REQ-020Optional automated failover after independent quorum/fencing validation

Dependency and Milestone Map​

PhasePrimary ownerDepends onRelease gate
1. Decisions and contractsProduct, architect, security, operationsExisting design reviewApproved policy and contracts
2. Configuration/control foundationPlatform backend, infrastructurePhase 1Reproducible config and durable identity/authority
3. License serviceLicensing teamPhases 1-2Distinct member authorization, legacy compatibility
4. Core/data semanticsCore, Cube and web teamsPhases 1-2; Phase 3 for credential integrationPassive means no external effects or writes
5. Replication/backupInfrastructure, storagePhases 1-2; Phase 4 file protocolMeasured, recoverable DB/Redis/files
6. Manager/release integrationGo updater, release teamPhases 2-5Trusted agent and safe explicit-target updates
7. Installer journeysInstaller teamPhases 2-6All three install paths and safe DR removal
8. Partner portalPortal frontend/backendPhases 2-3; Phases 6/9 for real operationsTenant-scoped workflows and operational UI
9. Recovery orchestrationController, infrastructurePhases 3-6; Phase 8 for UI integrationResumable fenced promotion and failback
10. Validation and rolloutQA, security, operationsPhases 3-9Measured first-release acceptance
11. Optional automationPlatform, security, operationsPhase 10 and separate approvalQuorum-backed automatic recovery

Phases are workstreams, not a mandatory serial schedule. Core and infrastructure work can overlap after their contracts are approved; portal screens can use contract fixtures before recovery integration. Do not enable real actions until the named gates pass. Assign individual owners and estimates after Phase 1; no dates or person-day estimates are assumed here.

Phase 1: Decisions and Executable Contracts​

Plan item: 1.1. Owner: product/architect/security/operations. Requirements: REQ-001 through REQ-020. Input: HA/DR design and current deployment page.

  • T001 [Plan:1.1] Approve separate transactional/reporting RTO and DB/file/queue RPO in uc-enterprise-wiki/docs/02-architecture/high-availability-and-dr.md; define sync/async acknowledgement and partition behavior.
  • T002 [Plan:1.1] Select supported upload storage and migration path in the same design page: shared HA object storage versus local one-way sync; define deletions, integrity checks and restore-point rules.
  • T003 [Plan:1.1] Approve DR allowance, passive capacity, expiry/grace, entitlement revocation, host replacement and offline promotion policy in the design page; distinguish licensing grace from production authority.
  • T004 [Plan:1.1] Assign one authoritative role/epoch store, durable controller repository/stack and ownership of existing CRM-backed Manager records versus the planned portal; record the decision in the design page.
  • T005 [Plan:1.1] Approve independent fence provider, traffic provider, private connectivity, secret distribution and control-plane recovery topology in the design page; list provider permissions and failure evidence.
  • T006 [Plan:1.1] Specify deployment/member/config/readiness schemas and enrollment, authority, telemetry, operation and release API contracts under proposed uc-enterprise-wiki/staged-docs/deployment/dr-contracts.md, including versioning and error states.
  • T007 [Plan:1.1] Inventory every worker, startup write, integration consumer, file writer and credential store across uc-enterprise-core/services, utils and config; attach path-to-owner and data criticality tables to the contract artifact.
  • T008 [Plan:1.1] Convert this checklist into assigned issues with component owners, dependencies, estimates, first-release scope and acceptance evidence; resolve controller/portal code homes before tasks referencing proposed locations start.

Acceptance: policies and contracts are reviewed; first-release choices are explicit. An unresolved storage/fencing/authority choice blocks dependent coding, not standalone operation.

Phase 2: Configuration and Control Foundation​

Plan item: 2.1. Owner: platform backend/infrastructure. Requirements: REQ-003, REQ-005, REQ-009, REQ-013, REQ-014, REQ-016, REQ-018. Input: Phase 1 contracts.

  • T009 [Plan:2.1] Implement versioned DR config schema for disabled/enrolling/syncing/ready/degraded states, role, member IDs, private endpoints, thresholds and providers; shared contract home approved by T006.
  • T010 [Plan:2.1] Implement validated configuration generation in uc-enterprise-installer/backend/services/envService.js; separate deployment config from host identity, credential paths and role state.
  • T011 [Plan:2.1] Make installer/updater config application atomic, redacted, revisioned and reversible through envService.js and uc-enterprise-updater/internal/services/structure_service.go; reject conflicting host/role settings.
  • T012 [Plan:2.1] Add member-key provisioning, rotation, revocation and protected local storage through installer identity helpers and manager config; retain distinct INSTANCE_UUID and fingerprint on each host.
  • T013 [Plan:2.1] Implement durable deployment/member and compare-and-set epoch persistence in the controller/control store approved by T004; restore cannot reset/reuse an old authority epoch.
  • T014 [Plan:2.1] Define and implement shared readiness reporting: observed role, digest/config/schema revision, DB replay, Redis/file backlog, authority validity and sample age; stale/missing signals block eligibility.
  • T015 [Plan:2.1] Provision independently hosted control store/controller services, TLS, restricted networking, monitoring and backup/restore manifests in the infrastructure home approved by T005.
  • T016 [Plan:2.1] Implement secret-reference distribution, certificate renewal and key-version compatibility across both hosts without copying hardware IDs or primary enrollment credentials; document rotation/recovery ownership.

Acceptance: configuration renders deterministically for standalone/active/passive; identity/config revisions are separately inspectable; credentials are never exposed in manifests, browser payloads or audit logs.

Phase 3: License Service and Existing Manager Integration​

Plan item: 3.1. Owner: license-server team. Requirements: REQ-001, REQ-002, REQ-003, REQ-004, REQ-015, REQ-018. Input: identity/entitlement contracts and current v2 routes.

  • T017 [Plan:3.1] Extend uc-enterprise-license-manager/src/models/License.ts, mappers and repositories with deployment/DR allowance references; implement approved durable member persistence, not just extra Redis cache fields.
  • T018 [Plan:3.1] Migrate each existing single-instance binding into a deployment plus active member idempotently, preserving activation/expiry, commercial limits and existing standalone clients.
  • T019 [Plan:3.1] Add scoped one-time invitation issuance/redemption with expiry, partner/customer binding, replay protection and member credential proof through existing v2 route/controller patterns.
  • T020 [Plan:3.1] Separate first activation from passive join in src/services/validate/LicenseValidateServiceImpl.ts; secondary enrollment cannot overwrite active binding, extend expiry, reseed data or duplicate public-domain activation.
  • T021 [Plan:3.1] Add member-bound entitlement authentication/renewal in src/services/token/TokenServiceImpl.ts and controllers/TokenController.ts; require member identity for the new protocol and isolate legacy missing-ID compatibility.
  • T022 [Plan:3.1] Integrate signed active-authority verification/issuance with the one role-store owner from T004; include member, deployment, epoch, expiry/audience and operation ID without independent duplicate active-state ownership.
  • T023 [Plan:3.1] Make src/services/Manager/ManagerSyncService.ts lookups and telemetry updates member-aware; migrate/read legacy CRM records safely rather than select the first instance for a license.
  • T024 [Plan:3.1] Update usage accounting so only validated active-member reports affect deployment seats/modules/counts; passive zero/stale telemetry cannot overwrite production usage.
  • T025 [Plan:3.1] Implement approved member replace/revoke/detach and entitlement change APIs with partner-scoped authorization, conflict checks and audit; distinguish host replacement from license replacement /instance/sync.
  • T026 [Plan:3.1] Preserve/configure grace and refresh behavior for old standalone clients; define blocked/expired/revoked member behavior, revocation propagation and control-plane failure responses in v2 contracts.

Acceptance: distinct hosts authenticate under one entitlement; join never changes active expiry/domain; foreign/revoked members cannot renew. License denial/grace rules are explicit and never grant two production writers.

Phase 4: Core, Uploads, Workers and Client Behavior​

Plan item: 4.1. Owner: core/Cube/web teams. Requirements: REQ-007, REQ-008, REQ-009, REQ-010, REQ-012, REQ-018. Input: writer inventory and authority/file contracts.

  • T027 [Plan:4.1] Separate migrations/seeds/system scripts from normal boot in uc-enterprise-core/package.json and deployment launch contracts; new application boot is schema validation plus runtime startup only.
  • T028 [Plan:4.1] Add boot-safe role/authority validation at the core startup boundary and a shared guard for writes/background effects; DR uncertainty or stale epoch defaults to passive, standalone follows approved legacy policy.
  • T029 [Plan:4.1] Gate all inventoried queue consumers, cron jobs, workflows, campaigns, email pollers, synchronization and telephony connections under uc-enterprise-core/services; check role before each side effect, not merely at HTTP ingress.
  • T030 [Plan:4.1] Add quiesce/drain/resume lifecycle handling for requests, uploads and workers, with bounded timeouts and clear status; resumed work verifies current epoch.
  • T031 [Plan:4.1] Adapt services/license/handlers/license.handlers.js, license methods and utils/generateInstanceId.js to member enrollment/auth; separate local credentials from replicated license/token rows and preserve legacy standalone validation.
  • T032 [Plan:4.1] Correct upload acknowledgement/finalization in services/fileUpload/file-management.service.js: complete durable storage before success, atomically publish final files and retain recoverable pending state.
  • T033 [Plan:4.1] Implement upload metadata/key resolution, checksums, pending replication, tombstones and orphan/missing-file reconciliation in core file-management ownership; preserve private-file permissions and existing download links.
  • T034 [Plan:4.1] Implement migration of existing local/S3 upload references to the chosen storage protocol with resumable copy, count/checksum checks and rollback; include attachments/thumbnails and other writers identified by T007.
  • T035 [Plan:4.1] Add durable critical-job intent/outbox and idempotency/reconciliation at inventoried BullMQ/workflow/campaign boundaries; flag uncertain email/SMS/call outcomes for review rather than replay blindly.
  • T036 [Plan:4.1] Add role/schema/config-aware readiness distinct from liveness to core health handlers; expose sanitized DR status and maintenance errors through the existing API authorization pattern.
  • T037 [Plan:4.1] Version/rebuild generated schemas and gate Cube refresh in uc-enterprise-cube/cube.js, schema generation/change handlers and runtime settings; restore reports from approved source data without passive writes.
  • T038 [Plan:4.1] Update uc-enterprise-web/src/configs/axiosInstance.ts, websocket.ts and utils/cubeClient.ts for stable public endpoints, maintenance state, reconnect/state resync and safe retries; never automatically retry non-idempotent commands.

Acceptance: passive/rebooted/stale hosts produce no writes or external effects; upload success has defined durability; failover preserves authenticated access and handles socket/session/report recovery explicitly.

Phase 5: Replication, Storage, Backup and Host Security​

Plan item: 5.1. Owner: infrastructure/storage. Requirements: REQ-005, REQ-006, REQ-007, REQ-008, REQ-010, REQ-014, REQ-016, REQ-019. Input: approved provider/storage policy and file protocol.

  • T039 [Plan:5.1] Produce active/passive Docker/config profiles through uc-enterprise-installer/backend/services/dockerConfigService.js; pin images, restrict public DB exposure and separate private cross-host replication from local Docker bridges.
  • T040 [Plan:5.1] Implement secure PostgreSQL base-backup bootstrap/streaming replication with same-major checks, least-privilege replication credentials, slots/retention and replay position monitoring.
  • T041 [Plan:5.1] Implement idempotent PostgreSQL promotion and re-seeding/eligible pg_rewind procedures in the approved infrastructure/agent ownership; never rsync live database directories.
  • T042 [Plan:5.1] Configure native Redis replica/auth/persistence with identical Stack modules; report replica state/backlog and provide guarded promotion/re-seed procedures.
  • T043 [Plan:5.1] Provision the T002-selected upload backend: resilient object storage/versioning/replication or one-way local sync with atomic transfer and periodic checksum reconciliation; enforce private data ACLs.
  • T044 [Plan:5.1] Implement safe file-sync pause/drain/direction reversal after role change, pending-age metrics and tombstone retention; preserve recovery copies and prohibit blind live bidirectional merge.
  • T045 [Plan:5.1] Configure independent encrypted DB base/WAL, file versions and config backups with retention/immutability; define DB/file checkpoint correspondence and protected restore credentials.
  • T046 [Plan:5.1] Implement CubeStore persistence or measured rebuild according to the approved policy; reconcile installer/updater analytics settings and document reporting degradation/recovery time.
  • T047 [Plan:5.1] Provision and verify independent fencing and traffic-provider credentials, TLS/public-domain routing and DR-side 3CX/license/release connectivity; scope agent, host and provider privileges separately.
  • T048 [Plan:5.1] Add external metrics/alerts for replica lag, WAL retention disk, Redis/file backlog, stale readiness, storage capacity, certificate expiry, backup age and degraded DR status.

Acceptance: source/replica consistency and independent restore are demonstrable; file/DB/queue loss windows are reported separately; replica lag and unreachable providers cannot be mistaken for ready DR.

Phase 6: Updater/Manager Agents and Release Coordination​

Plan item: 6.1. Owner: Go updater/release team. Requirements: REQ-009, REQ-011, REQ-012, REQ-013, REQ-016. Input: config/member contracts, runtime safety and replication status.

  • T049 [Plan:6.1] Add authenticated outbound controller connectivity in uc-enterprise-updater/cmd/server/main.go and internal handler/service boundaries; verify member/deployment, command signature, nonce, expiry and authorization.
  • T050 [Plan:6.1] Implement durable idempotent local operation execution, restart-resumption and unknown-outcome inspection under internal/operations; keep local leases subordinate to pair authority, not an alternate controller.
  • T051 [Plan:6.1] Report actual Docker image digests, config/schema revision, role/member and fresh replica/file readiness through internal/monitoring, metrics and managed-service config.
  • T052 [Plan:6.1] Add allowlisted quiesce/start-passive/promote/reseed/verify commands using bounded local adapters; do not accept arbitrary shell commands or expose Docker control directly to the browser.
  • T053 [Plan:6.1] Make Resource Guard/manual restart paths in internal/resourceguard, resourceallocator and operation handlers role/epoch-aware; restart cannot resurrect old-active writers or bypass deployment locks.
  • T054 [Plan:6.1] Extend release metadata/manifests to pinned service digests, config/migration checksums, schema compatibility and rollback support; include Manager and infrastructure images, not only API/Web/Cube tags.
  • T055 [Plan:6.1] Add explicit target-release behavior in internal/services/deployment_service.go; prevent independently selecting next version on each host and preserve approved rollback images.
  • T056 [Plan:6.1] Replace unsafe migration-on-boot/temp-container validation in internal/services/container_service.go; run one audited migration/seed phase on the writer and isolate candidate validation from production side effects.
  • T057 [Plan:6.1] Normalize installer/updater mounts, Docker DNS, ports, environment and health contracts through internal/config/services.go and installer config generation; retain deployable manifests after install/update.
  • T058 [Plan:6.1] Implement externally driven Manager upgrade/recovery and pair release status/progress reporting; validate identity, authority and command resumption after manager replacement.

Acceptance: stale/replayed commands fail; actual versions are measurable; temporary containers do not mutate production; recovery and updates share exclusion rules; standby cannot self-upgrade into incompatibility.

Phase 7: Installer Features and Optional DR Lifecycle​

Plan item: 7.1. Owner: installer team. Requirements: REQ-001, REQ-002, REQ-003, REQ-005, REQ-012, REQ-017. Input: enrollment, configuration, manager and storage contracts.

  • T059 [Plan:7.1] Add default standalone/DR-primary/DR-join choices in uc-enterprise-installer/ui/src/pages/InstallationConfig/InstallationConfig.js; expose only applicable fields and retain the existing standalone experience.
  • T060 [Plan:7.1] Add entitlement, storage/fence profile, private connectivity, capacity and host-identity preflight checks through backend/services/specsService.js and setup controllers; failures cannot overwrite existing active settings.
  • T061 [Plan:7.1] Add primary DR provisioning and invitation handoff through backend/services/licenseService.js and installationPipeline.js; report primary installation success separately from pair readiness.
  • T062 [Plan:7.1] Implement secure join mode and resumable member enrollment in installer routes/controllers; do not call normal license activation/domain bind or seed a second empty production DB.
  • T063 [Plan:7.1] Bootstrap secondary DB/Redis/files using approved agents/profiles; persist resumable provisioning phases in installationPipeline.js and progress/status ownership.
  • T064 [Plan:7.1] Add retrofit-to-DR workflow for existing installations; preserve license expiry/domain, active image/config and data while adding a member without reinstalling primary.
  • T065 [Plan:7.1] Generate persistent role-aware compose/environment/Nginx artifacts through dockerConfigService.js, envService.js and nginxService.js; application boot must not run migrations on a standby.
  • T066 [Plan:7.1] Add readiness/replication/partial-failure status to installer Progress/Summary/Finish pages; show enrolling/syncing/degraded separately from install-complete and DR-ready.
  • T067 [Plan:7.1] Implement enrollment cancellation/retry and failed-join cleanup with retained recovery data; do not revoke/rebind the healthy active license as rollback.
  • T068 [Plan:7.1] Implement safe DR disable/detach and host replacement interfaces; retain the confirmed active member as standalone, revoke removed credentials and apply approved data retention rather than delete automatically.

Acceptance: all three journeys work; join/retrofit never reset active license/data; failed secondary provisioning leaves the primary usable; removal works even when the original secondary is active.

Phase 8: Partner Portal Features​

Plan item: 8.1. Owner: portal frontend/backend. Requirements: REQ-002, REQ-003, REQ-004, REQ-011, REQ-015, REQ-016, REQ-017, REQ-018. Input: approved contracts and ownership of existing Manager records.

Portal code home is a T004/T008 decision. uc-enterprise-partner/src/app and src/components are existing candidate anchors, not an instruction to repurpose that application without approval. New API/service ownership must be selected explicitly; the portal must never hold raw Docker access.

  • T069 [Plan:8.1] Implement partner/customer tenant boundaries, scoped operator roles, MFA/step-up and approval policy in the approved portal authentication/API home.
  • T070 [Plan:8.1] Implement license inventory/details: modules, seats, status, expiry, DR allowance and deployment links; expose only authorized commercial mutations and audit all changes.
  • T071 [Plan:8.1] Integrate license issuance/assignment/renewal/replacement workflows with existing license APIs and commercial ownership; do not change active deployment expiry implicitly during DR enrollment.
  • T072 [Plan:8.1] Implement deployment/member inventory showing stable deployment ID, host fingerprint/member, current active role/epoch, versions and freshness; label legacy/imported records clearly.
  • T073 [Plan:8.1] Implement Enable/Add DR and secondary invitation workflows with one-time secret handling, progress and expiry; support existing standalone customers and install-time primary enrollment.
  • T074 [Plan:8.1] Implement replace/remove member and disable DR actions with dependency checks, current-active preservation, retention warnings and explicit confirmation.
  • T075 [Plan:8.1] Implement DR dashboard for DB/Redis/files lag, backup health, storage, connectivity, readiness blockers, authority validity and last drill; stale telemetry must not show ready.
  • T076 [Plan:8.1] Implement planned switch/emergency failover/failback forms with target member, preflight, expected epoch, idempotency key, loss acknowledgement and configured approvals.
  • T077 [Plan:8.1] Implement pair upgrade scheduling/status, exact release selection, compatibility warnings, migration progress and permitted rollback actions using controller APIs.
  • T078 [Plan:8.1] Implement durable operation timeline/history with phase, actor, approvals, fencing evidence, target version, errors and permitted resume/retry; never offer unsafe automatic reversal.
  • T079 [Plan:8.1] Implement tenant-scoped alert subscriptions/notification delivery for enrollment failures, degradation, release drift, promotion, expiring credentials/licenses and failed backup/drills.
  • T080 [Plan:8.1] Integrate/migrate CRM-backed Manager license/member records through the T004-approved ownership model; provide an operator API/CLI path using the same authorization/audit when portal UI is unavailable.

Acceptance: unauthorized partners cannot read or mutate another customer's deployment; UI never enables an action based on stale status or bypasses controller checks; browser payloads/logs do not leak invitation or provider secrets.

Phase 9: Recovery Controller and Switching​

Plan item: 9.1. Owner: controller/infrastructure teams. Requirements: REQ-003, REQ-009, REQ-011, REQ-012, REQ-013, REQ-014, REQ-016, REQ-017, REQ-018. Input: role/operation contracts, agents, fence/traffic adapters and licensing.

Controller code home and framework are approved by T004/T008. Tasks below belong to that home; do not put orchestration into the browser or two competing host-local managers.

  • T081 [Plan:9.1] Implement authenticated tenant-scoped operation APIs, request approvals, idempotency and expected-epoch checks backed by durable operation records.
  • T082 [Plan:9.1] Implement pair-wide locking for promotion/update/enrollment/removal and resumable state-machine phases; expired leases and controller restarts require outcome inspection before retry.
  • T083 [Plan:9.1] Implement eligibility/preflight aggregation for member entitlement, actual release/config/schema parity, replica/file readiness, fresh evidence and provider connectivity; return actionable blockers.
  • T084 [Plan:9.1] Implement provider-specific independent fencing with audited proof, timeouts and unknown-outcome handling; inability to prove fence blocks promotion.
  • T085 [Plan:9.1] Implement planned switchover: quiesce, drain, final WAL/file/queue checkpoint, fence, atomic epoch reservation, promote dependencies and authorize target startup.
  • T086 [Plan:9.1] Implement operator-approved emergency failover with explicit data-loss window, fencing and uncertainty review; unreachable primary is not sufficient evidence of isolation.
  • T087 [Plan:9.1] Implement private smoke verification followed by coordinated Web/API/WebSocket/report/webhook routing through the traffic provider; record DNS/cache effects and partial provider failures.
  • T088 [Plan:9.1] Implement operation resumption/needs-attention and safe cancellation before irreversible steps; after promotion do not automatically restore the stale old writer.
  • T089 [Plan:9.1] Implement old-primary quarantine, DB/Redis/file re-seeding and eligibility checks for failback; use planned switching only after convergence, never merge divergent live state.
  • T090 [Plan:9.1] Implement coordinated releases/migration ownership and promotion blocking during incompatible phases; integrate schema replay checkpoints and explicit rollback compatibility with manager commands.
  • T091 [Plan:9.1] Implement enrollment/removal/replacement orchestration and consistent active authority publication; interrupted operations cannot leave duplicate production authorizations.
  • T092 [Plan:9.1] Implement controller/license/store outage handling, external alerts, durable audit retention and control-state backup/restore without epoch regression; first release refuses promotion without required control authority.

Acceptance: every recovery can resume after interruption; one active authority/writer set holds under concurrency; fencing precedes promotion; traffic moves only after private verification; returned hosts cannot boot as stale writers.

Phase 10: Acceptance, Security and Production Rollout​

Plan item: 10.1. Owner: QA/security/operations, supported by all teams. Requirements: REQ-001 through REQ-019. Input: all first-release component gates and the design's acceptance scenarios.

Tasks here specify release evidence, not tests already implemented. Use existing Jest/Go test conventions for component checks and a separate representative two-host environment for cross-host recovery; one Docker bridge or a browser-only mock cannot prove fencing/replication safety.

  • T093 [Plan:10.1] Establish repeatable isolated two-host/controller/provider test fixtures with representative uploads, CRM schema, Redis jobs and mocked external effects; never point candidate validation at production writers.
  • T094 [Plan:10.1] Verify standalone install/renew/update, install-time DR, later retrofit, failed join retry, DR removal and distinct-identity license lifecycle; preserve legacy commercial limits and expiry.
  • T095 [Plan:10.1] Exercise passive/reboot/stale-epoch behavior for every inventoried writer, cloned disks, revoked tokens, duplicate/replayed commands, concurrent operations and Resource Guard restarts.
  • T096 [Plan:10.1] Exercise uploads/deletes/partial files, DB replay, Redis loss/replay and external-effect uncertainty during host loss/partitions; prove measured RPO and one-active invariants.
  • T097 [Plan:10.1] Interrupt upgrades/migrations/promotion at each phase; verify schema compatibility, actual digest parity, no duplicate seeds, resumable outcomes and no unsafe rollback.
  • T098 [Plan:10.1] Perform loaded planned switch, emergency failover, old-primary quarantine, host replacement and failback; measure transactional/reporting RTO and 3CX/WebSocket recovery.
  • T099 [Plan:10.1] Restore DB/files/config/secrets and control state from independent backups; validate file references and that revoked membership/epochs cannot be reauthorized by restoring old state.
  • T100 [Plan:10.1] Review tenant isolation, MFA/approvals, credential handling/rotation, command authorization, Docker privileges, replica network ACLs, invitation replay and audit tampering.
  • T101 [Plan:10.1] Publish provider-specific deployment, troubleshooting, switchover/failover/failback, lost-control-plane and recovery runbooks in uc-enterprise-wiki/staged-docs/deployment; train support and assign incident owners/escalation.
  • T102 [Plan:10.1] Roll out behind an explicit entitlement/feature gate: internal pilot, selected customers, staged general availability; approve acceptance evidence, fallback/disable policy and recurring drill/backup ownership.

Acceptance: first-release gates and measured RTO/RPO pass in representative infrastructure. Documentation approval is not runtime sign-off. Known gaps and accepted residual risks require named approvers.

Phase 11: Optional Automatic and Offline Recovery​

Plan item: 11.1. Owner: platform/security/operations. Requirements: REQ-018, REQ-020. Input: proven manual recovery plus separate policy approval.

  • T103 [Plan:11.1] Approve and provision independent quorum/witness and automatic detection policy, using a proven DB failover control plane where appropriate; integrate rather than duplicate existing epoch authority.
  • T104 [Plan:11.1] Automate promotion only through the existing fenced state machine with freshness/lag/quorum thresholds, cooldowns and circuit breakers; no two-host ping-based auto-promotion.
  • T105 [Plan:11.1] If approved, implement audited offline break-glass authorization, bounded authority, independent fencing and epoch reconciliation; no local unconditional force-active bypass.
  • T106 [Plan:11.1] Validate partition/quorum loss, simultaneous reboot, witness/controller failure, stale leases and break-glass recovery under load before separately enabling automation for customers.

Acceptance: optional capability meets the same one-active safety invariants and measured recovery goals; customers remain on manual DR unless explicitly enabled.

Requirement Mapping​

Implementation evidence below is required deliverable evidence, not a claim that the feature exists.

RequirementPlan itemsTasks and expected evidence
REQ-0011.1, 3.1, 7.1, 10.1T003, T018, T026, T059, T064, T094: preserved standalone activation/renew/update
REQ-0021.1, 3.1, 7.1, 8.1, 10.1T019-T020, T061-T067, T073, T094: primary/join/retrofit workflows
REQ-0031.1, 2.1, 3.1, 7.1, 8.1, 9.1, 10.1T012-T013, T017-T025, T068, T072-T074, T091, T095: member records and lifecycle
REQ-0041.1, 3.1, 8.1, 10.1T018, T023-T026, T070-T071, T080, T094: entitlement/usage compatibility
REQ-0051.1, 2.1, 5.1, 7.1, 10.1T009-T016, T039, T047, T060, T065, T100: validated host-separated config/secrets
REQ-0061.1, 5.1, 10.1T040-T041, T045, T096, T099: WAL replay, promotion and PITR evidence
REQ-0071.1, 4.1, 5.1, 10.1T032-T034, T043-T045, T096, T099: durable uploads/reconciliation/restore
REQ-0081.1, 4.1, 5.1, 10.1T035, T042, T096: Redis replica and durable job reconciliation
REQ-0091.1, 2.1, 4.1, 6.1, 9.1, 10.1T013, T028-T030, T036, T053, T085-T088, T095: authority/passive gates
REQ-0101.1, 4.1, 5.1, 10.1T029, T037-T038, T046-T047, T098: Cube/3CX/socket recovery
REQ-0111.1, 6.1, 8.1, 9.1, 10.1T051, T054-T058, T077, T090, T097: manifests and pair release state
REQ-0121.1, 4.1, 6.1, 7.1, 9.1, 10.1T027, T056, T062, T065, T090, T097: audited single-writer migrations
REQ-0131.1, 2.1, 6.1, 9.1, 10.1T013-T015, T049-T052, T081-T082, T088, T100: secure durable control/agents
REQ-0141.1, 2.1, 5.1, 9.1, 10.1T013, T047, T084-T087, T096, T098: fence/epoch/routing evidence
REQ-0151.1, 3.1, 8.1, 10.1T025, T069-T080, T100: tenant-scoped management and audit
REQ-0161.1, 2.1, 5.1, 6.1, 8.1, 9.1, 10.1T014, T048, T051, T075, T078-T079, T092, T102: readiness and alert evidence
REQ-0171.1, 7.1, 8.1, 9.1, 10.1T068, T074, T076, T085-T091, T098, T101: recovery/member lifecycle runbooks
REQ-0181.1, 2.1, 3.1, 4.1, 8.1, 9.1, 10.1, 11.1T003, T015, T026, T028, T080, T092, T099, T105-T106: grace/control outage policy
REQ-0191.1, 5.1, 10.1T045, T093-T102: independently restored and staged release evidence
REQ-0201.1, 11.1T103-T106: separately approved quorum/automation evidence

Completion and Tracking​

Every task requires an owner, linked PR/config change or approved decision, acceptance evidence and updated operational documentation. Mark a checkbox complete only after review, not after a container starts. Pair readiness is a runtime health state and can degrade after tasks are complete.

Coverage checkpoints in this folder track 20 requirements, 11 plan items and 106 tasks. They establish documentation traceability only; design approval and runtime acceptance remain separate gates. All implementation checkboxes are initially unchecked.