Skip to main content

Data model and migrations

The data plane is MongoDB Atlas. Types live in a single shared library (libs/shared/types-data) consumed by both the API and the UI, so the wire shape and the database shape are the same shape.

This page covers the entity layout, the base-entity + Mongoose-discriminator pattern, the migration registry, the MigrationBootCoordinator (cluster-safe boot-time migrations), and the idempotency log.

Entity layout​

All entity types are defined in libs/shared/types-data/src/lib/, organized by domain:

DomainPathExamples
Corecore/organization, user, membership, role, permission, auth, audit-event, impersonation, file, email, brand-kit
Productproduct/ProductDefinition, OrgProduct
Integrationsintegrations/IntegrationDefinition, IntegrationInstance, VendorCredential
Schedulerscheduler/booking-flow, booking-flow-router, booking-question, booking-session, booking-session-event, etc.
Notificationnotification/notification-dispatch, notification-settings
Marketingmarketing/contact-request, waitlist-request
Commoncommon/errors, idempotency-log
Data migrationdata-migration/migration registry types

Every entity extends a common BaseEntity shape — _id, createdAt, updatedAt (camelCase, per libs/shared/types-data/src/lib/common/entities.ts:28), plus org-scope discriminator fields where applicable via BaseEntityWithOrganization. Repository implementations consume these via the generic BaseRepository<TEntity, TDocument> at libs/api/utils-api/src/lib/repository/base-repository.common.ts.

Discriminated entities (Mongoose discriminators)​

Several entities are discriminated unions on a type-like field. They use Mongoose's discriminator support so the same collection holds multiple sub-shapes with different validation rules:

  • Organization — discriminated on org_type (PLATFORM | BRAND | PORTFOLIO | PARTNER). See libs/shared/types-data/src/lib/core/organization/organization.entity.ts:26-46.
  • OrgProduct — discriminated on product_key (Scheduler today). See libs/shared/types-data/src/lib/product/org-product.entity.ts:48.
  • IntegrationInstance — discriminated on integration_vendor. See libs/shared/types-data/src/lib/integrations/integration-instance.entity.ts:49.
  • NotificationSettings — discriminated on product_key.

The discriminator pattern is documented at apps/dev-docs/docs/api/patterns/discriminator-schema-pattern.md.

Migration registry​

Schema changes ship as migrations registered with the migration module. The migration types live at libs/shared/types-data/src/lib/data-migration/; the runtime module lives at libs/api/services-api/src/lib/data-migration/.

Each migration has:

  • A unique key (the registry name)
  • An up() function (apply the change)
  • A down() function (revert)
  • Status tracking (PENDING, RUNNING, COMPLETED, FAILED, ROLLED_BACK) persisted per migration record

The status field is the source of truth for "has this migration run on this cluster?"

MigrationBootCoordinator​

Cluster boot ordering is the hard part. Every ECS task boots roughly together; without coordination, multiple tasks would race to apply the same migration. The platform solves this with the MigrationBootCoordinator.

  • Source: libs/api/services-api/src/lib/data-migration/migration-boot-coordinator.ts
  • E2E pin: apps/api/e2e/multi-instance/migration-boot-coordination.e2e-spec.ts covers the cluster-coordinated boot scenario.

The pattern is a composition of two primitives at the consumer layer:

  1. Orchestration: a single RedisDistributedLockService lock held by the winning task; lock losers wait by polling MongoDB for convergence.
  2. Correctness: per-migration atomic CAS on the migration record (findOneAndUpdate with $or matching recoverable terminal states) plus a heartbeat lease that refreshes updatedAt across the entire up() and rollback down() window. The heartbeat covers the full side-effecting span, so a per-migration single-writer invariant holds even when up() throws and down() is mid-flight.

The composition lets the per-migration claim survive orchestration-lock TTL co-acquisition: even if two tasks both think they're the orchestrator, the per-migration CAS guarantees exactly one writer per migration.

Trigger paths:

  • Bootstrap — runs at API task boot. Losers wait for convergence.
  • Admin endpoint — manual trigger via the admin controller. Losers return contended.

The two paths share a private run core; the LoserPolicy value (wait-for-convergence vs return-contended) parameterizes the outcome.

The coordinator also implements OnApplicationShutdown. When the host receives SIGTERM (a Nest graceful shutdown, a hot-reload-driven dev restart, an ECS task replacement), the hook releases the held Redis lock so the next replica doesn't wait out the lock's full TTL or fall back to the stale-takeover window for a recovery the orchestrator has already initiated. The release is bounded by MIGRATION_SHUTDOWN_LOCK_RELEASE_TIMEOUT_MS (default 5s) so a Redis stall cannot block the shutdown sequence — the lock's own TTL remains the safety net on timeout. Correctness is preserved by the per-migration CAS + heartbeat lease: any in-flight migration is still single-writer protected regardless of what happens to the orchestration lock.

OpenSpec contract: openspec/specs/data-migration-coordination/spec.md.

MigrationsAppliedListener (post-migration hook)​

After the coordinator reaches CONVERGED (or CONVERGED_WITH_FAILURES), it invokes every registered MigrationsAppliedListener so process-local state derived from migration-touched collections can refresh.

  • Contract: libs/api/services-api/src/lib/data-migration/lifecycle/migrations-applied-listener.types.ts
  • Report types (shared with the admin HTTP response): libs/shared/types-data/src/lib/data-migration/migration-listener.types.ts
  • Today's implementers: ProductDefinitionService, IntegrationDefinitionService (in-memory catalog caches — see Catalog and enablement → In-memory cache and post-migration rehydration)

Each listener is raced against a 30s per-call timeout. Outcomes are aggregated into a structured MigrationListenerReport:

type MigrationListenerReport = {
  reason: 'bootstrap' | 'admin';
  invokedAt: string;
  totalDurationMs: number;
  outcomes: MigrationListenerOutcome[];   // per-listener: ok | failed | timeout
};

Escalation policy is path-specific:

  • Bootstrap path: any non-OK outcome throws an aggregate error from notifyOrAbortBootstrap; Nest aborts bootstrap; ECS replaces the pod. A partial cache state is worse than no cache.
  • Admin path: the report is returned in the HTTP response (MigrationRunResponseDto.listenerReport) so an operator triggering migrations sees per-listener results inline, without grepping logs. The path does not throw on listener failure.

Listeners are wired via direct constructor injection on MigrationBootCoordinator — no DI token, no multi-provider, no DiscoveryService. Adding a new listener means appending it to the coordinator's constructor parameters and to the private readonly listeners array assigned in the constructor body. Cross-pod coordination is handled at the Mongo-durability layer; see Catalog and enablement → In-memory cache and post-migration rehydration for the invariant.

Post-deploy verification: SuperAdmin platform-exclusivity​

Migration 034 (scripts/034-strip_super_admin_role_from_non_platform_memberships__05_2026.ts) is the data-side fix that strips SuperAdmin from non-Platform memberships. The operational verification surface is the audit log: the permission guard emits an access_bypass event on every SuperAdmin short-circuit, so the volume on non-Platform contexts is the regression signal.

Audit events persist in MongoDB (Collections.AUDIT_EVENTS) with a 24-hour TTL — verification must happen inside that window. The target_org_id field is emitted only on org-scoped routes (permission.guard.ts spreads it conditionally), so the query must guard against documents where the field is absent.

db.auditevents.find({
  event_type: 'access_bypass',
  'metadata.target_org_id': {
    $ne: '000000000000000000000001', // PLATFORM_ORG_ID
    $exists: true,                    // skip non-org-scoped routes
  },
}).count();
  • Expected: count drops to near-zero within ~5 minutes of deploy, with a rare in-flight-request tail.
  • If count stays elevated: a write path is re-introducing SuperAdmin to non-Platform memberships. Today's mutation endpoints don't enforce the platform-exclusivity invariant on the write side; the follow-up gate on MembershipService.applyMembershipRoles closes that hole.

Idempotency log​

The platform uses a MongoDB-backed idempotency log for cross-domain dedup. The compound unique index on (organization_id, domain, idempotency_key) enforces "don't run this side-effect twice."

  • Source: libs/api/services-api/src/lib/core/idempotency-log/ (or under common/idempotency-log — see types at libs/shared/types-data/src/lib/common/idempotency-log/)
  • Contract: tryAcquire({ org, domain, key, ttl_days }) → { acquired: true; record } or { acquired: false; reason: 'duplicate' | 'error' }. finalize(recordId, { status, metadata }) updates post-execution.

Use cases today: notification dispatch dedup, webhook idempotency, side-effect emits where the dedup gate is at-the-emit rather than at-the-resource.

For a comparison of when to reach for idempotency-log vs. DistributedLock vs. ResourceLease, see Cluster coordination pattern → idempotency-log.

OpenSpec contract: openspec/specs/idempotency-log/spec.md.

Migration pattern documentation​

The base repository pattern, the discriminated schema pattern, and the cross-cutting type-system shape are documented separately under apps/dev-docs/docs/api/patterns/:

This page is the architectural overlay; those pages are the implementation reference.