Tracking

Tracking connects destination-side work to Migration Item State. The runtime provides a Tracking service inside each Pipeline Execution Scope. Process code uses it to stage a final record, record confirmed changes, and retain diagnostic context. The runtime then persists the appropriate evidence with the item's outcome.

Tracking has two main representations: a Tracking Record exposes a stable result that other operations can use; a Destination Journal preserves recorded execution evidence. Both belong to one source identity within one migration definition and live in its Migration Store.

Tracking Record Contract

A Tracking Record Contract is the definition's static agreement about the record required for successful processing. It contains a stable contract ID and a schema. The SDK derives a fingerprint from the schema to detect incompatible changes.

Declare it with Tracking.record, then assign it to the definition's tracking property:

import { Schema } from "effect";
import { Tracking } from "migrate-sdk";

export const referenceTracking = Tracking.record({
  id: "destination-reference@v1",
  schema: Schema.Struct({ destinationId: Schema.NonEmptyString }),
});

export type ReferenceRecord = typeof referenceTracking.schema.Type;

The migration definition owns this contract. A destination helper can return data and record changes without deciding the migration's final record shape.

The contract ID describes the meaning of the stored record, not a particular item or run. Changing that ID or its schema while item state exists can block execution. Plan compatibility changes with the saved state, rather than treating the ID as a display label.

Tracking Record

A Tracking Record is the optional schema-validated object persisted for one migration item. It can contain a destination reference, multiple named references, destination versions, or other information required for later operations. It must encode as a JSON-compatible object.

The process stages the record with Tracking.setRecord. It does not return the record as its process result. The runtime validates and commits the staged value only after the process completes successfully.

Calling setRecord twice does not replace the first value. Stage one complete record after the required work succeeds. Use the schema's inferred type or satisfies to check its shape in TypeScript; the runtime enforces the declared contract at the success boundary.

Migration Reference Lookup reads this record through the referenced definition's schema. A definition intended to supply downstream references should therefore declare a tracking contract.

Destination Journal and its entries

A Destination Journal contains ordered evidence captured during processing and failed rollback attempts. Its entries have two kinds:

A Destination Change Descriptor names and validates one kind of change. Create it with DestinationChangeDescriptor.make(id, schema). Its is predicate identifies matching entries and its decode method returns the typed payload. Destination modules normally expose their descriptors alongside the helpers that record them.

A Destination Journal Segment groups ordered entries from one execution scope. The journal's process segment contains processing evidence; rollbackAttempts contains preserved failed compensation attempts. Repeated changes with the same descriptor remain distinct ordered entries.

Ordinary API calls and Effect logs do not automatically produce journal entries. A helper or process must explicitly record them. Definitions without a tracking contract can still persist nonempty journals.

Pipeline Execution Scope and persistence

A Pipeline Execution Scope is the runtime boundary for one item's processing or rollback. It isolates staged tracking from other items, including items executing concurrently. Each Process Batch Settlement also retains its own scope.

The scope holds live evidence until the runtime saves an item outcome. Neither recordChange nor setRecord is an immediate durable checkpoint or a transaction with the destination. A crash after a remote write but before state persistence can leave unrecorded destination work. Repeatable operations and reconciliation address that gap.

During reprocessing, context.previousState exposes saved evidence. Failed or skipped attempts preserve an existing tracking record and process journal when present. Successful reprocessing replaces them with the new record and process segment. The process journal is not an append-only history of every attempt.

Destination Journal Extensions

A Destination Journal Extension is a named, schema-validated value owned by a helper. Use it for data with semantics the helper defines, such as an import receipt or correlation information.

DestinationJournalExtension.make declares the extension; Tracking.setExtension replaces its value and Tracking.removeExtension removes it. Untouched extensions survive reprocessing, and changes persist with migrated, failed, or skipped outcomes. Extensions are saved at the same item boundary and have no automatic retry, rollback, or history behavior.

Apply the primitives

Use Rollback to interpret saved records and journal changes. The multiple-destination cookbook combines individual change entries with one final composite record.

import { mkdir, writeFile } from "node:fs/promises";
import { Effect, Schema } from "effect";
import {
  DestinationChangeDescriptor,
  DestinationError,
  Tracking,
} from "migrate-sdk";

export const Author = Schema.Struct({
  id: Schema.NonEmptyString,
  name: Schema.NonEmptyString,
});

export const authorTracking = Tracking.record({
  id: "author-file@v1",
  schema: Schema.Struct({ path: Schema.NonEmptyString }),
});

export const authorFileWritten = DestinationChangeDescriptor.make(
  "author-file-written@v1",
  Schema.Struct({ path: Schema.NonEmptyString })
);

// Call this helper inside a process with tracking: authorTracking.
export const writeTrackedAuthor = Effect.fn("writeTrackedAuthor")(function* (
  author: typeof Author.Type
) {
  const path = `./authors/${encodeURIComponent(author.id)}.json`;

  yield* Effect.tryPromise({
    try: async () => {
      await mkdir("./authors", { recursive: true });
      await writeFile(path, JSON.stringify(author, null, 2));
    },
    catch: (cause) =>
      new DestinationError({ cause, message: "Could not write author" }),
  });

  // Evidence of a confirmed write, available if later work fails.
  yield* Tracking.recordChange(authorFileWritten, { path });

  // Exactly one final record for this item's successful outcome.
  const record = { path } satisfies typeof authorTracking.schema.Type;
  yield* Tracking.setRecord(record);
});