AI Data Pipeline Handoffs: Contracts, Lineage, and Rollback

Both files pass their local checks.

The document file has three rows, three unique IDs, and complete lineage fields. The embedding file does too. Neither has a malformed footer, a null identifier, or a duplicate key.

Joined together, they fail.

One embedding points to a document that is no longer in the normalized release. One current document has no embedding. The shared rows claim they came from the previous normalization run, and one embedding row declares an older text revision. Nothing is wrong with either file in isolation. The handoff is wrong.

That is the failure this guide is about: two valid Parquet artifacts can form an invalid AI data release. A folder named final is not lineage, and a path named latest is not an immutable input.

The files passed; the handoff failed#

The two downloadable fixtures are synthetic and deliberately mismatched. This teaching contract has exactly one embedding per document; a chunked retrieval corpus should join on a unique chunk_id or an explicit composite chunk key instead. The document artifact contains doc-001, doc-002, and doc-004 from normalization run norm-042. The embedding artifact contains doc-001, doc-002, and doc-003, but says it consumed norm-041. These invented vectors let us verify the lineage declarations; they cannot prove which text a real model actually received.

Each artifact looks healthy on its own:

The boundary check is where the story changes. Put both fixtures and the complete SQL bundle in one directory, change into it, and run duckdb < ai-pipeline-handoff-audit.sql. The bundle creates temporary documents and embeddings views before executing every check. Its cross-stage query is:

WITH documents AS (
  SELECT
    document_id,
    normalization_run_id,
    text_revision
  FROM read_parquet('ai-pipeline-documents.parquet')
), embeddings AS (
  SELECT
    document_id,
    source_normalization_run_id,
    text_revision AS source_text_revision
  FROM read_parquet('ai-pipeline-embeddings.parquet')
)
SELECT
  count_if(d.document_id IS NULL) AS orphan_embeddings,
  count_if(e.document_id IS NULL) AS documents_without_embeddings,
  count_if(
    d.document_id IS NOT NULL
    AND e.document_id IS NOT NULL
    AND d.normalization_run_id IS DISTINCT FROM e.source_normalization_run_id
  ) AS mismatched_source_run_rows,
  count_if(
    d.document_id IS NOT NULL
    AND e.document_id IS NOT NULL
    AND d.text_revision IS DISTINCT FROM e.source_text_revision
  ) AS mismatched_text_revision_rows
FROM documents AS d
FULL OUTER JOIN embeddings AS e USING (document_id);

The recorded result is:

The row contracts did their job: they told us that each visible file was internally coherent. They could not tell us that the two files belonged to the same release. That requires a contract at the boundary between producer and consumer.

Before using a FULL OUTER JOIN like this, require non-null, unique join keys on both inputs. Duplicate keys can multiply joined rows and turn reconciliation counts into misleading evidence. Vector dimension and numerical quality are separate row-level questions; the LLM shard audit covers those without mixing them into this handoff test.

A handoff is more than a path#

A reliable handoff has four named parts:

1. Immutable data objects — the exact Parquet files or table snapshot the producer wrote.
2. A versioned contract — what those rows mean and which changes consumers accept.
3. A lineage manifest — inputs, outputs, producer revision, run, counts, hashes, and acceptance evidence.
4. A committed pointer — the small catalog record that says which accepted manifest is current.

Those parts repeat across an AI pipeline, even when the row shapes change:

The path is only where a consumer looks. The manifest is what it should find there. If a consumer reads normalized/latest/ repeatedly while a producer is updating it, the same job can observe more than one dataset under one name.

Make the boundary contract executable#

The contract should live in source control or a registry beside the code that validates it. It needs enough information for a producer and consumer to disagree before production, not after a training run. This is an illustrative production contract; the downloadable teaching fixture intentionally implements a smaller schema and does not claim conformance to it.

name: normalized_document
version: 2.1.0
owner: data-platform
producer: normalize-documents
consumers:
  - embed-documents
  - curate-training-corpus
 
primary_key: [document_id]
fields:
  document_id:          {type: string, nullable: false}
  normalization_run_id: {type: string, nullable: false}
  text_revision:        {type: string, nullable: false}
  source_uri:           {type: string, nullable: false}
  normalized_text:      {type: string, nullable: false}
  language:             {type: string, nullable: false}
 
compatibility:
  add_optional_field: consumer_test_required
  add_required_field: major
  rename_or_type_change: major
 
acceptance:
  validator: sql/normalized-document-v2.1.0.sql
  input_manifest_required: true
  rejection_reconciliation_required: true

Parquet stores physical and logical schema information. A writer may still encode a field as physically optional even when this semantic contract says nullable: false; the acceptance query must enforce the declaration. Parquet does not enforce this YAML, know which producer owns a field, or decide whether a type change is compatible with an embedding job. The executable validator and consumer tests give those declarations consequence.

Semantic versioning is a local policy, not a fact encoded in the file. A nullable addition may be a minor change for one reader and a breaking change for another that serializes a fixed column list. The contract should therefore name compatibility rules and the consumers that proved them.

Lineage is the join key between releases#

A useful manifest names the run, the exact inputs it consumed, the objects it produced, and the evidence used to accept them. A compact version looks like this:

{
  "manifest_schema_version": "1.0.0",
  "dataset_id": "normalized-documents/norm-042",
  "contract": {
    "uri": "s3://example/contracts/normalized-document/2.1.0.yaml",
    "sha256": "…"
  },
  "producer": {
    "job": "normalize-documents",
    "run_id": "norm-042",
    "code_revision": "git:8c40d2a7781d130944c9ec728e45087eff94bcdc"
  },
  "inputs": [
    {
      "dataset_id": "raw-capture/capture-118",
      "manifest_uri": "s3://example/manifests/raw/capture-118.json",
      "manifest_sha256": "…"
    }
  ],
  "objects": [
    {
      "uri": "s3://example/run=norm-042/part-000.parquet",
      "version_id": "…",
      "sha256": "…",
      "rows": 3
    }
  ],
  "acceptance": {
    "validator": {
      "uri": "s3://example/validators/normalized-document-v2.1.0.sql",
      "sha256": "…",
      "engine": "DuckDB v1.5.3"
    },
    "result": {
      "status": "accepted",
      "blocking_findings": 0,
      "reviewed_exceptions": 0,
      "sha256": "…"
    },
    "accepted_at": "2026-08-24T12:00:00Z"
  },
  "previous_accepted": {
    "manifest_uri": "s3://example/manifests/normalized/norm-041.json",
    "manifest_sha256": "…"
  }
}

The ellipses are placeholders, not valid evidence. The downloadable teaching manifest contains complete hashes for the supplied fixtures and audit bundle.

Parquet key-value metadata can carry a contract ID, run ID, or manifest ID near the rows. Do not make the footer your only lineage catalog. One release can contain many files and derive from many inputs; a manifest or lineage store is easier to reconcile and query across stages.

The OpenLineage object model (opens in a new tab) makes a similar distinction between datasets, jobs, and individual runs, with input and output facets for run-specific facts. You do not have to adopt that standard to use the idea, but avoid collapsing all three identities into a single path.

Reconcile reality with the manifest#

A manifest is only evidence if publication stops when reality differs from it. Before promotion, compare:

• the exact listed objects with what was produced—no missing or extra parts;
• object versions or hashes, not filenames alone;
• per-object and total row counts;
• input rows against output plus explicitly rejected rows when that equation fits the transformation;
• schema fingerprints and semantic contract versions;
• producer code, dependency, model, tokenizer, and rule revisions where they affect output;
• cross-stage identifiers and source revisions, as in the fixture query;
• acceptance SQL or code hash, engine version, results, exceptions, and reviewer.

Reconciliation equations must describe the actual transformation. A deduplicating stage may intentionally produce fewer rows. A chunker normally produces more. “Input rows equal output rows” is useful only when the contract says the relationship should be one-to-one.

Keep measured results separate from policy and judgment. “One orphan embedding” is a measured result. “Any orphan blocks release” is a local policy. “The candidate was rejected and rebuilt from norm-042” is the decision. Combining those into a single green badge makes incident review much harder.

The pointer is the commit boundary#

Build a candidate where consumers cannot mistake it for the current release:

1. Write Parquet objects under a run-scoped, immutable location.
2. Close writers and resolve the exact object list and versions.
3. Run file, contract, reconciliation, and consumer-compatibility gates.
4. Write an accepted manifest that names the evidence.
5. Update the current catalog pointer with an appropriate conditional or transactional operation.
6. Retain the previous accepted manifest; expire unreferenced candidates later under policy.

Never assume an object-store rename has filesystem semantics. In Amazon S3 general-purpose buckets, renaming uses copy and delete (opens in a new tab); S3 Express One Zone directory buckets instead provide a single-object RenameObject operation (opens in a new tab). Neither makes renaming a prefix full of Parquet objects a dataset-level transaction. Use the manifest or catalog pointer as the logical publication boundary, verify your backend's behavior, and protect concurrent pointer updates with an operation such as S3 If-Match conditional writes (opens in a new tab) or a catalog transaction.

The candidate is now complete, but it is still not current until the pointer changes. A consumer should resolve that pointer once, open the immutable manifest it names, and record the manifest identity in its own output. It should not keep resolving current throughout the run.

The same pointer that commits forward gives rollback a precise target. Rollback changes the pointer to a previously accepted manifest; it does not guess which files happened to be present in an old folder.

Atomic publication does not guarantee compatibility#

A committed release can still break a consumer. Publication answers “Is this the whole accepted snapshot?” Compatibility answers “Can this consumer interpret it safely?”

Inspect every candidate footer when a release contains raw Parquet files. This query intentionally reports leaf fields from each physical schema; for nested data, retain and review the parent structure rather than treating leaf names as globally unique:

SELECT
  file_name,
  name AS column_name,
  field_id,
  column_id,
  type AS physical_type,
  logical_type,
  duckdb_type
FROM parquet_schema('candidate/*.parquet')
WHERE type IS NOT NULL
ORDER BY file_name, column_id;

DuckDB can read differing files with union_by_name = true. That is a read strategy, not an approval. A missing column filled with NULL may still violate a downstream contract.

The DuckDB Parquet guide (opens in a new tab) documents multi-file reads and name-based schema unification. If you need table-level transactions, snapshots, and defined schema evolution, use a table format rather than attributing those properties to Parquet files. Apache Iceberg schema evolution (opens in a new tab) uses field IDs so compatible changes are not inferred only from names and positions; those guarantees come from Iceberg metadata. Rollback is still bounded by retention: expired snapshots are no longer available (opens in a new tab) for time travel or rollback.

Every downstream release names its input#

Once norm-042 is accepted, the embedding job should consume that manifest—not a mutable prefix—and publish its own:

• normalization output names the exact raw-capture manifest and normalization code revision;
• enrichment output names the normalized manifest and model or rule revision;
• embedding output names the normalized manifest, text revisions, and embedding model revision;
• curated training output names the source manifests, split/dedup policy, and tokenizer/template revision;
• a trained model artifact names the exact corpus manifest, code, configuration, and checkpoints;
• an evaluation result names the model artifact, frozen evaluation manifest, scorer, and settings.

This chain answers the incident question that a folder tree cannot: which derived artifacts consumed the bad release? Rolling the normalized pointer back does not delete embeddings already indexed or untrain a model already produced. Lineage identifies what must be invalidated, rebuilt, or separately remediated.

For a model-facing handoff, “model version present” is too weak. Validate one expected immutable model or tokenizer revision—preferably a full artifact digest or commit—and record the input text revision or digest, tokenizer configuration and chat-template hash when relevant, special-token and truncation policy, embedding dimension and dtype, pooling, and normalization policy. The teaching audit enforces its expected model revision and single-revision release, but it does not pretend to exercise every one of these production controls.

For each stage, record a start event only after resolving exact inputs and a completion event only after publishing exact outputs. The producer run and dataset release are related but not interchangeable: a retry may share a logical job while producing a new run, and a run may fail without producing an accepted dataset.

Reproduce the example, then name the release#

The evidence package was generated on August 24, 2026 with native DuckDB v1.5.3. The nine published SQL statements are also executed in the repository test against the shipped DuckDB-WASM v1.4.3 build. It is synthetic teaching data, not a production benchmark, and the three-dimensional vectors are invented values with no semantic meaning.

To rebuild the package, download the generator and audit SQL into one directory, run python -m pip install duckdb==1.5.3, then run python generate_ai_pipeline_handoff_fixtures.py. The fixtures and manifest are written beside the script.

To verify the exact downloads, the document fixture is 2,032 bytes with SHA-256 04224164c6cf520359d9a8e4ef0ada743a4e483df988258ebceb3673019b4247; the embedding fixture is 2,138 bytes with SHA-256 8253195df3f13bbf273aa999e28e85955d41fa1fe49e6d15bb7d23e148316a9b. The teaching manifest records the remaining artifact hashes and the five failing IDs.

You can inspect either file in the local Parquet viewer. The cross-artifact join belongs in native DuckDB or the publishing pipeline because it needs both inputs and must gate the complete release.

Before a consumer starts, it should be able to name four things:

1. Which exact dataset did I read? An immutable manifest or table snapshot, not only a mutable path.
2. Which contract did it satisfy? Including the compatibility policy tested for this consumer.
3. What produced it? Exact input manifests plus the producer run and relevant code/model revisions.
4. Where is the rollback target? The previous accepted manifest or snapshot.

Parquet is the payload. The release is the contract, manifest, and committed pointer around it. If a consumer cannot name all three, it cannot tell you what it read.

Primary references:

• DuckDB Parquet reading and schema inspection (opens in a new tab)
• DuckDB Parquet metadata functions (opens in a new tab)
• Amazon S3 copying, moving, and renaming objects (opens in a new tab)
• Amazon S3 conditional writes (opens in a new tab)
• Amazon S3 Express One Zone RenameObject (opens in a new tab)
• OpenLineage object model (opens in a new tab)
• Apache Iceberg schema evolution and field IDs (opens in a new tab)
• Apache Iceberg snapshot maintenance and expiration (opens in a new tab)

For row-level leakage, tokens, and embedding quality, use the LLM shard audit. For partitions, row groups, codecs, file sizing, and compaction, continue with the Parquet pipeline layout guide.