How to Audit Parquet Data for LLM Training and RAG

The first leakage check on our training fixture returns no rows. The source IDs are different, the split labels look independent, and the Parquet file opens without complaint.

The validation split still contains training text.

One validation example is a case-and-whitespace variant of a training example. It entered under a different source ID, so the obvious group-overlap query missed it.

That small failure captures the larger problem. A valid Parquet file is only a container. It cannot tell you whether a row matches the trainer. It cannot prove that evaluation text stayed out of training, token IDs came from the intended tokenizer, or an embedding still belongs to the text beside it.

The useful question is not “Is this valid Parquet?” It is “What evidence would make this exact dataset safe enough for this exact consumer?”

Meet the deliberately broken shard#

The training fixture contains eight synthetic rows. We planted one failure of each kind so every result can be inspected by hand: a duplicate ID, blank text, an invalid split, missing provenance, a missing license, empty tokens, a mismatched tokenizer revision, and the copied text that the first leakage check will miss.

In this guide, a release gate is a deterministic check that can stop or flag one candidate dataset before a consumer uses it.

Open it in the local Parquet viewer, inspect the schema, and run this aggregate query against the loaded data relation:

SELECT
  count(*) AS rows,
  count_if(example_id IS NULL OR trim(example_id) = '') AS missing_example_ids,
  count(*) FILTER (WHERE example_id IS NOT NULL AND trim(example_id) <> '')
    - count(DISTINCT example_id)
      FILTER (WHERE example_id IS NOT NULL AND trim(example_id) <> '')
    AS duplicate_id_excess_rows,
  count_if(text IS NULL OR trim(text) = '') AS blank_text_rows,
  count_if(split IS NULL OR split NOT IN ('train', 'validation', 'test'))
    AS invalid_split_rows,
  count_if(source_uri IS NULL OR trim(source_uri) = '') AS missing_source_rows,
  count_if(license IS NULL OR trim(license) = '') AS missing_license_rows,
  count_if(token_ids IS NULL OR array_length(token_ids) = 0) AS empty_token_rows,
  count_if(
    tokenizer_revision IS NULL
    OR tokenizer_revision <> 'demo-tokenizer-v1'
  ) AS unexpected_tokenizer_rows
FROM data;

The observed result is deliberately untidy:

That is seven blockers under this demo contract before we have selected a single text value. The copied-text failure is the eighth, and we will find it separately.

Keep those stages separate. A count of one is a clue, not a repair instruction. A repeated ID might be a copied row. It might also show that the contract needs a composite key. Trace the failing identifier upstream; do not add a generic “delete duplicates” step and call the dataset clean.

These are aggregate blockers, not necessarily cheap ones. COUNT(DISTINCT), text trimming, and later similarity checks can be expensive on a large corpus. Measure them on your layout before moving the same rules into a publishing pipeline.

Not every LLM dataset has the same row contract#

“LLM data” is not one schema. A pretraining reader, a supervised fine-tuning (SFT) trainer, an evaluation harness, and a retrieval-augmented generation (RAG) indexer can all read Parquet while disagreeing about what a row means. That meaning is the row contract.

The Hugging Face TRL dataset-format guide (opens in a new tab) makes the contract distinction concrete for language-modeling and preference trainers. Hugging Face Datasets (opens in a new tab) and NVIDIA NeMo DocumentDataset (opens in a new tab) document Parquet ingestion. Reader support does not tell you what a row ought to mean. Pin the consumer, its version, and the semantic contract you actually deploy.

Parquet itself can encode strings, lists, maps, and nested structures. The Apache Parquet logical-types specification (opens in a new tab) describes how those values are represented. Whether they are the right values is still your responsibility.

Prove the bytes and schema before interpreting values#

Start with file identity. Record the file hash or immutable object version, byte size, row count, physical schema, writer, and reader. If a dataset can change behind the same URL, the URL is not an identity.

Then record semantic identity: the row-contract version, generation run, and exact source snapshot. These facts connect readable bytes to the release you intended to build.

For the opened fixture:

SELECT version() AS duckdb_version;
DESCRIBE SELECT * FROM data;

For a native multi-shard directory, inspect every footer rather than trusting the first inferred schema:

SELECT
  file_name,
  name AS parquet_node,
  type AS physical_type,
  logical_type
FROM parquet_schema('candidate/*.parquet')
WHERE name IS NOT NULL
ORDER BY file_name, parquet_node;

The DuckDB Parquet documentation (opens in a new tab) covers DESCRIBE, parquet_schema, pushdown, and multi-file reads. DuckDB can reconcile columns by name with union_by_name, which is useful for investigation but is not evidence that a schema change was approved. A silently null-filled column can still break a trainer or change the meaning of a row.

Provenance needs the same precision. A source URL may let you revisit a public page today and still fail to reproduce a transformed row six months later. Keep a stable source ID, exact source revision, transformation run, and deletion key. A license label is not a legal conclusion. If the value is not in the row, keep an authoritative external mapping so policy checks can still resolve it. Datasheets for Datasets (opens in a new tab) and Hugging Face dataset cards (opens in a new tab) are practical starting points for documenting composition, collection, preprocessing, intended use, and maintenance.

The first leakage check passes—and that is the trap#

Our first check groups rows by source document. If a document produced many chunks, it should be assigned to one split before chunking, or the equivalent invariant should be enforced afterward.

SELECT
  source_document_id,
  list_sort(list(DISTINCT split)) AS splits
FROM data
WHERE source_document_id IS NOT NULL
GROUP BY source_document_id
HAVING count(DISTINCT split) > 1;

The query returns no rows. That sounds like good news. It is also incomplete: train-001 and train-003 carry the same sentence under different source IDs. The second copy differs only in case and surrounding whitespace.

Now normalize the text and compare a digest—a fixed-length text fingerprint—across splits:

WITH normalized AS (
  SELECT
    example_id,
    split,
    sha256(regexp_replace(lower(trim(text)), '\s+', ' ', 'g')) AS text_digest
  FROM data
  WHERE text IS NOT NULL AND trim(text) <> ''
)
SELECT
  text_digest,
  list_sort(list(DISTINCT split)) AS splits,
  list_sort(list(example_id)) AS example_ids
FROM normalized
GROUP BY text_digest
HAVING count(DISTINCT split) > 1;

Observed result: one normalized-text digest appears in both train and validation.

The two rows are small enough to inspect directly:

After trimming, lowercasing, and collapsing whitespace, both rows become a parquet row group stores column chunks for a subset of rows.. Their SHA-256 is the same: f798ac6b0c6daa373b1055d786fd8a18c437adda4a40dc09810ed131eadc74cc.

The report contains a digest and row IDs, not normalized text. That reduces accidental disclosure, but a digest of predictable text is not anonymization. Treat it like another corpus-derived identifier.

This simple normalization catches case and whitespace changes. It misses Unicode variants, punctuation edits, boilerplate, overlapping chunks, translations, and paraphrases. A fuller review can layer source-group overlap, exact and normalized digests, n-gram or chunk overlap, near-duplicate retrieval, and manual review of borderline clusters. Evaluation prompts and answers deserve their own contamination checks.

There is no universal similarity threshold. Tune one against known duplicates and legitimate repeated material, then retain false-positive and false-negative examples with the policy. The point is not to make the report say zero; it is to understand what each signal can and cannot see.

Text, conversations, and tokens are different evidence#

The fixture stores flat text and invented token IDs. It can demonstrate blank-text, revision, and length checks; it cannot demonstrate that a nested conversation matches an SFT trainer.

Token IDs mean something only under one tokenizer artifact and preprocessing configuration. Bind them to the tokenizer repository and immutable revision, added and special tokens, chat-template revision, beginning- and end-of-sequence token policy, truncation side and limit, packing policy, and mask construction. Then inspect their distribution:

SELECT
  split,
  count(*) AS rows,
  round(avg(length(text)), 1) AS mean_characters,
  approx_quantile(length(text), 0.5) AS p50_characters,
  approx_quantile(array_length(token_ids), 0.5) AS p50_tokens,
  max(array_length(token_ids)) AS max_tokens
FROM data
GROUP BY split
ORDER BY split;

The invalid archive split and zero-token test row stand out immediately. On a real corpus, compare the 50th, 90th, and 99th percentile lengths (p50, p90, and p99). Also compare truncation counts, packed-example counts, and loss-mask coverage by source and split. The model and trainer contract should set the limits; this toy distribution should not.

A direct semantic check re-tokenizes every row when feasible, or a deterministic stratified sample otherwise. Compare token IDs, rendered template, truncation, and masks with the pinned training code. Range checks alone are weak because added and special tokens can extend a base vocabulary.

What this flat-text fixture does not demonstrate

For conversational SFT, validate non-empty message lists, allowed roles, present content, the required assistant target, and structurally paired tool calls and results. Render examples with the exact chat template and compare the supervised mask with the training code. For preference data, confirm that chosen and rejected responses are present, different, and attached to the same prompt. Counting messages in Parquet cannot prove that the trainer masks prompt tokens correctly.

The RAG branch: a valid vector can still be wrong#

Embeddings normally belong to the retrieval path, not an ordinary language-model training row. Open the separate RAG fixture. Its dimension of three is only a teaching device.

This query numbers rows before unnesting their vectors. That detail prevents duplicate or missing chunk IDs from accidentally combining independent vectors during the audit.

WITH numbered AS (
  SELECT row_number() OVER () AS audit_row_number, *
  FROM data
), vector_stats AS (
  SELECT
    audit_row_number,
    count_if(value IS NULL OR NOT isfinite(value)) AS non_finite_values,
    sum(value * value) AS squared_norm
  FROM numbered, UNNEST(embedding) AS vector(value)
  GROUP BY audit_row_number
)
SELECT
  count(*) AS rows,
  count_if(d.chunk_id IS NULL OR trim(d.chunk_id) = '') AS missing_chunk_ids,
  count(*) FILTER (WHERE d.chunk_id IS NOT NULL AND trim(d.chunk_id) <> '')
    - count(DISTINCT d.chunk_id)
      FILTER (WHERE d.chunk_id IS NOT NULL AND trim(d.chunk_id) <> '')
    AS duplicate_chunk_id_excess_rows,
  count_if(d.embedding IS NULL) AS missing_embedding_rows,
  count_if(d.embedding IS NOT NULL AND array_length(d.embedding) <> 3)
    AS wrong_dimension_rows,
  count_if(coalesce(v.non_finite_values, 0) > 0) AS non_finite_rows,
  count_if(
    array_length(d.embedding) = 3
    AND coalesce(v.non_finite_values, 0) = 0
    AND v.squared_norm = 0
  ) AS zero_vector_rows,
  count_if(
    d.embedding_model_revision IS NULL
    OR d.embedding_model_revision <> 'demo-embed-v1'
  ) AS unexpected_model_rows
FROM numbered AS d
LEFT JOIN vector_stats AS v USING (audit_row_number);

Dimension and finiteness are necessary, not sufficient. Two models can share a dimension while producing incompatible spaces. A finite zero vector can pass numerical validation and still be useless for similarity. A correctly shaped vector can still belong to yesterday’s text.

A production RAG contract should bind each vector to a stable chunk ID, source document and position, exact text revision or digest, embedding model revision, preprocessing or instruction-prefix revision, dimension, numeric type, generation run, and target index version. Compare vector norms and duplicate-vector rates with a known-good build, then re-embed a deterministic sample to verify both the join and the numerical tolerance. Do not copy a norm threshold from an unrelated model.

Distribution drift is a different kind of failure#

A dataset can satisfy every row-level rule and still be the wrong release. Compare the candidate with the last accepted build by split, source, language, label, document length, token length, conversation turns, safety category, and—when relevant—embedding norm. Start with aggregates, then inspect stratified samples; a global average can hide one source family collapsing to blanks or one language disappearing from evaluation.

Keep measurement, policy, and judgment separate. Here is a hypothetical example; it is not output from the supplied fixtures:

• We measured: validation p99 token length rose from 1,820 to 3,940.
• Our local policy says: changes above 20% require review.
• The reviewer decided: accept the change because an approved long-document source was added.

The distinction prevents a local threshold from masquerading as a universal fact, and it leaves a future reviewer enough context to challenge the decision.

Let the model draft SQL, not decide the release#

So far, every query has been written and reviewed by hand. A model can help translate a contract into DuckDB SQL, explain a query plan, or suggest a missing check. Give it a bounded task that another person can review:

Task: draft one DuckDB SELECT that audits a loaded Parquet relation.
Relation: data
Schema:
- example_id VARCHAR
- source_document_id VARCHAR
- split VARCHAR
- text VARCHAR
- source_uri VARCHAR
- license VARCHAR
- tokenizer_revision VARCHAR
- token_ids INTEGER[]
 
Contract:
- example_id is present and unique
- split is train, validation, or test
- trimmed text, source_uri, and license are required
- tokenizer_revision must equal demo-tokenizer-v1
- token_ids must be non-empty
 
Safety and output rules:
- return aggregate counts only; never select text or source_uri
- treat NULL as a failure for every required field
- use only the listed columns and the data relation
- do not invent thresholds or repair data
- use explicit aliases
- return SQL only

The first aggregate query in this guide is a reviewed answer to that prompt. Treat the prompt as a reusable specification, not as a model benchmark; the SQL still has to be reviewed and executed before use.

AI-drafted SQL adds a few particular risks: invented columns or thresholds, raw-value projection that the task did not require, and unexpectedly expensive joins, sorts, regexes, or UNNEST operations. Run reviewed SQL first on the tiny fixture and then on a bounded sample. Save the accepted query; do not regenerate the release gate on every run. Deterministic code—not the assistant—should assign pass, review, or fail.

Local file viewing and SQL execution happen in the browser without uploading the dataset to viewparquet. Optional AI is a separate network path: messages and configured context go to the selected provider, while chat tools or saved AI context may also provide structural metadata and SQL. A prompt saying “do not expose text” is not access control. Review optional AI data sharing, and provide schema plus aggregate context when raw values are unnecessary.

Turn the investigation into a release gate#

Before accepting a shard, ask five questions:

1. Can the exact consumer read and interpret this row shape? Reader support alone does not prove trainer compatibility.
2. Can each source be traced and excluded from a future corpus build? Keep stable example, source, conversation, and chunk identifiers plus provenance. Removing a source from the corpus does not itself untrain a released model.
3. Are splits isolated at the source or conversation level? Row-level IDs are not enough when one source yields many examples.
4. Are derived values tied to immutable revisions? Tokens need a tokenizer and template revision; embeddings need a model and source-text revision.
5. Can another person reproduce the decision? Retain input hashes, contract version, engine version, reviewed SQL and its hash, result counts, exceptions, reviewer, and timestamp.

Use fail for unambiguous contract violations, review for distribution or similarity signals, and pass only for the checks actually run. “Passed the audit” should never imply checks that were out of scope.

The browser is a good place for the first investigation, but practical limits depend on browser memory, file layout, and query shape. The same reviewed rules can be automated after their cost has been tested on the real corpus. Freeze candidate inputs, validate every shard, reconcile aggregate failures, review only the identifiers and values the failed rule requires, rerun after upstream correction, and retain the previous accepted snapshot. Never patch the evidence silently.

What this audit cannot prove#

Even a clean report cannot establish collection consent or legal fitness. It cannot prove truth, usefulness, representativeness, safety, annotator reliability, complete contamination detection, retrieval quality, or the behavior of a model trained on the data. Verify tokenization and masks in the training process. Evaluate embeddings on the retrieval task. SQL narrows uncertainty; it does not erase it.

Reproduce the checks#

The two fixtures are synthetic teaching data, not a benchmark or a representative corpus. Their token IDs and vector values do not belong to real models. The recorded evidence below comes from native DuckDB v1.5.3 on August 24, 2026; browser-engine compatibility was not part of that recorded run.

To rebuild the package, download the generator and both SQL bundles into one directory, run python -m pip install duckdb==1.5.3, then run python generate_llm_audit_fixtures.py. The fixtures and manifest are written beside the script.

Hash your downloads before comparing results. In viewparquet, the opened relation is data; in native DuckDB, replace it with read_parquet('path/to/file.parquet'). The manifest also records the evidence-file hashes, so a changed query or generator cannot silently pose as the same run.

If a result differs, include the page URL, fixture hash, DuckDB version, complete error, and safe reproduction steps on the Support page. Do not send credentials or sensitive data.

Primary references#

• Apache Parquet logical types (opens in a new tab) — representations for LIST, MAP, strings, timestamps, and other logical values.
• DuckDB Parquet overview (opens in a new tab) and metadata functions (opens in a new tab) — the schema, footer, pushdown, and multi-file operations used here.
• Hugging Face Datasets loading guide (opens in a new tab) and TRL dataset formats (opens in a new tab) — current Parquet ingestion and trainer-specific row shapes.
• NVIDIA NeMo DocumentDataset (opens in a new tab) — versioned sharded JSONL and Parquet curation workflows.
• Datasheets for Datasets (opens in a new tab) and Hugging Face dataset cards (opens in a new tab) — documentation practices for composition, provenance, intended use, and maintenance.

For a shorter operational checklist, use the Parquet FAQ audit guide.