Building Reliable Parquet Pipelines: Layout, Schema, and Compaction

A scalable Parquet pipeline is not “CSV plus compression.” It is a set of explicit contracts for files, partitions, row groups, schema, publication, and recovery. The right layout comes from the queries and engines that will read the data; there is no universal codec, file size, or row-group number that makes every workload fast.

This guide replaces anecdotal performance claims with a reproducible benchmark pattern and a promotion checklist. For AI-specific handoffs, use the separate contract and lineage guide. For browser constraints, see how view-based loading changes large-file behavior.

Start with the read workload#

Write down the important reads before designing the writer:

A page that says “partition by date” without naming the query pattern is incomplete. Hourly partitions can be appropriate for a very high-volume stream and disastrous for a small daily feed.

Treat an object path as a location, not an identity#

Prefer immutable run- or version-scoped objects:

s3://analytics/events/
  dataset_version=2026-08-21T09-30-00Z/
    event_date=2026-08-20/
      part-00000.parquet
      part-00001.parquet
    manifest.json

The manifest or catalog pointer should record:

• exact object keys and version IDs/hashes;
• contract version;
• producer code/dependency revision;
• input identities;
• row and rejection counts;
• min/max event times;
• schema fingerprint;
• writer settings;
• creation and acceptance times.

Publishing the manifest/catalog pointer is the commit. Copying files into a “final” prefix one by one can expose a partially written dataset, and object-store rename is commonly copy plus delete rather than an atomic filesystem rename.

Partition by bounded, useful predicates#

Good partition fields often have low-to-moderate cardinality and appear in common filters: event date, region, tenant under a controlled count, or train/eval split.

Avoid partitioning by a near-unique key:

# Usually harmful: one directory/file family per user.
user_id=8a4f.../part.parquet

Prefer retaining that key as a column with statistics or another purpose-built index. If most queries filter by event_date and region:

event_date=2026-08-20/region=eu/part-00000.parquet

Test whether both levels actually prune. Extra directory levels and tiny files add planning and object-listing work even when their names look organized.

Understand row groups in the writer you use#

A Parquet file contains row groups; each row group has one column chunk per column. Readers may skip row groups using statistics and may avoid unselected column chunks. Larger row groups can improve sequential I/O and compression but increase writer buffering and reduce pruning granularity.

The Apache Parquet configuration guide (opens in a new tab) discusses 512 MB–1 GB row groups in an HDFS-oriented sequential-scan setup. That is context, not a universal browser, object-storage, or interactive-query recommendation. DuckDB’s ROW_GROUP_SIZE setting is expressed as a minimum number of rows, not bytes, and its defaults belong to DuckDB. PyArrow exposes different writer options.

Record both rows and actual compressed/uncompressed bytes per row group, then benchmark the target engine.

Run a codec benchmark instead of copying a table#

This native DuckDB script generates a deterministic one-million-row input and writes two files with the same row-group setting:

CREATE OR REPLACE TABLE benchmark_input AS
SELECT
  i AS event_id,
  TIMESTAMP '2026-01-01'
    + INTERVAL (i % 86400) SECOND AS event_time,
  i % 1000 AS account_id,
  repeat(md5(i::VARCHAR), 4) AS payload
FROM range(1000000) AS t(i);
 
COPY benchmark_input
TO 'benchmark-snappy.parquet'
(FORMAT PARQUET, COMPRESSION SNAPPY, ROW_GROUP_SIZE 100000);
 
COPY benchmark_input
TO 'benchmark-zstd.parquet'
(FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 100000);
 
SELECT
  filename,
  count(*) AS rows
FROM read_parquet([
  'benchmark-snappy.parquet',
  'benchmark-zstd.parquet'
])
GROUP BY filename
ORDER BY filename;

Expected result: each filename reports exactly 1,000,000 rows. That is a correctness check, not a performance winner.

Inspect what the writer produced:

SELECT
  file_name,
  row_group_id,
  compression,
  count(*) AS column_chunks,
  sum(total_compressed_size) AS compressed_bytes,
  sum(total_uncompressed_size) AS uncompressed_bytes
FROM parquet_metadata([
  'benchmark-snappy.parquet',
  'benchmark-zstd.parquet'
])
GROUP BY file_name, row_group_id, compression
ORDER BY file_name, row_group_id, compression;

Then time at least:

-- Full scan
SELECT sum(account_id)
FROM read_parquet('benchmark-snappy.parquet');
 
-- Narrow time filter
SELECT count(*)
FROM read_parquet('benchmark-snappy.parquet')
WHERE event_time >= TIMESTAMP '2026-01-01 12:00:00'
  AND event_time <  TIMESTAMP '2026-01-01 13:00:00';
 
-- Wide materialization
SELECT *
FROM read_parquet('benchmark-snappy.parquet')
WHERE account_id = 42;

Repeat for Zstandard, restart or control caches, and capture median plus spread rather than one run. Record DuckDB version, CPU, memory, storage, file sizes, query plans, selected columns, and rows returned. A smaller file can decode more slowly; a faster local scan can make more remote requests. Choose from the real workload.

File size is a system property#

Target files large enough to avoid a small-file planning/request storm and small enough for useful parallelism, retries, rewrites, and interactive access. The answer depends on:

• object-store request latency and cost;
• executor count and memory;
• file/row-group statistics;
• compression ratio and codec;
• expected partition volume;
• update/late-data policy;
• browser or desktop consumers;
• failure and retry boundaries.

Publish the observed distribution: file count, p50/p95 bytes, rows per file, row groups per file, and empty/tiny-file count. “About 256 MB” is not an acceptance criterion unless the workload test explains why.

Compact with a publish-then-retire protocol#

Do not delete source parts inside the same loop that is writing the compacted output. A safer compaction job:

1. Resolve and record the exact input object versions.
2. Read them into a run-scoped temporary output prefix.
3. Write compacted files without exposing them as current.
4. Verify source and output row counts under the deduplication policy.
5. Compare contract/schema, null counts, key counts, and partition bounds.
6. Re-read representative output rows and run the workload smoke queries.
7. Hash or version the output objects.
8. Publish a new manifest/catalog snapshot.
9. Keep old inputs for a defined rollback period.
10. Let a separate retention job expire unreferenced objects.

If compaction intentionally removes duplicates or invalid rows, record rejected IDs/counts and make the expected delta explicit. “Input rows equal output rows” is then the wrong invariant; the approved reconciliation equation is the right one.

Parquet does not manage schema evolution for you#

A directory of Parquet files can contain different schemas. What happens next depends on the reader:

-- Strict/default behavior may fail or follow one inferred schema.
SELECT *
FROM read_parquet('events/*.parquet');
 
-- DuckDB can align columns by name and fill missing values with NULL.
SELECT *
FROM read_parquet(
  'events/*.parquet',
  union_by_name = true
);

The DuckDB Parquet documentation (opens in a new tab) describes union_by_name. It is useful for investigation and controlled compatible additions; it is not a governance mechanism.

Adopt a compatibility matrix:

A table format such as Apache Iceberg adds schema IDs, snapshots, and defined evolution rules around Parquet data files. Those guarantees come from the table metadata. The Iceberg schema documentation (opens in a new tab) explains its durable field IDs.

Validate schema and statistics in CI#

At minimum, CI or the publishing job should run:

DESCRIBE SELECT *
FROM read_parquet('candidate/*.parquet');
 
SELECT
  file_name,
  name,
  type,
  logical_type
FROM parquet_schema('candidate/*.parquet')
WHERE name IS NOT NULL
ORDER BY name, file_name;
 
SELECT
  filename,
  count(*) AS rows
FROM read_parquet('candidate/*.parquet')
GROUP BY filename
ORDER BY filename;

Add contract-specific checks for required values, unique/composite keys, ranges, enumerations, timestamps, and joins. Store the SQL and result summary with the manifest. Do not call an undefined helper such as can_cast_safely from an article and assume every engine shares the same cast rules; enumerate approved transitions.

Remote reads need observable checks#

For S3-compatible storage, use DevTools or engine metrics to verify request behavior rather than claiming that Parquet always reads “only what it needs.” DuckDB can use Parquet metadata and HTTP ranges for partial reads, but actual transfer depends on the query, projection, statistics, row-group layout, and host.

The private S3 guide shows how to distinguish 206 Partial Content, 200 full responses, CORS, and authorization failures. Pin object versions for repeatable remote queries when the storage/engine supports it.

Cloud-agnostic means adapters plus tests#

A script that instantiates only S3FileSystem is not cloud-agnostic. Separate:

• dataset paths and partition contract;
• filesystem/authentication adapter;
• writer configuration;
• atomic publication/catalog action;
• observability;
• integration tests per provider.

Apache Arrow’s Dataset API (opens in a new tab) provides common dataset abstractions and partitioned writes, but credentials, consistency, multipart upload, conditional writes, CORS, and object versioning remain provider concerns.

Operating metrics#

Track trends, not just job success:

• input/output/rejected rows;
• bytes and rows per file/row group;
• file count per partition;
• schema and contract version;
• null/duplicate/range failures;
• compression ratio and write/read time;
• object requests and bytes transferred for representative queries;
• late-data and retry counts;
• manifest age and rollback target;
• consumer compatibility test status.

Alert on distributions and invariants that matter to the workload. An individual small file may be valid; a sudden tenfold increase in tiny files is the operational signal.

Release checklist#

• Queries and consumers are documented before layout choices.
• Output paths are immutable and run-scoped.
• A manifest records inputs, outputs, hashes/versions, contract, and counts.
• Partition fields have bounded cardinality and demonstrated pruning value.
• File and row-group settings were measured in the target engine.
• Codec choice comes from the same-input benchmark.
• Schema drift is classified, not silently masked.
• Compaction publishes before old inputs are retired.
• Cloud adapters have provider-specific integration tests.
• Rollback points to a previous accepted snapshot.
• Documentation links to the benchmark artifacts and current primary sources.

Use the open-file task page to inspect a candidate locally, but run recurring acceptance gates in CI/native DuckDB.

Primary references#

• Apache Parquet concepts: files, row groups, column chunks, pages (opens in a new tab)
• Apache Parquet configuration guidance (opens in a new tab)
• DuckDB reading and writing Parquet (opens in a new tab)
• DuckDB Parquet metadata functions (opens in a new tab)
• Apache Arrow Dataset API (opens in a new tab)
• Apache Iceberg schemas (opens in a new tab)