Why Won’t This Parquet File Open? A Reproducible Failure Lab

Three files named results.parquet can fail at three different boundaries. One is HTML. One contains real column data but no footer because the writer never closed. One is valid but asks the browser for more memory than it has.

“It won’t open” is a symptom. Preserve the bytes, then find the first operation that fails: retrieval, footer read, schema bind, value decode, or materialization.

This guide uses a downloadable failure lab so each boundary is backed by exact bytes and observed reader output.

Preserve the failed file before changing anything#

Keep the exact bytes that failed. A second download, a rewritten file, or an automatic “repair” can erase the evidence you need.

Record:

• byte size and SHA-256;
• first and last 16 bytes;
• source URL, object version, ETag, or local path as appropriate;
• writer and version, if known;
• reader, version, operation, and complete original error;
• whether the failure happened during metadata inspection, a value scan, or result rendering.

On macOS or Linux, these four commands establish a useful baseline without trying to parse the file:

wc -c candidate.parquet
shasum -a 256 candidate.parquet
xxd -l 16 candidate.parquet
tail -c 16 candidate.parquet | xxd

Do not paste a presigned URL or credential-bearing command into a shared incident log. Record an immutable object version or a redacted source description instead.

An error is a clue, not a verdict. “Magic bytes not found” can mean the writer never completed a real Parquet file. It can also mean the downloaded bytes were HTML, XML, text, or a Git LFS pointer and were never Parquet at all.

If it came from a URL, prove what arrived#

A browser can fail even when the same URL works in a command-line tool. Before debugging Parquet, separate a file problem from a retrieval problem. Cross-origin resource sharing (CORS) rules can prevent browser code from reading a response that a command-line client can read.

Download the object to a fresh path while preserving HTTP status, redirects, final URL, response headers, and body size. Then try that exact local copy in a native reader.

• If the local copy also fails, continue with the byte and footer checks below.
• If the local copy opens but the browser URL fails, investigate CORS, authorization, redirects, range handling, TLS, and browser memory.
• If the response body begins with HTML or XML, inspect it as an error response rather than adding PAR1 or renaming it again.

For a remote Parquet read, 206 Partial Content plus a correct Content-Range proves that a requested range was honored. A 200 OK response to a ranged request is not automatically corrupt, but it may mean the host returned the whole object. A 416 response means the requested range was unsatisfiable. Those outcomes change transfer cost and browser feasibility before any page decoding occurs.

The private S3 guide covers CORS, 403, 404, expired signatures, provider endpoints, and range evidence in detail. DuckDB’s HTTP(S) documentation (opens in a new tab) describes its remote filesystem behavior.

A filename does not prove the bytes are Parquet#

The table reports the first check that failed in native DuckDB v1.5.3. Exact messages vary by reader and version.

The manifest keeps the exact sizes, hashes, and boundary bytes.

Ordinary Parquet files begin with PAR1 and end with serialized file metadata, a four-byte little-endian footer length, and PAR1. The official file layout (opens in a new tab) explains why readers start at the end. Encrypted-footer mode is a deliberate exception: the modular-encryption layout (opens in a new tab) uses PARE so a legacy reader can tell that it cannot parse the footer.

Boundary bytes are a screening test, not a clean bill of health. The corrupted-page fixture has the same size and both PAR1 markers as its control. Conversely, the encrypted fixture’s PARE markers describe a valid file, not damage.

The repairs for wrong bytes are upstream: fetch the actual Git LFS object, stop following a redirect to a login page, correct the storage endpoint, or download the intended object version. Renaming text to .parquet, or manually adding magic bytes, does not convert or repair it.

The footer is written last#

Parquet writers can emit column data in a forward pass because file metadata is written after the data. That design is efficient, but it means a file may contain a substantial amount of genuine Parquet-encoded data and still be unreadable until the writer closes and publishes the footer.

We captured the unclosed-writer fixture before closing the writer. PyArrow wrote one row group, the generator flushed and copied 1,120 in-progress bytes, and only then closed the temporary source. The copied fixture starts with PAR1 but has no final metadata or magic.

Try footer-level checks before scanning values:

SELECT *
FROM parquet_file_metadata('candidate.parquet');
 
SELECT *
FROM parquet_schema('candidate.parquet');

If those succeed, the footer is parseable. They do not prove that every data page is readable. Even this query may use footer metadata rather than decoding column values:

SELECT count(*)
FROM read_parquet('candidate.parquet');

For a small candidate, force the columns you care about to decode. The fixture uses a known string column:

SELECT sum(length(payload)) AS payload_bytes
FROM read_parquet('candidate.parquet');

A broader validation can copy all selected rows to a separate temporary file, then inspect that output. Do not overwrite the failed source.

If the footer is missing, close and republish the writer, download the complete object again, regenerate it, or restore an intact version. The Parquet error-recovery notes (opens in a new tab) explain the hard boundary: a normal reader cannot recover a file without its metadata.

Footer reconstruction and partial-row salvage are separate forensic jobs. Treat recovered output as a new artifact, and validate it independently.

A readable footer does not prove readable pages#

The most instructive fixture is not the obviously broken one.

The valid control contains 5,000 rows, GZIP-compressed Data Page v2 pages, and page checksums. Its SHA-256 is 3a3cc6591562bd909eb0384c77a4028b0a7d0ace1d4657e514cfe89d625748be. To create the damaged copy, the generator locates the first GZIP header inside the payload column, changes one byte from 0x1f to 0x1e, and leaves the footer, page checksum, length, and boundary markers untouched.

Native DuckDB still reports the two-column schema and answers COUNT(*) = 5000. Only the value scan reaches the damaged page:

Input is invalid/unsupported GZIP stream

PyArrow 18.1.0 with page_checksum_verification=True identifies the page boundary more directly:

could not verify page integrity, CRC checksum verification failed for page_ordinal 0

Parquet page checksums are optional. When present, they help identify which page was damaged, but they do not make the reader repair it. The checksum specification (opens in a new tab) describes the cyclic redundancy check (CRC) field and the trade-off: checksum verification detects damage at read time and costs additional CPU.

The practical lesson is simple: schema inspection, boundary markers, and row counts answer different questions. A validation that must prove values are readable needs a value-decoding step.

For genuine internal corruption, restore or recompute from an authoritative source. If a recovery tool manages to salvage some row groups, publish the result under a new identity with rejected ranges and reconciliation counts; do not silently replace the original and call it repaired.

A directory may be a dataset, not one file#

If the path ends at a directory, you may have a Parquet dataset rather than one file. Spark and Hive commonly write part-*.parquet files plus markers such as _SUCCESS. A single-file reader cannot treat the directory or marker as one Parquet file.

There is a second dataset failure: every file opens alone, but the collection does not bind under the reader’s schema policy. Inspect the members before assuming corruption:

SELECT
  file_name,
  name,
  type,
  logical_type
FROM parquet_schema('candidate/*.parquet')
WHERE name IS NOT NULL
ORDER BY file_name, name;

Spark’s generic file-source options (opens in a new tab) distinguish corrupt-file handling, while its Parquet guide (opens in a new tab) documents schema merging. DuckDB’s union_by_name can align columns and fill missing values with nulls. Those are reader policies, not proof that a schema change was intended.

Select only data files, maintain an explicit manifest, and reconcile compatible schema changes under a versioned contract. Rewrite incompatible members rather than depending on file order or silent coercion. ViewParquet currently opens one local source file at a time; it does not load a local folder as a reconciled multi-file dataset.

Reader support can look like corruption#

The writer and reader may disagree even when the bytes are intact.

The Parquet compatibility documentation (opens in a new tab) makes an important distinction. An older reader may ignore an unfamiliar logical annotation and expose the underlying physical value, while an unsupported encoding or page format can make value decoding impossible. Encryption adds another requirement: implementation support plus the correct keys.

The encrypted lab fixture is useful because it is deliberately valid. It uses encrypted-footer mode and two encrypted columns. Without keys, native DuckDB v1.5.3 reports:

File 'encrypted-footer.parquet' is encrypted, but 'encryption_config' was not set

Without decryption properties, PyArrow reports that encrypted metadata cannot be read. Recreating the fixture’s synthetic test key-management service (KMS) and supplying the matching decryption properties returns all four original rows. The generator publishes those keys so the example is reproducible; its key wrapper is deliberately insecure and must never be copied into production. Encrypted bytes also change on regeneration because data keys and nonces are random, so the manifest records the hash of the published copy.

Do not diagnose every reader mismatch as a missing Snappy, ZSTD, Brotli, or LZ4 codec. Current DuckDB supports the common Parquet codecs listed in its Parquet overview (opens in a new tab). Check the exact writer feature, encoding, encryption mode, and reader version. Then upgrade the reader, provide the required key configuration, or rewrite through a reader that can decode the source. Editing PARE into PAR1 is not decryption.

A valid file can still exceed the environment#

Metadata can open while the requested work fails later. In a browser, distinguish:

1. reading the footer;
2. binding the schema;
3. decoding selected columns and row groups;
4. building query intermediates;
5. rendering or exporting the result.

DuckDB-WASM documents a WebAssembly memory ceiling of 4 GB, with browsers potentially imposing a lower practical limit. An engineering note added on June 12, 2026 records one ViewParquet allocation failure at 3.1 GiB used. That is one operational observation, not a universal file-size limit. It also says nothing about whether the Parquet bytes were valid.

Project only the columns needed, filter before a wide materialization, avoid using SELECT * as a universal validation step, and test metadata separately from a full decode. Move a browser-unsuitable scan, sort, join, or export to native DuckDB. The large-file article explains how row groups, codecs, query shape, and view-based loading affect the working set. DuckDB’s WASM limitations (opens in a new tab) describe the environment boundary.

Can you append to an existing Parquet file?#

With the conventional PyArrow APIs, you cannot reopen a closed Parquet file and append rows in place. Append is related to the unclosed-writer case, but it is not a general repair for a finished file.

A ParquetWriter can accept additional tables or row groups while it remains open. Closing it writes the final metadata. The conventional PyArrow workflow does not reopen an already closed Parquet file and extend its footer in place. Instead, write a new, uniquely named file into a dataset and compact or publish the dataset under a controlled policy. The PyArrow writer API (opens in a new tab) and dataset writer (opens in a new tab) document those two levels.

If a process crashes while it is “appending” through an open writer, the result may look like the 1,120-byte lab fixture: real page data at the beginning, no usable footer at the end. Finish the writer or republish from the source. Do not append more bytes to an incomplete artifact and assume the metadata can be guessed safely.

Use the error to choose the next test#

Exact wording changes across libraries and versions. Use the message to choose a test, not to skip the investigation.

Our telemetry cannot rank these causes yet. As of August 26, 2026, most parser and decoder error types fall into the broad, non-identifying load_error category. ViewParquet does not transmit raw filenames, URLs, values, SQL, credentials, or error text with that event. This article explains the failure families and how to separate them. It does not claim which one is most common among ViewParquet users.

Download and reproduce the lab#

The fixtures were generated on August 25, 2026 and checked with native DuckDB v1.5.3 and PyArrow 18.1.0. Repository tests repeat the important success and failure boundaries with the shipped DuckDB-WASM engine, which reports v1.4.3.

The machine-readable manifest records every fixture’s full SHA-256, size, first and last 16 bytes, native DuckDB result, PyArrow result, construction recipe, versions, and limitations. The complete generator creates all nine files, including the genuinely pre-close writer bytes, checksummed page corruption, and encrypted round trip.

After downloading a fixture, verify its hash against the manifest. In native DuckDB, separate metadata, row count, and value decoding:

SELECT * FROM parquet_file_metadata('fixture.parquet');
SELECT count(*) FROM read_parquet('fixture.parquet');
SELECT sum(length(payload)) FROM read_parquet('fixture.parquet');

The third query applies to the control and damaged-page fixtures; use a column that exists in another candidate. When trying a local fixture in ViewParquet, the opened relation is data, so the equivalent value scan is SELECT sum(length(payload)) FROM data.

The lab is not a benchmark, a representative corpus, or a complete reader-compatibility matrix. It does not simulate remote authentication, CORS, range behavior, mixed datasets, or memory limits. Test those against the actual endpoint, reader, and environment.

The repair belongs to the failed boundary#

• Wrong bytes: reacquire the intended object.
• Incomplete footer: close, regenerate, re-download, or restore an intact version.
• Dataset selection or schema mismatch: select real data members and reconcile under an explicit contract.
• Reader incompatibility or encryption: upgrade, configure keys, or rewrite through a capable reader.
• Page corruption: restore or recompute; treat salvage as a new verified artifact.
• Memory failure: reduce the work, change the layout after measurement, or move to a native environment.

A reliable Parquet diagnosis starts at the first boundary that lacks evidence. Preserve the bytes that failed there, then apply the repair that belongs to that boundary.

Primary references#

• Apache Parquet file layout (opens in a new tab)
• Apache Parquet page checksums (opens in a new tab)
• Apache Parquet modular encryption (opens in a new tab)
• Apache Parquet format compatibility (opens in a new tab)
• DuckDB Parquet reader and metadata functions (opens in a new tab)
• DuckDB-WASM limitations (opens in a new tab)
• PyArrow ParquetWriter (opens in a new tab)
• PyArrow Dataset writer (opens in a new tab)