Opening Large Parquet Files in the Browser: How View-Based Loading Helps

You found the dataset you need. It's a Parquet file — maybe 500 MB, maybe 3 GB. You drag it into a viewer, the progress bar crawls, the fan spins up, and then: a frozen tab, a cryptic out-of-memory error, or a browser crash. So you give up and write yet another throwaway Python script just to peek at twenty rows.

We've been there, and so have our users. For a while, viewparquet had the same ceiling every in-browser tool hits: files that were perfectly reasonable on disk would blow past the browser's memory limit the moment you opened them. We recently changed the loading path so opening a Parquet file no longer requires materializing the whole dataset up front. That removes one common failure mode, while practical limits still depend on the browser, DuckDB-WASM, the file's layout and codecs, and the work a query asks it to do.

This post explains, in plain terms first and engineering terms second, why large Parquet files can overwhelm browser tools, how view-based loading reduces upfront memory pressure, and where browser limits still apply.

Compressed file size is not decoded memory use#

Here's the part most people never see: Parquet is a compressed, encoded format. The number you see in your file manager is the size after compression codecs like Snappy or ZSTD have done their work, and after dictionary encoding has replaced repeated strings with small integers.

When a tool decodes a Parquet file into in-memory values, the working set can be substantially larger than the compressed file. The ratio depends on column types, cardinality, encodings, codecs, the query, and the engine's in-memory representation; there is no reliable multiplier from file size alone.

That is why compressed file size alone cannot predict whether a browser query will fit in memory.

Browser and WebAssembly memory budgets#

For local files, viewparquet runs DuckDB-WASM inside the browser and does not upload file contents to viewparquet for viewing or SQL. Remote sources connect to their configured host. Optional AI and site telemetry make separate, disclosed network requests. AI sends your messages and configured context to the chosen provider; chat tools and saved AI context may also provide structural metadata and SQL.

DuckDB-WASM still operates within memory limits imposed by its build, the browser, the device, and other open tabs. Those limits vary: a nominal address-space maximum is not the same as memory the browser will actually make available. viewparquet also cannot assume the same spill-to-disk behavior as a native database, so memory-heavy operations can still fail.

Put those two facts together and the old failure mode is obvious:

1. You open a compressed Parquet file.
2. The old viewer decodes the entire file into an in-memory table, and the working set grows beyond the compressed size.
3. Available browser or WebAssembly memory can run out before the decode finishes.
4. Crash. Frozen tab. "Out of Memory Error."

This is exactly the error our monitoring caught real users hitting: failed to allocate data of size 32.0 KiB (3.1 GiB/3.1 GiB used). The database wanted 32 more kilobytes and there were none left. That's a brutal experience for someone who just wanted to look at their data — and it is the failure mode we set out to reduce.

The change: avoid the upfront copy#

The old pipeline did the obvious thing, which turned out to be the wrong thing:

-- Old approach: decode EVERYTHING into memory up front
CREATE TABLE data AS SELECT * FROM read_parquet('your_file.parquet');

That one statement forces the entire file — every row, every column — to be decompressed and held in RAM, whether you ever look at it or not. All cost is paid up front, and for big files the bill exceeds the budget.

The new pipeline avoids intentionally materializing the full table up front:

-- New approach: a deferred view over the file
CREATE VIEW data AS SELECT * FROM read_parquet('your_file.parquet');

Creating a view does not materialize the rows. It saves a query over the registered file, so DuckDB can request the footer and relevant row groups and columns as queries execute. Exactly how much it reads or decodes depends on row-group statistics, selected columns, filters, file layout, codecs, and the query plan; a full scan still reads the data it needs.

When you open a file now, the initial grid query requests a limited page instead of intentionally materializing the full table. DuckDB may still need to read and decode metadata plus one or more row groups, depending on their size, encodings, and what the query can prune. Scrolling or running SQL triggers additional work. Memory therefore follows the active query, decoded chunks, and result more closely than a full decoded copy, but expensive queries or poorly laid-out files can still consume substantial memory.

Why Parquet makes this possible#

This pattern is especially useful for Parquet because its structure can support selective reads, although the savings depend on how the file and query line up. Three design decisions in the format do the heavy lifting:

• Row groups: A Parquet file is split into independent horizontal chunks, each decodable on its own. A limited query can target relevant row groups instead of deliberately materializing every row group, although row-group size still affects the work.
• Column chunks: Within each row group, every column is stored separately. Projection pushdown can often skip unreferenced column chunks, subject to the query plan and reader behavior.
• Footer metadata: The file ends with an index describing row groups, row counts, byte offsets, and available column statistics. DuckDB can use that metadata for planning and may skip row groups whose statistics rule them out, but missing or unhelpful statistics limit predicate pushdown.

-- What the grid actually runs when you open a big file:
SELECT * FROM data LIMIT 100 OFFSET 0;
-- Work depends on row-group layout, selected columns, codecs, and the query plan.
 
-- A row count can often use Parquet metadata without decoding every value:
SELECT COUNT(*) FROM data;

CSV, by contrast, has none of this. There's no index, no row groups, no column separation — you can't know where row 5,000,000 starts without reading everything before it. That's why CSV files still get fully parsed on load (and why, if you work with big data, Parquet is the format worth standardizing on).

What changed, in one table#

We verified that the new path creates a view rather than a hidden materialized copy, returns initial rows through a limited query, and can run search and custom SQL against the registered file. Time, transfer, and peak memory still vary with the browser, device, file layout, codecs, and query.

What this means for you, practically#

• Avoid one common opening-time crash. Files that failed during full upfront materialization may now open because the initial view does less work. File size alone does not determine success, and there is no guaranteed maximum.
• First rows can appear sooner. The grid no longer waits for an intentional full-table copy, but startup time still depends on metadata, row groups, codecs, storage speed, and browser resources.
• Search and filter the registered file. Column and row-group pruning can reduce work when the query and statistics support it. A full scan, large result, sort, or aggregation can still be expensive or exceed the available memory.
• Local-file contents stay on the local processing path. Opening a local file through this path does not upload its contents to viewparquet; DuckDB-WASM reads it inside the browser tab.

Honest limits (because every engineering choice has them)#

View-based loading usually makes memory follow the active query and result instead of forcing a full copy up front, so a query that needs substantial state or returns a large result can still be heavy. Two examples worth knowing:

• Sorting an enormous result set requires holding the candidates in memory. A limited grid sort may be less demanding; a full-file sort or a large sorted export can still exceed the available memory.
• CSV and JSON files are still decoded up front, because those formats don't support random access. Very large CSVs can still exhaust available memory. Converting them to Parquet can enable selective reads, but the resulting file and queries remain subject to browser and WebAssembly limits.

We'd rather tell you where the edges are than pretend there are none.

Tips for working with very large files#

1. Prefer Parquet over CSV for larger analytical datasets when your tools can use its columnar layout and metadata; it is generally smaller and better suited to selective queries.
2. Select the columns you need in SQL instead of SELECT * — fewer column chunks read, faster results.
3. Filter early. A WHERE clause on a column with natural ordering (dates, IDs) lets row-group statistics skip huge swaths of the file.
4. Use LIMIT while exploring. You rarely need a million rows to understand a dataset.
5. Choose compression and row-group sizing for the workload. ZSTD and moderately sized row groups are useful starting points, but the best values depend on query patterns, network latency, and reader support.

The bigger picture#

"Open a big file in the browser" sounds like a small feature. Under the hood it follows a broader architecture shift: avoid unnecessary copies, bring the query engine closer to the data, and use metadata and selective reads where the format, layout, and query permit. Parquet was designed for exactly this, DuckDB executes it beautifully, and your browser turns out to be a perfectly good place for both.

When someone hands you a large Parquet file, try a limited projection and row limit in viewparquet first and watch the browser's memory use. View-based loading may be enough for inspection; a workload that requires a full scan, large materialized result, or heavy sort may still be better suited to a native tool.

Primary references#

• Apache Parquet concepts: row groups and column chunks (opens in a new tab)
• DuckDB Parquet scans, projection pushdown, and filter pushdown (opens in a new tab)
• DuckDB-Wasm limitations (opens in a new tab)

For a controlled input with exact row counts and hashes, use the verified public Parquet samples.