Convert CSV to Parquet and Check the Schema Before You Trust It

CSV is an exchange format. Parquet is an analytical storage format. Moving from one to the other can reduce file size and speed up queries, but the conversion also introduces something CSV does not have: a schema.

That schema is the part to inspect. A converter may see 00127 and infer an integer, see an empty column and infer the wrong type, or interpret a date differently from the system that produced it. The Parquet file can be perfectly valid while the data inside it no longer means what you intended.

When Parquet is a better fit

Apache Parquet stores data by column and supports efficient encoding and compression. Query engines can read only the columns needed for a calculation instead of scanning every field in every row.

Parquet is usually a good choice when:

• the data will be queried in DuckDB, Spark, Python, R, or a warehouse;
• a large table is read repeatedly;
• analysts often select a subset of columns;
• you need explicit data types; or
• storage and transfer size matter.

CSV remains useful for manual inspection, broad compatibility, small exports, and systems that expect plain text. Parquet is binary, so it is not meant to be opened in a text editor.

Prepare the CSV first

Run CSV Diagnostic when the delimiter, encoding, or row structure is uncertain. Fix structural problems before introducing a schema.

Pay particular attention to:

• duplicate or blank headings;
• rows with the wrong number of fields;
• mixed date formats;
• decimal and thousands separators;
• placeholder values such as N/A and -; and
• identifiers made entirely of digits.

Consider this file:

facility_code,report_date,admissions,occupancy_rate
0012,2026-07-31,146,0.82
0013,2026-07-31,,0.75

facility_code should remain text because its leading zeros carry meaning. admissions may be an integer with a null value in the second row. occupancy_rate may be a decimal or floating-point value, depending on the precision required.

Convert the file in HappyCSV

Open CSV to Parquet, select the CSV, and review the detected columns. Processing runs locally in a browser worker; the source file is not intentionally uploaded to a HappyCSV server.

Download the Parquet file and keep the original CSV until validation is complete. Conversion should create a new artefact, not replace the only copy of the source.

Why type inference can go wrong

A CSV cell is text. Parquet supports physical and logical types. The converter must therefore infer or assign a type for each column.

Common mistakes include:

Identifiers interpreted as numbers

Postal codes, account numbers, case IDs, telephone numbers, and SKUs may contain only digits. They are still identifiers. Numeric conversion can remove leading zeros or exceed a safe integer range.

Mixed columns reduced to text

A mostly numeric column containing one value such as unknown may become a string column. That preserves the raw values but prevents direct numerical aggregation.

Dates interpreted inconsistently

01/02/2026 has no universal meaning. Normalize dates before conversion, preferably to an unambiguous format, and decide whether a field represents a date, local date-time, or instant in UTC.

Empty values confused with empty strings

Parquet can represent null values. An empty string is a real string with zero characters. Decide which state the source intends.

Validate the resulting schema

DuckDB can inspect the file without loading it into a separate database:

DESCRIBE SELECT * FROM 'output.parquet';

Then inspect representative rows:

SELECT *
FROM 'output.parquet'
LIMIT 20;

Check row count and null counts:

SELECT
  count(*) AS rows,
  count(*) - count(admissions) AS missing_admissions
FROM 'output.parquet';

In Python with PyArrow:

import pyarrow.parquet as pq

table = pq.read_table("output.parquet")
print(table.schema)
print(table.num_rows)

Compare the results with the source. At minimum, verify:

1. row count;
2. column names and order;
3. inferred types;
4. leading-zero identifiers;
5. null and empty-string behaviour;
6. minimum and maximum dates; and
7. a small sample of exact values.

Compression and file size

Parquet's columnar layout groups similar values together, making compression effective. The result is often much smaller than CSV, especially for repetitive data. There is no honest universal compression ratio: it depends on cardinality, column types, encoding, compression codec, and row-group layout.

A tiny CSV can produce a larger Parquet file because Parquet includes schema and metadata. The format pays off more clearly with larger analytical datasets.

The Parquet file-format documentation explains its row groups, column chunks, pages, and metadata. Those details matter when tuning large production datasets, but they are not prerequisites for a careful one-file conversion.

Converting Parquet back to CSV

Use Parquet to CSV when a spreadsheet or import process requires text. Expect some information to become less explicit: CSV does not carry Parquet's types or schema metadata.

Dates, nulls, binary values, nested structures, and high-precision numbers need particular attention. A round trip from CSV to Parquet and back is not guaranteed to reproduce the original bytes, even when the table still looks similar.

A sensible hand-off checklist

Before publishing or delivering the file, record:

• the source filename and date;
• the conversion tool and version or date used;
• the expected schema;
• the number of rows written and skipped;
• known transformations; and
• the validation query or script.

Parquet is valuable because it gives analytical systems structure. That same structure makes silent assumptions more consequential. Review the schema once, and the faster queries that follow will be based on data you can defend.