What Is a BOM in CSV Files? (Byte Order Mark)

Open a CSV in a hex viewer and the file sometimes starts with three strange bytes: EF BB BF. That is a BOM — a Byte Order Mark — an invisible signature saying "this file is UTF-8." Whether it helps or breaks your workflow depends entirely on what reads the file next.

The good: Excel wants it

Excel assumes plain ASCII unless told otherwise. A UTF-8 CSV without BOM opens with José as José. With BOM, Excel detects UTF-8 and renders accents correctly. This is why Windows exports from Salesforce, Shopify, and countless enterprise tools include one — see our encoding fixes for imports.

The bad: everything else

• Scripts: Python's naive open() keeps \ufeff attached to the first header, so df.columns[0] becomes \ufeffid and lookups fail mysteriously.
• Databases: some loaders refuse files that do not start with an expected token.
• JSON converters: a BOM before { is invalid JSON to strict parsers.
• String comparisons: "id" !== "\ufeffid" — equality checks fail while printing looks fine.

Which do you have?

Run CSV Diagnostic on the file. The report states "UTF-8 with BOM" versus plain text, alongside row-width issues and duplicate headers. If the first column name displays oddly in any tool you use, suspect the BOM first.

Removing (or keeping) it

Keep the BOM when Excel is the consumer. Strip it when scripts, databases, or JSON tools are next in line. Fix Encoding re-emits clean UTF-8, and Repair CSV output drops the mark while normalizing quoting.

Rule of thumb worth memorizing: BOM for humans opening Excel; no BOM for machines parsing files.

Seeing it yourself

Any hex editor shows the signature, but you do not need one:

• macOS/Linux: head -c 3 file.csv | xxd prints efbb bf when a BOM exists.
• Python: open('file.csv', encoding='utf-8-sig') strips it transparently; plain 'utf-8' leaves \ufeff on the first header.
• Notepad++: the status bar reads "UTF-8-BOM" versus "UTF-8."

The reason utf-8-sig exists in Python is exactly this format war — read with the sig variant when humans produced the file, plain utf-8 when machines did.

How common tools treat the BOM

Consumer	BOM behavior
Excel (double-click)	Required for correct UTF-8 accents
Python pandas (read_csv)	Strips automatically since v0.19+
Node fs.readFile + split	Keeps \ufeff glued to first header
Strict JSON parsers	Reject files starting with BOM
PostgreSQL COPY	Historically embeds it in the first column name

That table explains the classic support ticket: the file looks perfect in Excel, then a script fails comparing "id" to a value that prints identically but carries invisible bytes.

Prevention beats repair

Standardize at the boundary where files enter your pipeline:

1. Run unknown files through CSV Diagnostic — the report names the encoding variant explicitly.
2. Decide the consumer first. Human + Excel → keep the BOM. Scripts, loaders, APIs → strip it.
3. Convert once with Fix Encoding, then treat the output as canonical.

If you control the exporting system, configure it directly: most enterprise tools have an "include byte order mark" checkbox buried in export options, and unchecking it removes an entire class of downstream bugs.

FAQ

How do I tell BOM problems apart from encoding problems? BOM damage hits only the first column name; encoding damage mangles every accented value. One weird \ufeffid versus a whole file of José — different fixes entirely.

Does the BOM affect file size? Three bytes. Irrelevant practically, but it explains hex dumps that start unexpectedly.

Do I need it for UTF-16 files? Yes — there the mark genuinely marks byte order (UTF-16 LE vs BE) and parsers rely on it. The controversy is specific to UTF-8, where byte order is meaningless.