Combine CSV Files from Multiple Folders

You have a data dump organized like this:

/2023
  /Jan/data.csv
  /Feb/data.csv
/2024
  /Jan/data.csv

You want ONE master CSV file containing everything.

Standard merge tools often only look in one folder. Here is how to do a "Recursive Merge".

Method 1: Command Line (Windows)

1. Open Command Prompt (cmd).
2. Navigate to the top folder: cd C:\MyData
3. Run this command:

for /r %i in (*.csv) do @type "%i" >> all_data.csv

The Problem: This includes headers from every file repeatedly in the output. The Fix: You'll need to use a tool to remove duplicate header rows later.

Method 2: Command Line (Mac/Linux)

1. Open Terminal.
2. Navigate to folder.
3. Run:

find . -name "*.csv" -exec cat {} + > all_data.csv

Same Problem: Repeated headers.

Better Version (Skip headers for subsequent files): This is complex in bash. It's easier to use Python.

Method 3: Python (The Clean Way)

This script walks through all folders, finds CSVs, and merges them smartly (keeping only one header).

import os
import pandas as pd

master_df = pd.DataFrame()

# Walk through all folders
for root, dirs, files in os.walk("."):
    for file in files:
        if file.endswith(".csv"):
            path = os.path.join(root, file)
            # Read CSV
            df = pd.read_csv(path)
            # Add a 'Source' column so you know where it came from
            df['Source_File'] = path
            # Append to master
            master_df = pd.concat([master_df, df])

master_df.to_csv("combined_all.csv", index=False)

Method 4: Power Query (Excel)

1. Data > Get Data > From File > From Folder.
2. Select the top-level folder.
3. Excel will show a list of all files in all subfolders.
4. Click Combine & Load.
5. Excel automatically handles the headers and merges them.

Pros: No code, handles headers correctly. Cons: Can be slow with thousands of files.

Summary

• Quick & Dirty: Command line (but watch out for repeated headers).
• Clean & Automated: Python script.
• User Friendly: Excel Power Query.

---

Files everywhere? HappyCSV can accept drag-and-drop of multiple files and combine them instantly.