Go File Processor

Sample project to efficiently process massive files in Golang using concurrency, buffering, and object pooling.

Requirements

• Go 1.20+

Features

• Efficient processing of large files line by line or in chunks
• Process lines in configurable chunks
• Concurrency with goroutines and a scalable worker pool
• Buffered I/O for high-throughput reading
• Object pooling to minimize memory allocations and reduce GC pressure
• Command-line interface with flexible parameters (-file, -chunk, -workers)
• Include a script to generate massive CSV files

Command-line Flags

The application accepts the following command-line arguments:

Flag	Type	Description	Default	Required
-file	string	Path to the CSV or text file to process	—	Yes
-chunk	int	Number of lines to group per chunk (helps with batching and performance)	1000	No
-workers	int	Number of concurrent workers to process each chunk	Number of CPU cores	No

Example to use:

go run . -file hugefile.csv -chunk 5000 -workers 8

Generate a massive dummy CSV file

./generate.sh 10000 #Number of lines, default 1000000