Big Data File Formats

Big-data file formats trade off storage efficiency, schema evolution, and query speed. The best choice depends on whether the workload mostly scans records, reads a few columns across many rows, or exchanges events between systems.

Row-Oriented Formats #

Row-oriented formats place the fields of one record together. Avro is a common example. It is useful for streaming, event interchange, and write-heavy workloads because reading a complete record is efficient. Avro stores its writer schema with the data and has explicit schema-resolution rules that support compatible evolution.

Column-Oriented Formats #

Parquet and ORC organize data by column. Analytical engines can read only the columns a query needs, apply compression and encodings to similar values, and skip data using file or row-group statistics. These properties make them good defaults for large analytical tables.

Partitioning is a directory-level layout convention, not an inherent property of Parquet. A table may be partitioned by a low- to medium-cardinality field such as date, while each partition contains Parquet files. Avoid creating excessive tiny partitions or files; metadata and scheduling overhead can overwhelm the benefit.

Practical Selection #

Use a columnar format such as Parquet for data-lake analytics. Use Avro when a row-oriented, schema-evolving message or record format fits the pipeline. Choose ORC when its ecosystem integration or features are advantageous for a particular Hive-oriented workload. Regardless of format, manage schemas, file sizes, compression, and partitioning as part of the table design.