Getting Started with parquet-tools: A Handy CLI for Parquet Files

Parquet is a popular columnar storage format in the data world, widely used with Hadoop, Spark, and cloud-based analytics platforms like Athena. While Parquet files are efficient and compact, inspecting their contents without loading them into a full data processing engine can be tricky. That’s where the parquet-tools command-line utility comes in.

This guide walks you through installing parquet-tools and using it to inspect and troubleshoot your Parquet files.

---

What is parquet-tools?

parquet-tools is a CLI utility for inspecting and interacting with Parquet files. It lets you:

• View file metadata
• Print schemas
• Dump contents
• Validate files

It’s ideal for quickly checking the structure or content of a Parquet file without needing Python, Spark, or Hive.

---

Step 1: Installing parquet-tools

macOS (Homebrew)

brew install parquet-tools

Ubuntu / Debian

sudo apt update

sudo apt install parquet-tools

Note: Depending on your Linux distribution, the package might come from a different repository. If apt doesn’t find it, you can download the JAR from the Apache Parquet releases and run it with Java.

Running the JAR directly (cross-platform)

1. Download the latest parquet-tools JAR from Apache Parquet releases.
2. Run it using Java:

java -jar parquet-tools-<version>.jar <command> your_file.parquet

---

Step 2: Viewing File Metadata

The first thing you often want to do is check the schema and metadata of your Parquet file.

parquet-tools meta your_file.parquet

This shows:

• Number of rows
• Column names and types
• Compression used
• Row groups and other file-level information

---

Step 3: Inspecting the Schema

If you just want to see the structure:

parquet-tools schema your_file.parquet

This prints the columns, data types, and nested structure, making it easier to understand the data before querying.

---

Step 4: Dumping the Contents

To see the actual rows:

parquet-tools dump your_file.parquet

Or, if you just want a sample of the data:

parquet-tools head your_file.parquet

Tip: For large files, using head prevents your terminal from being flooded.

---

Step 5: Validating a File

If you suspect a Parquet file might be corrupted:

parquet-tools check your_file.parquet

This will alert you to any structural issues in the file.

---

Step 6: Counting Rows

Sometimes you just want to know how many rows are in a file:

parquet-tools rowcount your_file.parquet

---

Tips for Effective Usage

1. Combine with other CLI tools: Use grep, less, or head to filter large dumps.
2. parquet-tools dump your_file.parquet | grep “some_value”
3. Check compression: Knowing the compression format (SNAPPY, GZIP, etc.) helps when reading files in Spark or Pandas.
4. Inspect nested structures: parquet-tools schema is very helpful when dealing with complex, nested columns.
5. Version awareness: Some features may depend on the version of the Parquet format; make sure your tools match your file version.

---

Conclusion

parquet-tools is a simple yet powerful utility for exploring and validating Parquet files. Whether you need to check a file’s schema, preview the data, or debug issues, parquet-tools provides quick insights without loading the file into a full data processing engine. For anyone working with Parquet regularly, it’s a must-have tool in the toolkit.