MOLO17 ParquetKt: a high-performance Apache Parquet library for Kotlin

Working with Apache Parquet is a common requirement in modern data platforms, especially when performance, portability and schema control matter.
While solid implementations exist in other ecosystems, the Kotlin/JVM world has historically lacked a fully managed, production-ready Parquet library designed around Kotlin’s language features and runtime characteristics.

This gap led us to develop MOLO17 ParquetKt, a pure Kotlin library for reading and writing Apache Parquet files, now released as open source and already running in production within Gluesync.

Why ParquetKt exists

ParquetKt was born from a very practical need: efficiently handling Parquet files at scale in Kotlin-based systems, without relying on native bindings or heavyweight dependencies.

Instead of starting from scratch, we took inspiration from the excellent parquet-dotnet project and adapted its design principles to the Kotlin ecosystem, rethinking memory management, APIs and concurrency to fit the JVM runtime and Kotlin idioms.

The result is a library that focuses on:

• predictable performance
• controlled memory usage
• clear, type-safe APIs
• production stability

Designed for Kotlin developers

ParquetKt embraces Kotlin’s strengths rather than working around them.
At the high level, developers can read and write Parquet files directly from Kotlin data classes, with automatic schema generation and serialization. This makes common use cases concise and expressive, without sacrificing control.

For more advanced scenarios, the library also exposes low-level APIs that allow full control over schemas, row groups and column structures, making it suitable for complex pipelines and custom storage layouts.

Performance and memory behavior

One of the main design goals of ParquetKt is consistent performance under load, especially when dealing with large datasets.

The library uses memory-efficient defaults, smaller row groups and automatic flushing to disk to prevent uncontrolled memory growth. For long-running or high-volume jobs, this significantly reduces the risk of out-of-memory errors.

When needed, memory behavior can be further tuned through manual flush control and array pooling, allowing developers to adapt ParquetKt to different workloads and infrastructure constraints.

Async and streaming-friendly by design

ParquetKt supports both synchronous and asynchronous access patterns.
In addition to traditional blocking APIs, it provides coroutine-based functions and Flow support for non-blocking I/O and streaming reads.

This makes it a natural fit for modern Kotlin applications built around asynchronous processing and back-pressure-aware pipelines.

Compression, schemas and data types

The library supports the most commonly used Parquet compression codecs, offering a good balance between file size and throughput depending on the use case.

Schema handling is flexible: schemas can be defined explicitly or derived automatically via reflection from Kotlin data classes, including support for nullable fields and nested structures. This allows ParquetKt to be used both in strongly typed domain models and in more dynamic data processing scenarios.

Production-ready and open source

ParquetKt is already used in production as part of Gluesync, where it handles large-scale Parquet workloads daily.
The project comes with extensive test coverage and transparent benchmarks, all publicly available in the repository.

By open-sourcing ParquetKt, we aim to provide the Kotlin community with a reliable, well-tested Parquet implementation that can be reused beyond Gluesync and adapted to different data architectures.

Explore the project, benchmarks and documentation.