How to build an on-premise data lake using Gluesync Universal File Store Agent

As organizations adopt increasingly distributed data architectures, requirements around data sovereignty, compliance, and infrastructure control are becoming more stringent. In many industries, including finance, healthcare, and regulated enterprise environments, data must often remain within private infrastructure rather than being streamed exclusively to public cloud storage services.

While cloud object stores are widely used for real-time data pipelines, many organizations require the ability to land streaming data directly into on-premise or privately managed file systems.

With Gluesync 2.2.9, the Universal File Store Agent provides a direct way to stream real-time replicated data into file-based storage systems using standard network protocols.

A unified file-based integration layer

The Universal File Store Agent acts as a target connector that enables real-time data delivery from supported source systems into file storage endpoints.

It supports a range of standard enterprise protocols, including:

• FTP
• FTPS
• SFTP
• WebDAV
• WebDAVS
• SMB/CIFS
• NFS

This allows teams to integrate with a wide variety of file-based storage systems, including network shares, on-premise storage arrays, and private cloud file services.

Flexible output formats for downstream systems

To support different consumption patterns, the agent can generate data in multiple formats:

• Apache Parquet (default, optimized for analytics workloads)
• Line-delimited JSON
• CSV

Each format is designed to support interoperability with downstream processing systems, data lakes, and analytics engines.

Metadata-driven recovery with sidecar files

File-based pipelines do not inherently preserve transactional boundaries. To support reliable recovery and traceability, the Universal File Store Agent generates sidecar JSON metadata files alongside each output file.

These metadata files store replication-related information such as offsets and transactional context, enabling Gluesync to resume processing accurately in the event of interruptions or restarts.

 

Designed for on-premise and private data lakes

The Universal File Store Agent is particularly well suited for:

• on-premise data lake architectures
• network-attached storage (NAS) environments
• private cloud file systems
• hybrid file-based integration workflows

By using standard protocols, it avoids dependency on proprietary object storage APIs and enables deployment in controlled infrastructure environments.

Operational considerations

When configuring file-based targets, the destination storage system must be accessible through a pre-configured network share or mount point, and appropriate read/write permissions must be in place for the Gluesync runtime environment.

Try Gluesync 2.2.9 today

The Universal File Store Agent in Gluesync 2.2.9 extends real-time replication capabilities beyond traditional object storage systems, enabling organizations to deliver streaming data directly into file-based infrastructures using standard protocols and formats.

This makes it possible to design flexible on-premise or hybrid data lake architectures without relying exclusively on public cloud storage services.

For the complete list of raw JSON schemas, connection parameters, and Web UI input boxes, visit the official Gluesync Documentation Hub.

Want to evaluate the Universal File Store Agent directly against your own on-premise network shares or private file targets? Kickstart a sub-45ms real-time data replication evaluation kit by requesting a free 30-day trial.