Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

olake-go

olake-go: An open-source data ingestion engine for replicating databases, S3, and Kafka into Apache Iceberg or plain Parquet.

What problem it solves

Organizations need to move and continuously replicate data from transactional databases, event streams, and object storage into open lakehouse formats without depending on proprietary ingestion stacks. OLake Go addresses this with replication workflows, CDC, schema handling, and deployment options that avoid separate Spark, Flink, Kafka, or Debezium infrastructure.

Key features

  • Replication from multiple source types — OLake Go supports full loads and replication from transactional databases, Apache Kafka, and S3-compatible object storage. Supported databases include PostgreSQL, MySQL, MongoDB, Oracle, DB2, and MSSQL.

    Source: README, drivers

  • CDC and schema handling — The engine provides full-load, CDC, and incremental modes, along with automatic schema discovery and schema evolution. The CLI can discover available streams and their schemas from a configured source.

    Source: README

  • Iceberg and Parquet destinations — Data can be written as Apache Iceberg or plain Parquet. Iceberg outputs support Glue, Hive, JDBC, and REST catalogs, while Parquet output supports filesystem destinations.

    Source: README

  • Web UI and CLI workflows — A Docker Compose deployment provides a web interface for managing jobs, sources, destinations, and configurations. For automation, the CLI exposes commands for schema specifications, connection checks, discovery, synchronization, and destination clearing.

    Source: README, docker-compose.yml

  • Open lakehouse workflows — OLake Go is intended for use cases such as migrating transactional data to Iceberg, serving fresh CDC data to analytics tools, and building data lakehouse pipelines on cloud object stores.

    Source: README

Quick start

Copied verbatim from the project's README.

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d

Who it may suit

This may suit data engineers and platform teams building ingestion pipelines from databases, Kafka, or S3 into Iceberg or Parquet. It may also suit users who want a self-service UI for configuring pipelines or a CLI for automation.

Topics to discuss

  • Which connector or deployment mode should receive the most community attention next?

  • How should teams choose between full load, CDC, and incremental replication for new sources?

  • What examples or documentation would be most useful for configuring an Iceberg destination?


Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/datazip-inc/olake-go#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/datazip-inc/olake-go

Comments

…