olake-go: An open-source data ingestion engine for replicating databases, S3, and Kafka into Apache Iceberg or plain Parquet.
What problem it solves
Organizations need to move and continuously replicate data from transactional databases, event streams, and object storage into open lakehouse formats without depending on proprietary ingestion stacks. OLake Go addresses this with replication workflows, CDC, schema handling, and deployment options that avoid separate Spark, Flink, Kafka, or Debezium infrastructure.
Key features
Replication from multiple source types — OLake Go supports full loads and replication from transactional databases, Apache Kafka, and S3-compatible object storage. Supported databases include PostgreSQL, MySQL, MongoDB, Oracle, DB2, and MSSQL.
Source: README, drivers
CDC and schema handling — The engine provides full-load, CDC, and incremental modes, along with automatic schema discovery and schema evolution. The CLI can discover available streams and their schemas from a configured source.
Source: README
Iceberg and Parquet destinations — Data can be written as Apache Iceberg or plain Parquet. Iceberg outputs support Glue, Hive, JDBC, and REST catalogs, while Parquet output supports filesystem destinations.
Source: README
Web UI and CLI workflows — A Docker Compose deployment provides a web interface for managing jobs, sources, destinations, and configurations. For automation, the CLI exposes commands for schema specifications, connection checks, discovery, synchronization, and destination clearing.
Source: README, docker-compose.yml
Open lakehouse workflows — OLake Go is intended for use cases such as migrating transactional data to Iceberg, serving fresh CDC data to analytics tools, and building data lakehouse pipelines on cloud object stores.
Source: README
Quick start
Copied verbatim from the project's README.
curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -dWho it may suit
This may suit data engineers and platform teams building ingestion pipelines from databases, Kafka, or S3 into Iceberg or Parquet. It may also suit users who want a self-service UI for configuring pipelines or a CLI for automation.
Topics to discuss
Which connector or deployment mode should receive the most community attention next?
How should teams choose between full load, CDC, and incremental replication for new sources?
What examples or documentation would be most useful for configuring an Iceberg destination?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/datazip-inc/olake-go#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…