Snorkel AI uses Lakebase branches for isolated PR databases
Databricks Blog reports that Snorkel AI uses ephemeral Lakebase branches for each pull request in its Kubernetes-based CI pipeline, allowing tests to run against production-like data without sharing databases. The vendor says provisioning dropped from over 20–30 minutes to a target of under 60 seconds at p95, while the p95 result remains marked for verification and rollout reached 60–70 developers.
The case study shows how an AI data-tool company applies database branching to provision isolated, production-like test environments at scale, although the key latency result is still awaiting verification.
The problem: testing against production data shouldn't take over 30 minutes
Snorkel AI builds data development tools for AI teams, used by expert data annotators and frontier model providers. In this specific use case, Snorkel ships code through a CI pipeline built on Kubernetes. Every open pull request, plus ad-hoc commits, gets its own ephemeral Kubernetes namespace (a "PR namespace"). Developers deploy their branch there and run the full test suite before merging.
These tests are most useful when they run against real data. Snorkel's production database is an Amazon Aurora PostgreSQL cluster. Giving each PR namespace its own copy of that database was the hard part. Restoring a copy took about 20 minutes per environment, which is too slow when developers open PRs all day and the merge queue needs a fresh environment for each run. The alternatives (shared staging databases, trimmed fixtures, or synthetic data) bring back the problems production-like testing is supposed to solve: tests interfering with each other, stale schemas, and bugs that only show up with real data shapes.
Success criteria
Snorkel would consider Lakebase a success for CI/CD if it met these criteria:
#Snorkel AI#Databricks#Lakebase#Amazon Aurora PostgreSQL#Kubernetes#AWS DMS#Okta