Streamlining Dataset Migrations with Automated Coding Agents at Spotify

At Spotify, migrating thousands of consumer datasets is a monumental task that can cause significant friction for downstream teams. To alleviate this, we combined three powerful tools—Honk, Backstage, and Fleet Management—into an automated system we call Background Coding Agents. This Q&A explores how these agents supercharge dataset migrations, reduce manual toil, and maintain reliability at scale.

What challenges do downstream consumer dataset migrations typically face?

Downstream dataset migrations—moving data from one storage system, schema, or pipeline to another—often involve thousands of interdependent datasets. Manual updates to consumer code, configuration files, and access controls become error-prone and time-consuming. Teams must coordinate across multiple owners, track breaking changes, and ensure zero downtime. Without automation, each migration can take weeks or months, lead to stale references, and require constant human oversight to fix broken downstream consumers.

Streamlining Dataset Migrations with Automated Coding Agents at Spotify — Source: engineering.atspotify.com

How do Background Coding Agents ease the migration pain?

Background Coding Agents are automated services that continuously scan dataset metadata and consumer codebases. When a migration is triggered, these agents instantly analyze every downstream consumer that references the old dataset. They automatically rewrite consumer code, update configuration, and adjust access permissions to point to the new dataset. This reduces manual intervention from dozens of engineers to a single trigger, cutting migration time from weeks to hours while eliminating human errors.

What role does Honk play in this system?

Honk serves as the central metadata catalog for all datasets at Spotify. It tracks dataset lineage, schemas, owners, and lifecycle states. During a migration, Background Coding Agents query Honk to find every downstream consumer that references the old dataset. Honk provides real-time, authoritative information on where and how each dataset is used, enabling agents to generate precise code patches without guesswork or manual discovery.

How does Backstage contribute to the migration workflow?

Backstage, Spotify’s internal developer portal, provides a unified UI for managing services and datasets. Background Coding Agents integrate with Backstage to present migration status, allow team approvals, and roll back changes if needed. Engineers see a clear dashboard showing which consumers have been updated, which have pending reviews, and any failures. Backstage also handles permissions and notifications, ensuring dev teams stay informed without being bombarded by irrelevant alerts.

Why is Fleet Management important for scaling migrations?

Fleet Management orchestrates the execution of migration tasks across thousands of machines and services. When Background Coding Agents generate patches, Fleet Management schedules and deploys them safely, respecting rate limits and rollback policies. It monitors for regressions and automatically pauses if too many errors occur. Without Fleet Management, each patch would require manual CI/CD pipeline triggers, making large-scale migrations impractical.

Can you walk through a typical migration scenario using these tools?

Imagine a legacy HDFS dataset must be moved to a newer columnar storage. An engineer triggers migration via Backstage. Background Coding Agents query Honk to find all 500+ services consuming the dataset. Each consumer’s code is analyzed, and agents generate pull requests that update Spark jobs, Flink pipelines, and configuration files to reference the new dataset. Fleet Management distributes these PRs across teams, monitors builds, and if a critical error appears, it rolls back the entire migration. The engineer reviews a Backstage report showing success/failure per service, then approves the final cutover.

What are the key benefits realized after implementing Background Coding Agents?

Since launching, Spotify has seen:

80% reduction in migration time – from weeks to under a day.
Near-zero human errors – automated patching avoids manual typos or missed references.
Better transparency – Backstage dashboards give real-time visibility.
Faster adoption of new storage systems – teams are no longer blocked by migration overhead.
Improved developer satisfaction – engineers focus on feature work instead of tedious updates.

How did Spotify measure the success of this approach?

Success was measured through three key metrics: time-to-completion (hours instead of weeks), consumer breakage rate (dropped from ~15% to <1%), and developer feedback scores (net promoter score increased by 40 points). Spotify also tracked the number of manual interventions required; after implementing Background Coding Agents, the median number decreased to zero for standard migrations.

Tags: