Team Viz
How Viz.ai’s Platform team uses Renovate and a small merge runner to roll shared-library updates across 100+ microservices, with no manual bump.
Written by Roei Avraham, Staff Software Engineer, Platform team at Viz.ai
The problem every engineer recognizes
Someone ships an update to a shared library — a real performance boost, a new option people asked for, a fix for a bug that has bitten production before. They write a polished Slack announcement, drop it in the right channel, and wait for the rollout.
A day passes. Nobody updates. A day becomes three, three becomes a week, a week becomes a month.
Eventually they corner one of the devs who originally asked for the change. “It’s ready, it’s not a breaking change, it’s literally a one-line bump.” The dev nods and says:
“Thanks — it’s just not prioritized right now. I’m working on other things.”
Nobody is wrong here. The library author shipped something good, the dev has higher-priority work, and the outcome is still bad for everyone: hard-earned improvements sit unused, and production keeps running on old versions.
Individually, a dependency bump is trivial busywork. The cost shows up when you multiply. We have 100+ Python microservices at Viz.ai, and each manual bump costs roughly 20 minutes of a developer’s focus — the interruption, the changelog, the pull request (PR), the continuous integration (CI) wait, the review to chase — and buys nothing, because the update was safe to begin with.
At that scale, “ask each owning team to bump on their own time” stops being a social problem and becomes a structural one. Even when every team eventually does the right thing, the long tail of stragglers means a release isn’t actually deployed across the fleet for weeks. By then two more releases have shipped, so the fleet is permanently behind — and when a common vulnerabilities and exposures (CVE) drops, “please patch this” competes with every team’s roadmap at exactly the moment you can least afford the delay.
This post is how we made that problem disappear, and why we trust it enough to ship to production with no manual step for each bump.
What the core library aggregator is
I’m on Viz.ai’s Platform Engineering team, and we maintain a central library aggregator set we call shared-modules (“we” means that team for the rest of this post).
shared-modules isn’t just a wrapper library. It’s the boundary layer between our microservices’ business logic and the infrastructure they sit on — in the spirit of ports and adapters (a.k.a. hexagonal architecture). Each module owns one piece of infrastructure: the messaging module, for example, wraps underlying message-broker libraries to standardize consumption and production, enforcing the limits we care about, shipping the monitoring we expect, and exposing a simpler API to R&D. The point is to let services move fast on product code without each one re-deciding how to talk to infrastructure.
Every update to a module triggers a new release of the entire set. Each service installs it at a pinned version range, and selectively opts into the libraries it needs:
dependencies = [
"shared-modules[core-initializer, events, messaging-client, internal-api, logging, metrics, rest-client-api]>=X.Y.Z,<N",
]Adopting a new version is, mechanically, just bumping the number on that version line. Trivial per service — and unmanageable at scale.
What we wanted, and why nothing off-the-shelf fit
Before picking a tool, we wrote down what “good” looked like:
- Updates to
shared-modulesshould bump microservices automatically — patch and minor versions merge on their own, and major versions never do; they wait for a human. - We control the cadence of opened PRs and merges to master. Every PR triggers a CI run that burns real resources (GitHub API quota, runner minutes, money), and if an update sneaks in a bug, a controlled rollout limits the blast radius.
- At most one auto-bump per service per week, even when many library aggregator releases happen back-to-back.
With those in hand, we looked at three options:
GitHub’s Dependabot. The path of least resistance — already integrated, zero infrastructure to set up. We ruled it out because it didn’t fit our cadence and rate-limit requirements: we couldn’t see a clean way to express those cadence and rate-limit rules without fighting the tool.
A fully in-house bumper. Maximum control, but as soon as we sketched what it would have to do — repo scanning, version detection, PR creation, rebasing open PRs when a newer release ships — we realized we’d be reimplementing most of what Renovate already does well. Not a good use of platform-team time.
Renovate with its built-in automerge turned on. The option we wanted to pick: configure Renovate, flip a flag, done. We rejected it because automerge merges a PR the moment it goes green — no fleet-level cadence for our use case, no per-service weekly limit. Giving up those controls meant giving up the safety story we cared about most.
So we built a hybrid: Renovate for the part it does well, our own merge logic on top for the part it doesn’t.
What we built: Renovate + a merge runner
The system is a producer and a consumer connected by GitHub itself — the open PRs are the queue. Renovate opens PRs, close to out-of-the-box: repo scanning, version detection, and rebasing open PRs when a newer shared library release lands. A small Python merge runner we wrote decides which of those PRs actually get merged, and that’s where the interesting work is — per-service cadence, a fleet-wide merge cap, CI-status checks.
Both halves run on scheduled automated workflows, with the merge window offset behind the open window. By the time the merge runner wakes up, freshly opened PRs have had time to run CI, so we avoid trying to merge a PR that’s still pending.
Why throttle merges at all?
The natural objection is: if your CI is solid, why not merge every green PR the moment it’s ready? Two reasons we don’t.
First, a green CI run isn’t a green production run. Bad bumps tend to declare themselves in QA or in production, after the merge, and spreading merges out over hours and days keeps that failure mode debuggable. Second, Renovate is configured to open new PRs regularly while capping overall concurrency, so bumps don’t go stale while the library aggregator keeps shipping. Merging at an unthrottled rate would amplify any single regression across the entire fleet at once.
Cadence isn’t a continuous integration and continuous delivery (CI/CD) smell, it’s how we keep automation incidents bounded and explainable.
The merge rules
For an open auto-bump PR to actually get merged, all of these have to be true:
- It’s marked [AUTO-MERGE]. Patch and minor updates get this tag automatically; major updates don’t, and stay open until a human reviews.
- All CI checks are green (success, skipped, or neutral). Anything failing or pending blocks the merge.
- The same service hasn’t been auto-bumped in the last 7 days. This is the rule we lean on hardest: a rate limiter at the service level, not the repo level, so a bad release lands on a controlled slice of the fleet instead of all services at once. It also means a team sees at most one dependency PR land on master per week, rather than being paged by Renovate three times a day.
- Each merge workflow run has a fleet-wide merge cap. Even when many PRs are eligible across many repos, we merge up to that cap per run and leave the rest for the next scheduled merge.
Rule 3 has a nice side effect: when a service does get bumped, it gets the latest available version, not the stale one Renovate first opened the PR with. Renovate keeps rebasing the open PR onto each new release, so by the time the cadence rule lets it merge, the diff is to the freshest version.
The runner itself is intentionally small: filter the open PRs by the rules above, slice to the merge cap, merge what’s left. A few hundred lines of Python — the rules are doing the work, not the code.
Safety nets: from PR to prod, with a reverse gear
Auto-merging dependency PRs into master only makes sense if you trust your pipeline to catch bad versions before they reach production — and to get out of the way when they don’t. Every auto-bump PR rides the same delivery pipeline a human-authored PR would:
- Pre-merge: CI checks. Unit tests, integration tests, lints, type checks, builds — and the merge runner refuses to merge until all of them are green.
- Post-merge: deploy to a QA environment. The newly merged commit is built and deployed to a dedicated QA environment — smaller than production, but the closest thing we have to it.
- In QA: service tests. A heavier suite runs against the live service — integration tests, contract tests, smoke checks against real-ish dependencies. This is the final gate.
- If service tests pass: promote to production automatically. No human in the loop, no “click to approve” button.
So far, this is just our normal delivery pipeline. The interesting question is: what happens when step 3 fails?
Automatic rollback
If service tests fail in QA, the pipeline doesn’t just stop — it actively undoes the change. The bump commit is automatically reverted, restoring the previous version, and the owning team is pinged with a link to the failed run, the offending PR, and the revert. The auto-bumper owns both halves of the loop: it merged the bump, so it’s on the hook for taking it back when QA disagrees. The team gets pulled in only for the part a human is actually needed for — investigating why the new version broke them.
The cadence rules limit how far a bad release can spread before we catch it; the QA gate plus auto-revert limits its time to recovery to minutes, before a real user sees it. Those two layers together are what made us comfortable taking the human out of the merge loop. Without the auto-revert, auto-merge would have been a much harder sell.
Rollout: opt-in, not opt-out
A platform team rolling out auto-merge across all services has two options: enable it everywhere by default, or let services raise their hand. We picked the second.
Every repo at Viz.ai already has a JSON config file describing its own CI/CD procedures — branch rules, deploy targets, that sort of thing. We added one field to it:
{
"auto_bumper": {
"enabled": true
}
}When the bump workflow runs, its first step scans every repo, reads that file, and builds the list of services where auto_bumper.enabled is true. That list is handed to Renovate via its autodiscovery filtering settings, so Renovate only ever opens PRs for services that have explicitly asked for it.
Three things this gave us almost for free. Owning teams stay in control, because the flag lives in their repo behind their normal review process — turning auto-bump on or off is a regular PR they own end-to-end, no platform-team ticket required. Onboarding is one line: flip the flag, merge the PR, and the next scheduled run picks the service up. And we could roll out gradually, starting with a handful of pilot services, which let us shake out bugs in the auto-bumper itself without touching the rest of the fleet.
We deliberately resisted making this opt-out. It would have been faster to roll out org-wide, but it would also have meant enrolling teams in a process they didn’t choose — and owning the fallout the first time something broke for one of them.
A year in: the numbers
A year after we shipped the first version, the auto-bumper had quietly become a load-bearing part of how Viz.ai ships software. ~95% of services are opted in. Adoption climbed gradually as teams watched the pilot behave, not in one big flip:
Approximate milestones from our internal auto-bumper Grafana dashboard; the shape matters more than any single point — slow at first, then compounding once reverts and prod deploys became routine evidence.
The system is sized for ~60 merges per week at full throughput, which at roughly 20 minutes of developer time per manual bump is ~20 hours a week fleet-wide — about half of a full-time engineer (FTE) the org gets back every year (an internal estimate, per our internal Grafana dashboard). That’s the ceiling, not the average; day to day we land below it, because of blocked PRs, holidays, and the usual friction. What that throughput really represents is developers not being interrupted, not reading changelogs for safe bumps, not opening PRs, not waiting for CI, not chasing reviews. From R&D’s point of view, shared-modules simply stays current. From ours, the work we ship shows up in production within days instead of months.
The shift shows up clearly in our largest service monorepo, where library aggregator bump merges to master went from ~15 a month by hand to ~90 a month, 93% of them automated. Bars are auto-bumper merges, the line is manual bumps:
Library aggregator bump merges to master per month, by author, in our largest service monorepo. Values read from our internal Grafana dashboard and approximate.
There’s a second-order benefit we didn’t design for but came to rely on: security patch propagation. When a CVE lands in one of the libraries our central library aggregator wraps, the fix is a single bump and a release, and the auto-bumper is the rollout mechanism — still cadence-limited, but dramatically faster than asking every team to reprioritize during a disclosure window.
We also know when it isn’t working: a small Grafana dashboard tracks opted-in services, open PRs per repo, and merges per service over time. That’s how we first spotted the GitHub-API runtime creep described below.
Lessons learned
A year of running this in production also surfaced the rough edges. Four are worth calling out.
1. Communication is at least as important as the code
The biggest blocker to adoption wasn’t technical — it was persistent nervousness from R&D, and we underestimated how long it would take to work through. Hand someone a tool that promises to merge dependency PRs and ship them to production without a human in the loop, and the natural first reaction is “I don’t think I trust that” — and the second, third, and fourth reactions are often the same.
The conversation that consistently moved teams from hesitation to opt-in went like this: what would you actually do differently with a human-authored PR? If your service tests pass in QA, would you really inspect anything manually before letting it go to prod? In almost every case the honest answer was “no” — the tests are the gate. The friction wasn’t really about the auto-bumper at all; teams were treating “a human reviewed it” as a safety mechanism, when in practice the safety came from CI and QA. Once that became visible, the trust followed.
We also leaned hard on visibility. The auto-bumper posts to Slack on every meaningful event for an opted-in service: a PR with failing CI, a successful deploy to production, and — most importantly — a service that fails QA tests and gets auto-reverted. The reverts were the real reassurance: “yes, it does back itself out — here’s the Slack message proving it.” When you ship automation that takes humans out of a loop they’re used to being in, explaining why it’s safe needs as much investment as the safety mechanisms themselves.
2. Renovate configuration has a learning curve
Renovate is enormously powerful, but the config language takes time to learn and the feedback loop is slow — you often only find out a rule was wrong after it has ignored a package it should have bumped, or bumped one it shouldn’t have. We learned to test config changes against a small subset of repos first.
It ramped up further when we started using Renovate to update versions in a custom in-house project file, not just pyproject.toml. That meant reaching for the advanced parts of Renovate: customManagers with regex matching and post-upgrade steps to run extra validation or maintenance tasks after a bump. Great features, but a step up in both expressiveness and the chance of getting it subtly wrong.
3. We’re a heavy GitHub API user, and it shows
The merge runner makes a lot of GitHub API calls per cycle: list open PRs, fetch each PR’s check runs, list recent merges to filter out services bumped in the last 7 days. Multiply by every opted-in repo, every hour, and you start feeling rate limits and runtime creeping up. Most of this data is cacheable — check statuses don’t change retroactively, recently-merged PRs don’t move around — so a small cache layer that only re-fetches what could plausibly have changed since the last run is our next planned investment.
4. “Right cadence” is a moving target
The once-per-week-per-service rule is conservative on purpose. The downside is that on a busy week, services at the back of the queue wait noticeably longer than that, especially when many services share the same available updates and the fleet-wide merge cap kicks in.
We’re thinking about a follow-up mechanism that guarantees a bump reaches every opted-in service within a bounded amount of time, regardless of how full the queue is, while keeping the safety properties the current cadence gives us. No final design yet — it’s the kind of trade-off where it’s easy to relax the cadence and accidentally relax the safety story along with it.
Closing
Remember the Slack announcement from the start of this post? The one nobody acted on?
We don’t write it anymore.
When a new library aggregator version goes out, nothing happens in Slack. No nudges, no in-person follow-ups with the devs who originally asked for the change. The release just propagates — quietly, on a schedule, behind the same CI and QA gates everything else at Viz.ai rides on — and within a week or two it’s running in production across most of the fleet.
The auto-bumper isn’t impressive because of Renovate, or the merge runner, or the cadence rules. It’s impressive because the most reliable Slack message about a new release is the one we no longer have to send.
The lesson generalizes well beyond shared libraries. A lot of what slows engineering orgs down isn’t hard problems — it’s a long tail of small, safe, obvious tasks that each cost a little focus and collectively cost a fortune. If you’ve ever stared at a “please update” message that nobody acted on, whether you sent it or ignored it, there’s a good chance the right fix isn’t to nag harder. It’s to make the safe path happen on its own.
About Roei Avraham
Roei Avraham is a Staff Software Engineer on the Platform team at Viz.ai.