Safe rollouts, canaries and migrations

System Design · lesson 26 of 32 · 4 min read

Ship so a bad build reaches 1% of traffic, and change a schema with nobody noticing.

Open this lesson in the learning hub

Key points

  • A rolling update replaces instances a few at a time, so old and new code serve the same traffic for a while. Design for that overlap.
  • A canary sends 1% of traffic to the new build, watches error rate and p99 for a window, then goes 10%, 50%, 100% - or rolls back.
  • Blue-green keeps two full environments and flips the router. Instant rollback, at the cost of double infrastructure while you deploy.
  • Feature flags separate deploy from release: ship the code dark, switch it on for 1% of users, kill it again without a redeploy.
  • Migrate a schema in expand-contract order: add the column, write both, backfill in batches, read the new one, then drop the old.
  • Every single step has to be safe with both versions live. A column rename inside one release is a guaranteed outage mid-rollout.

Example

-- Expand / contract: five deploys, and no single one breaks the version still running.

-- 1. EXPAND. Nullable and no default, so there is no table rewrite and no long lock.
ALTER TABLE users ADD COLUMN full_name text;

-- 2. Deploy code that writes BOTH columns and still reads the old one.
UPDATE users SET name = $1, full_name = $1 WHERE id = $2;

-- 3. Backfill in batches, so the table is never locked for long.
UPDATE users SET full_name = name
 WHERE full_name IS NULL AND id BETWEEN 1 AND 10000;

-- 4. Deploy code that READS full_name. Every old instance is gone by now.

-- 5. CONTRACT, only once nothing reads the old column any more.
ALTER TABLE users DROP COLUMN name;

-- Never in one release: half the fleet is still selecting name at that moment.
-- ALTER TABLE users RENAME COLUMN name TO full_name;

Deploy small, watch a canary window, and never make a schema change only the new code survives.

This is a reading copy. The full lesson — with the visual explainer, the interactive lab and a Run button for the code — lives in the System Design course, and every lesson in it is listed on the System Design contents page.