Why This Matters
No matter how good your tests are, things will break in production. A database migration locks a table. A new feature causes a memory leak. A third-party API changes its response format. The question is not whether an incident will happen, but how quickly and calmly your team responds.
A rollback is the fastest way to stop the bleeding: revert to the last known good version while you investigate. Incident response is the structured process of detecting, triaging, mitigating, and learning from production failures. Teams that practice incident response recover in minutes instead of hours. A runbook turns tribal knowledge into a checklist that anyone on the team can follow under pressure.
Define Terms
Visual Model
The full process at a glance. Click Start tour to walk through each step.
Incident response: detect, triage, mitigate, communicate, resolve, and learn.
Code Example
// Feature flag: toggle features without redeploying
const featureFlags = {
newCheckout: true,
betaSearch: false,
darkMode: true
};
function isEnabled(flag) {
return featureFlags[flag] === true;
}
// Use feature flags to control rollout
if (isEnabled("newCheckout")) {
console.log("Showing new checkout flow");
} else {
console.log("Showing old checkout flow (safe fallback)");
}
// Deployment strategies
const strategies = {
blueGreen: {
description: "Two identical environments. Route traffic from blue (old) to green (new).",
rollback: "Switch traffic back to blue instantly."
},
canary: {
description: "Send 5% of traffic to the new version. Monitor for errors.",
rollback: "Route all traffic back to the old version."
},
rollingUpdate: {
description: "Replace instances one at a time. Each new instance gets traffic.",
rollback: "Stop the rollout and replace new instances with old ones."
}
};
for (const [name, strategy] of Object.entries(strategies)) {
console.log(`${name}: ${strategy.description}`);
console.log(` Rollback: ${strategy.rollback}`);
}Interactive Experiment
Try these exercises:
- Implement a simple feature flag system. Store flags in an object and create
isEnabled(flag)andsetFlag(flag, value)functions. Toggle a flag and see the behavior change without restarting. - Simulate a canary deployment: write a function that takes a
canaryPercentage(0-100) and randomly routes requests to either "v1" or "v2" based on that percentage. - Create a runbook as a checklist: list the exact steps to diagnose and fix a common problem (e.g., "database connection timeout"). Include the commands to run.
- Practice a rollback: deploy a "broken" version of a simple app (one that returns 500 errors), then switch back to a "working" version.
Quick Quiz
Coding Challenge
Write a function called `createCanaryRouter` that takes two version strings (`stableVersion` and `canaryVersion`) and a `canaryPercent` (0-100). It should return an object with two methods: `route(userId)` returns which version a user gets (use `userId % 100 < canaryPercent` to determine canary assignment), and `getStats()` returns an object with `stable` and `canary` counts tracking how many times each version was routed.
Real-World Usage
Rollbacks and incident response are practiced at every major tech company:
- Blue-green deployments at Amazon keep two identical production environments. Traffic switches instantly from blue (old) to green (new) and can switch back in seconds.
- Feature flags (LaunchDarkly, Unleash) let teams deploy code to production but keep it hidden until they are ready to enable it. If a feature causes problems, flip the flag off instantly.
- PagerDuty manages on-call rotations and escalation policies. If the primary on-call does not acknowledge an alert within 5 minutes, it escalates to a backup.
- Incident commanders at Google and Meta coordinate response during major outages, ensuring clear communication and preventing chaotic debugging.
- Blameless postmortems after every significant incident document what happened, the timeline, and action items to prevent recurrence.