Operations

During an incident, memory is unreliable and the clock is not. A timeline — who was paged, what was observed, what was tried, when the change landed — is what makes a response recordable and later reviewable. Write as it happens Memory is the least reliable instrument in an incident, so record while the alert is fresh. Capture the timestamp of the first alert, the initial symptoms, each action tried and its observable result, and the moment the fix actually took effect. That running record is what keeps a team from repeating a failed fix during the response, and what later makes an honest postmortem possible instead of a plausible story. Make the timeline a live document from the minute the page fires: note the tool that alerted, the metric, the suspected and the confirmed cause, and record ‘same as before, did not help’ rather than silently re-trying. ...

A standby node only helps if you know how to switch to it and that the switch works. Failover is a decision, a procedure, and a rehearsal. Delegating it to a switch you have never thrown is how a graceful handoff becomes an outage. ...

Capacity planning turns a demand forecast into a provisioned operating envelope. The useful question is not how many requests a machine can accept before crashing, but what workload the service can sustain while meeting its latency and correctness requirements. That answer depends on request cost, dependencies, and failure assumptions. A spreadsheet becomes useful when each of those assumptions has a source and an owner. ...

Deploying code and exposing it are different acts. A feature flag is a runtime switch that decides whether shipped code is active, letting you release continuously and switch behaviour without redeploying. Ship hollow, then turn on A feature flag splits the moment code ships from the moment users see it. The default value is off, the new code path is hidden behind the flag, and shipping changes nothing visible until the operator flips it. That decoupling is what lets a risky feature ride the normal release train: the deploy is reversible by turning a flag off, which is a faster rollback than replacing a build. Release and exposure become two separate verbs, so you can ship on a Tuesday and light the feature up when the room is actually watching. The empty shell also lets you validate deploy-time behavior — pod count, startup errors, metrics — with zero user-facing change. ...

A health endpoint is a contract with a consumer that will take action. Returning 200 because a handler ran does not prove that an instance can serve useful traffic. Before writing the endpoint, identify the decision it supports: allow initialization, admit requests, or restart a process that cannot recover. Different questions need different evidence. ...

A full backup provides a baseline without depending on an earlier backup from the same method. An incremental backup stores changes relative to a specified reference and depends on the material needed to reconstruct the omitted content. That distinction describes recovery dependencies, not a universal promise about speed or bytes transferred. A large changed working set can make an incremental backup substantial too. ...

Durable application state and disposable scratch files need different retention, capacity, and recovery policies. Separating their paths makes those policies easier to express, but directories alone do not create persistence or isolate disk capacity. The storage backing each path determines what survives process restart, container replacement, host failure, or cleanup. A reliable layout connects each class of data to an explicit lifecycle. ...

Latency, traffic, errors, and saturation are four complementary ways to ask whether a service is serving users well. They are a starting set of questions, not four universal counters that prove health by themselves. Their usefulness depends on where they are measured, which work they describe, and how their time windows align. A fast response can still be a failed operation. ...