On he topic cause vs symptom-based alerting there is one aspect missing: decoupling.
The systems that Rob is responsible for are decoupled from individual pieces of hardware (redundancy/fault-tolerance) and degrade gracefully (if one part fails only a fraction of the working set is affected). Otherwise the huge scale would not be manageable as some piece is always breaking.
So yes: if you run systems that are not somewhat decoupled from their base, cause==symptom and cause-based alerting is indeed the way to go. And you will be woken up by every single piece breaking. The way to improve this is decoupling, and then you'll also switch to symptom-based alerting.
The systems that Rob is responsible for are decoupled from individual pieces of hardware (redundancy/fault-tolerance) and degrade gracefully (if one part fails only a fraction of the working set is affected). Otherwise the huge scale would not be manageable as some piece is always breaking.
So yes: if you run systems that are not somewhat decoupled from their base, cause==symptom and cause-based alerting is indeed the way to go. And you will be woken up by every single piece breaking. The way to improve this is decoupling, and then you'll also switch to symptom-based alerting.