Google engineer unplugged every fiber they could see and – surprise! – took down a chunk of the G-Cloud
Google Cloud has disclosed that a partial outage in its us-central1-b region this week was caused by an engineer accidentally unplugging network fibre-optic cables during routine hardware maintenance. The incident, which ran from 07:41 to 11:52 PT on 1 September, matters because it exposed a gap in Google's much-vaunted network redundancy: the system is designed to withstand single or even multiple simultaneous device and fibre-path failures, but not the sequential disconnection of every path at once.
According to Google's service health report, a procedural error caused the maintenance action to unplug 100% of fibre paths across all devices within just 13 minutes, with no warning reaching the engineer before the disconnection was complete. Traffic drop rates hit 100% at the incident's peak, leaving virtual machines in the affected zone unreachable to the outside world, though they could still communicate with each other. Google resolved the issue by rerouting traffic to healthy capacity elsewhere in the region while technicians located and physically reseated the disconnected optical links, after which normal service resumed.
- Google engineer accidentally unplugged all fibre cables in a datacentre zone
- Outage lasted roughly four hours on 1 September, cutting off VM access
- Redundancy design couldn't handle every path failing sequentially in minutes