Two outages on 3 October, their causes and the fixes

On 3 October the control plane failed twice. The control plane is the service that tells gateways which site a domain belongs to, and the one nodes check in with. This entry covers both outages: what you would have seen, what caused each, and what has changed.

Outage 1: every site timed out, about 13:40 to 13:59 UTC

What happened. For about 20 minutes every customer site timed out. The control plane had stopped accepting new connections, so gateways couldn’t look up which site a domain belonged to. A restart brought sites back.

Cause. Pages that use peer delivery keep a connection open to the control plane while the tab is open. When a browser vanished without closing it, because a laptop went to sleep, a phone changed network or a tab was killed, the control plane never gave up on that connection. Dead connections piled up for about a day until there was no room for new ones.

Fix, live at 14:14 UTC. The control plane now checks each of these connections every 30 seconds and closes any that stop answering. It also gives up on a browser that has stopped reading, instead of waiting on it.

Outage 2: node check-ins hung, about 15:10 to 16:34 UTC

What happened. Node check-ins, new node registrations and node listings stopped answering. A restart at 16:13 cleared it, and it came back within 20 minutes. Sites kept loading throughout.

Cause. Two steps that run on every node check-in could each end up waiting for the other, forever. Once that happened, every later check-in queued behind them.

Fix, live at 16:34 UTC. The two steps now take their turns in the same order, so they can’t block each other. A follow-up the same day moved a slow lookup out of the way, so one slow answer no longer holds up every other node update.

What we don’t know

• The second cause was reproduced in a test, not confirmed from production. The record taken during the outage was incomplete. It matches this fault, and the test fails without the fix and passes with it, but we couldn’t confirm that this pair of steps was what froze.
• The first outage may have had the same cause. It ran on a version with the second fault too, so it may have been that freeze, as well as or instead of the build-up of connections.
• The connection fix was tested with test clients, not real browsers.

What else changed

• We now watch the service from outside. Nothing was monitoring the control plane on 3 October. It, the portal and our own sites are now checked every minute, and the results are public at status.evolvingedge.ai.
• Slow first requests were a separate problem, found the same weekend and fixed on 4 October. See First requests to a site no longer wait several seconds.

What you need to do

Nothing. Sites recovered when the control plane restarted, and nodes resumed checking in on their own.