Experiences / 04
04 Break the NetworkBreak the network
Every network fails. What matters is how much of it fails, for how long, and who notices. Cut fibre, kill routers and take out a whole region, then watch real routing algorithms find a way around, or fail to.
Fail anything. Measure the recovery.
Pick an architecture, choose how failures are detected, then cut a link or a router by clicking it. Shortest paths are recomputed with Dijkstra's algorithm on link costs, exactly as a link-state protocol such as OSPF or IS-IS does. The traced flow shows its packets dropping at the break until the modeled convergence completes.
SimulationEvery topology, cost, latency and timing on this page is hypothetical and labeled as a simulation.
Who can still reach whom
A single failed element rarely takes everything down. It takes down specific pairs, and the matrix shows exactly which. Green cells give the surviving path cost and hop count; a multiplier marks equal-cost paths the network can spread traffic across. Red cells have no path at all, and no protocol can fix that: only redundancy built in advance can.
The matrix follows the instrument above. Change the architecture or break something and it recomputes.
Flows hit by failures
Four shapes, one question: what breaks?
Resilience is a property of the graph, not of the equipment. The table below is computed rather than asserted: each architecture is searched for cut vertices and bridges, then every possible single failure is tried and the damage measured.
- A
Hub and spoke
The cheapest design to build and to understand. One core router is a single point of failure for every branch, and each branch circuit is a bridge: lose it and that site is an island. It suits small estates where a few hours of outage is tolerable and the spares sit on a shelf.
- B
Redundant core
Two cores, dual-homed sites, two internet exits. Roughly twice the circuits, and no single element can partition the network. Most of the engineering effort moves into making failover fast: detection timers, not the topology, decide how long users notice.
- C
Partial mesh
A ring with shortcuts. Many equal-cost paths spread load and give the routing protocol choices, and the ring survives any single cut. The cost is complexity: more adjacencies to monitor, more paths to reason about when something degrades instead of failing cleanly.
- D
Full mesh
Maximum resilience and the shortest possible paths, at a price that grows with the square of the number of sites: six sites need fifteen circuits, twelve need sixty-six. Usually reserved for a small set of core locations rather than an entire estate.
Link-state versus distance-vector
Link-state routers flood a description of the failure to every router, and each recomputes the whole map. Distance-vector routers only exchange their own tables with neighbours, one round at a time, and can talk each other into routes that no longer exist. This experiment replays the most recent failure from the instrument both ways, using synchronous Bellman-Ford rounds with RIP's hop-count metric and its infinity of 16.
The model, stated plainly
A simulator is only as honest as its assumptions. These are ours. Recovery time is modeled as detection, plus propagation of the news, plus a deliberate wait before recomputing, plus the computation itself, plus installing the new routes in the forwarding table.
Simulation
- Detection
- Selectable. BFD 3 × 50 ms = 150 ms; loss of light 10 ms; OSPF dead interval 40 s without BFD. At the three-WAN edge, gateway health checks of 3 s or 15 s. In the wireless mesh, three missed one-second hellos.
- Propagation
- The latency-weighted distance from the routers next to the failure to the furthest router on the new path, using each link's assumed latency plus 1 ms processing per hop.
- SPF delay
- 50 ms, a typical initial SPF throttle in link-state protocols. Zero for the gateway and the mesh.
- Path computation
- Measured, not assumed: the time your browser takes to run Dijkstra from every node. A router's CPU will differ.
- Forwarding update
- 10 ms + 0.05 ms per prefix to program the new routes.
- Traffic
- 1,000 packets per second per affected flow. Packets lost = rate × recovery time. Flows with no surviving path stay down until a repair.
- Equal-cost paths
- Each flow is pinned to one of its equal-cost paths, as per-flow hashing does, so a single failure moves some flows and not others.
- Distance-vector
- Synchronous rounds with hop count, infinity 16, one round per 30 s periodic update. Triggered updates shorten this in practice.
- What it ignores
- Fast-reroute (loop-free alternates, TI-LFA) that can restore traffic in under 50 ms before convergence; microloops during convergence; congestion on the surviving path; grey failures that degrade instead of failing cleanly; BGP convergence at internet exits, which is measured in seconds to minutes. Real networks deserve measurement, not just modeling.
Design a network that fails well
We design redundancy where it pays for itself, tune detection so failover is measured in milliseconds rather than minutes, and document how every part fails before it is installed. Multi-WAN sites, satellite backup and voice that survives an outage are where we start.