Permanent failures are handled by a garbage collection (GC) module. The GC module periodically scans local disks and discovers replicas that have edges to permanently failed nodes. When the GC module finds an edge to a failed bronze replica, it replaces the edge by performing a random walk starting from a gold replica (Section 4.4).
Recovering from a permanent loss of a gold replica is more complex.
When a gold replica, say 
, detects a
permanent loss of another gold replica, creates a new
gold replica on a live node chosen using the criteria described
in Section 4.1.
Because gold replicas form a clique (Section 3.3),
can always detect such a loss. This choice is flooded to all the
replicas of the file, using the protocol described in
Section 5, to let them update their uni-directional links to
the gold replicas. Simultaneously, updates the
local replica of the parent directory(ies), found in its backpointer(s), to
reflect 's new gold-replica set. This change is flooded
to other replicas of the directories.
Rarely, when the system is in transient state,
multiple gold replicas may initiate this protocol simultaneously.
Such a situation is resolved using the last-writer-wins policy, as
described in Section 5.2.
Recovering from a permanent node loss is an inherently expensive procedure, because data stored on the failed node must eventually be re-created somewhere else. The problem is exacerbated in Pangaea, because it does not have a central authority to manage the locations of replicas--all surviving nodes must scan their own disks to discover replicas that require recovery. To lessen the impact, the GC module tries to discover as many replicas that needs recovery as possible with a single disk scan. We set the default GC interval to be every three nights, which reduces the scanning overhead dramatically while still offering the expected file availability in the order of six-nines, assuming three gold replicas per file and a mean server lifetime of 290 days [3].