What Is Erasure Coding?

Erasure Coding Explainer

Erasure coding protects stored data by transforming it into multiple fragments that include both original information and calculated redundancy. A common configuration is described with two numbers: the count of data fragments and the count of coding or parity fragments. The system can reconstruct the original object from a sufficient subset even when some fragments are unavailable. Compared with storing several complete replicas, erasure coding can provide similar failure tolerance with less raw capacity, which makes it attractive for large object stores and archival systems.

To write an object, the storage system divides or encodes it into fixed groups. Mathematical operations combine the data fragments to produce independent coding fragments. Reed-Solomon families are widely used because they can recover from multiple missing fragments up to the scheme’s designed limit. The fragments are distributed across different drives, servers, racks, or sites according to a placement policy. Separation matters: storing several fragments behind one power supply or network link would make them vulnerable to the same failure.

Reading a healthy object may require only its data fragments, depending on the implementation. If a fragment is missing, the system fetches enough remaining pieces and decodes them to reconstruct the absent information. It can return the requested data immediately and later repair the lost fragment onto new storage. Recovery consumes CPU, network bandwidth, and disk I/O because multiple fragments must be read and combined. Small or partial reads can be more expensive than replica reads if the layout requires touching several locations.

The code’s failure tolerance is not the same as a blanket promise against any number of device failures. The system must retain enough compatible fragments from the same encoded group. Placement domains, correlated outages, unavailable networks, and simultaneous maintenance all affect whether those fragments can be reached. Metadata that records object layout is also essential. Operators choose parameters by balancing usable capacity, repair cost, fault domains, latency, and the probability of overlapping failures.

Updates are another tradeoff. Changing a small part of an encoded object may require recalculating parity and coordinating several writes. Many object stores therefore use erasure coding for immutable or relatively large objects and handle small mutable data differently. During backfill or recovery, they throttle work so repair does not overwhelm client traffic, but slow repair leaves reduced durability for longer. Checksums detect corrupted fragments, while scrubbing finds damage before another loss removes the remaining recovery margin.

Erasure coding is not encryption, compression, or backup. Parity fragments may still reveal information and require normal access controls and encryption. Coding cannot reverse an authorized deletion that removes all associated fragments, and a software bug can damage every copy of an object. Independent backups, versioning, and disaster plans remain necessary. The technique’s strength is efficient redundancy: rather than keep multiple complete copies, the system distributes mathematically related pieces across failure domains and can rebuild missing data when enough trustworthy fragments survive. Its cost appears in computation, network fan-out, write complexity, repair traffic, and the operational discipline required to keep fragment placement genuinely independent.

Capacity calculations should include more than the raw data-to-parity ratio. Fragment alignment, object size, metadata, incomplete stripes, reserved recovery space, and replication of small objects can reduce usable capacity. A technically tolerant code can still lose data if the cluster becomes too full to place repaired fragments in distinct failure domains. Operators set fullness thresholds well before absolute capacity, test node and rack failures, and measure time to restore the intended fragment count. Durability comes from the code plus placement, available space, detection, and repair speed; none of those pieces can be omitted from the design.

Regular failure drills verify that assumption.