When a recovery bench is asked about the most common stories that are told by all of them, they will tell this same story: an array loses a drive, the replacement for the lost drive is installed, and during the rebuild process a second "healthy" drive fails, resulting in there being no more lives left to live in the last remaining piece of redundancy. The occurrence of this failure event is so predictable, it needs some kind of logical explanation, because the logic behind it changes the course you should take in the window of time between failures.
Same batch, same hours, same wear: why a second drive so often follows the first.
Reliability maths is based on an assumption that each time a unit fails, it's as if the other units have never failed before. This is where arrays fail to meet this assumption from the very beginning. The drives are generally the same model, produced at the same facility during the same production run, installed within minutes or hours of each other, operate at the same temperature in the same enclosure, are worked by the same workload and have been running for nearly the same amount of time. It is simply not possible to consider them as individual events; rather, they were formed together and age together. Drive lifespans follow what's called the bathtub curve, with failures clustering early (defects) and late (wear-out). When an array is mature enough that one drive reaches the end of its life due to wear out, then all of the other drives in that array are near that point also. Thus when a single failure occurs it is not just bad luck; it indicates that the group of identical units has reached the final phase of their useful life.
The worst time for an array happens when it has been damaged (to some extent). It is then that the array will be rebuilt. In rebuilding an array, each sector on each available disk must be fully read from the disk, for hours or days. An array's disks usually do light reads, as they are reading scattered amounts of information. On the other hand, during a rebuild, those same disks will perform many long reads at maximum speed. The array may also be running warmer than usual since one of the bays is busy resilvering. Because there has been no recent disk read to identify latent bad blocks (bad sectors) on the disks, these blocks are now identified by the rebuild. The rebuild is the first operation in years that has completely read all data from each disk. The irony is that the process to restore redundancy is precisely the most stressful operation that could be performed on an array's remaining donor disks, and the one most likely to kill them. That is why the bench's rule for degraded arrays with irreplaceable data seems like heresy to common sense: don't rebuild, image first. Copies can survive a rebuild, but originals sometimes don't.
At this point in time, once an array loses its first disk (drive), you are within your window to make the most critical three moves. Back up now, and do not wait until after the rebuild has begun. The degraded array is still delivering data, and tonight could possibly be its final night. Evaluate what the data on the array is worth prior to making any changes: replaceable data will allow you to take a chance with rebuilding, while irreplaceable data dictates turning off the array and creating an image of each disk prior to beginning a rebuild. In addition, should a rebuild attempt fail at some point, stop immediately: there is documentation in the form of a 62% case file detailing the successful recovery from a failed rebuild when nobody tried a third time. RAID arrays provide one opportunity to act in an emergency, and they give no second chances to confidence.
If there's a hum from a degraded array behind you while reading this, that is your window. You get a no-charge diagnostic and a fixed price quote in writing, and totally free consultations with honest advice can be obtained by calling 0131 202 0491.