This is the exact problem that a write-intent bitmap solves. When you create the array, specify
--bitmap=internal on the command line. The bitmap records if a region of the array has been written. Depending on the size of the array, one bit might be 1 GiB, for instance.
During normal operation, the kernel sets bits to 1 just prior to writing a region. Then, once it knows that all writes have flushed to a region, it zeroes that bit out. If you fail a drive out of the array, so that the array is degraded, the bits just start accumulating 1's. They aren't cleared.
When you re-add the drive back into the array, md only synchronizes the regions that had been written, according to the bitmap.
You can add a bitmap to an existing array with