healing corrupted chunks ("wrong header") #352
Comments
|
This is strange and reminds me of #106 issue. Wrong header, file empty AFTER repair, all cased by HW problems... |
|
I don't see similarity with #106. I'm not sure if repair is happening. Healthy copies are available from other chunkservers and no missing chunks reported. |
|
The time of scanning all chunks on a single chunk server is measured in days or even weeks if you have a lot of chunks (on default configuration). It's a slow process, because it is meant as a lowest priority background maintenance process, "just in case". In normal circumstances chunks that just "lay there" on a disk should not get corrupted. Those physical chunks that are found to be bad, are replicated - the faulty chunk copy is marked as such and from this moment this chunk is considered undergoal and replicated via regular MooseFS mechanisms. And the faulty copies are deleted after the replication is finished. Of course, if you try to read from a chunk that has a yet-not-discovered bad chunk copy, then it will be discovered immediately - "out of order" of this regular maintenance check. |
|
Yes, I understand how it should work. But 8 hours passed with new corrupted chunk discovered every few minutes yet none of the files were fixed or removed. I don't see undergoal files in CGI either... |
|
As far as I'm concerned, not removing corrupted chunk files during testing is a bug. Why delay removing if it is already confirmed that the chunk file is invalid? |
Becau 8000 se "invalid" may mean that the whole chunk is zeroed or that two bytes are wrong. In the former case of course, the file is useless, in the latter case, if another copy of this chunk is also suddenly found missing or invalid, this copy might yet be repaired manually and some very, very important data might be saved. And, before you ask, no, we won't employ any algorithms to judge "how big" is the damage - this is a file system that is supposed to be fast. Now, how long this may take from discovering to deleting: first, the chunk is marked as invalid and undergoal situation appears. This will be dealt with quickly if your system is not extremely busy otherwise, because undergoal replications have high priority (and endangered even higher, which will be the case if you normally keep stuff in 2 copies). So unless you sit there and reload CGI or you have some script checking via CLI, you may not notice the undergoal before it is replicated. But the invalid copy is still there and will be marked for deletion in the next general check loop in the master. If you have a big installation, this loop may be quite long (CGI, info tab, check loop start time/check loop end time). After it is marked for removal, it will be with respect to deletion limits of course, so if your system deletes a lot of files, it may also take a while. So, on a big installation, 8 hours from discovering to deleting might not be enough. |
|
Of course I would not ask to "judge how big is the damage", that would be silly. But I'm asking to remove corrupted files immediately which is how LizardFS does it and it is the right way to handle corrupted files. You can implement a config option for those who want to leave junk/corrupted data around. It's been over 24 hours yet there were still no removals or repairs as far as I could see. |
|
Three days passed, corrupted files are still there, unrepaired, in the same quantity. Chunkserver logs the same errors over and over: Repair is definitely not working. |
Simple answer is no. I don't want to remove them immediately (and lizard also doesn't do that - I know it because basically all it does, it does because I've programmed it like that - they didn't change such things). The correct way is to inform the master about the corruption and let it do the rest. Usually removing data is exactly what the master will do (in like 99.99% of cases), but if ALL copies are corrupt then it is better to leave them around than to remove everything like stupid monkey. The problem you observed is just a result of stupid bug that I'm going to fix very soon. BTW I've just checked code of lizard and they have the same bug, so please never again tell me that lizard does anything better than moosefs. |
|
I'm glad to be wrong about LizardFS and it is awesome that you've found the problem, despite initial disbelief. I'll test soon. Thank you, Jakub. |
|
1d62a41 appears to work well: corrupted chunk files are being removed only minutes after discovery. Thank you for quick response, @acid-maker. |
Something strange happened. A chunkserver started to report large number of errors, all being
chunk_readcrc: ... - wrong header. There are no other errors whatsoever and all affected chunk files are on all local HDDs (on different file systems e.g. ext4, f2fs). All damaged chunk files are empty (filled with00) but their size varies from smallest being 73728 bytes.I've identified few thousand corrupted files, all empty, all created within 10 minutes between 16:10 and 16:20 on March 25 (chunkserver was 3.0.111 at a time). I'm investigating hardware but so far could not find anything to blame on the affected machine. I suspect that incident could have been caused by networking hardware (which I'm guessing would be unlikely).
Regardless of how those chunk files became corrupted, the problem seems to be that they are not being repaired or removed. Even after some time after reporting yet another corrupted file (which are discovered by background testing regularly) the files remain there, still empty and not even removed.
I suspect that auto-repair of damaged chunks files might be broken. Could you investigate please?
The text was updated successfully, but these errors were encountered: