Detecting and preventing HFS Plus/NTFS bit rot
Neither HFS Plus nor NTFS does any data integrity checking; aka checks for “bit rot” on data files stored on the system. This is concerning because Time Machine and similar tools cannot detect if they are backing up corrupt data.
Are there tools that can detect corruption and warn me of it?
What is the best consumer strategy for keeping my data integrous? Do I have to go all the way and create a ZFS/Btrfs NAS to store my information?
Update:
After some research I found that there are 2 ports of ZFS to Mac OS X:
- https://openzfsonosx.org/
- http://code.google.com/p/maczfs/
This seems the best approach to gaining data integrity on Mac OS.
ntfs hfs+
add a comment |
Neither HFS Plus nor NTFS does any data integrity checking; aka checks for “bit rot” on data files stored on the system. This is concerning because Time Machine and similar tools cannot detect if they are backing up corrupt data.
Are there tools that can detect corruption and warn me of it?
What is the best consumer strategy for keeping my data integrous? Do I have to go all the way and create a ZFS/Btrfs NAS to store my information?
Update:
After some research I found that there are 2 ports of ZFS to Mac OS X:
- https://openzfsonosx.org/
- http://code.google.com/p/maczfs/
This seems the best approach to gaining data integrity on Mac OS.
ntfs hfs+
add a comment |
Neither HFS Plus nor NTFS does any data integrity checking; aka checks for “bit rot” on data files stored on the system. This is concerning because Time Machine and similar tools cannot detect if they are backing up corrupt data.
Are there tools that can detect corruption and warn me of it?
What is the best consumer strategy for keeping my data integrous? Do I have to go all the way and create a ZFS/Btrfs NAS to store my information?
Update:
After some research I found that there are 2 ports of ZFS to Mac OS X:
- https://openzfsonosx.org/
- http://code.google.com/p/maczfs/
This seems the best approach to gaining data integrity on Mac OS.
ntfs hfs+
Neither HFS Plus nor NTFS does any data integrity checking; aka checks for “bit rot” on data files stored on the system. This is concerning because Time Machine and similar tools cannot detect if they are backing up corrupt data.
Are there tools that can detect corruption and warn me of it?
What is the best consumer strategy for keeping my data integrous? Do I have to go all the way and create a ZFS/Btrfs NAS to store my information?
Update:
After some research I found that there are 2 ports of ZFS to Mac OS X:
- https://openzfsonosx.org/
- http://code.google.com/p/maczfs/
This seems the best approach to gaining data integrity on Mac OS.
ntfs hfs+
ntfs hfs+
edited Dec 26 '14 at 15:07
asked Dec 4 '14 at 23:02
hekevintran
1,18121732
1,18121732
add a comment |
add a comment |
5 Answers
5
active
oldest
votes
I’ve researched dealing with this issue for fairly large—20TB+—enterprise-sized storage systems and the consumer reality is this: ZFS based systems is really the only way to deal with this. If data rot is a real concern, then I would recommend having at least one other hard drive for backups that you backup. Not RAID or anything magical, but simply another external drive that is synced using a tool like rsync
if you are comfortable using the command line or Carbon Copy Cloner which is basically an app that performs the same function as rsync
but has a nice user interface.
I did do some searching right now and found ZFS on Linux which sounds interesting; an open source implementation of ZFS for Linux systems. If you are comfortable rolling up your sleeves and setting up Linux/Unix stuff, this could be a potential solution for a do-it-yourself NAS. But I do not have direct experience with it, so can’t speak for it’s long term usefulness in a “production” environment.
1
I've been using ZFS for years now on my home server, with FreeBSD. It's really great but eats as much RAM as you throw at it. ZFS for Linux should be sufficiently stable by now, I'm using it on one of my dedicated servers. No problems there either.
– Daniel B
Dec 4 '14 at 23:41
@DanielB Very good to know! Now I have a new project to spend time on!
– JakeGould
Dec 4 '14 at 23:46
2
I'd add setting it up right is a bit tricky. There's a SF regular who's a wizard at this, and if I recall correctly he suggests having a fast SSD for ZIL and l2arc. You're also going to want a ton of ram on your storage box as Daniel B mentioned.
– Journeyman Geek♦
Dec 5 '14 at 0:16
It'd probably be easier to use FreeNAS to set up ZFS on FreeBSD than doing it from scratch, since you're looking to build a NAS anyway (unless, of course, you prefer to set it up manually so you understand it).
– Suchipi
Dec 5 '14 at 6:25
@Suchipi “Unless, of course, you prefer to set it up manually so you understand it.” For the original poster, maybe that is the best option. But I do Linux/Unix development, systems administration and security work. Even if I setup something the easy way I assure you at some point I will have to “dig deeper” to really get it to work the way a client desires. So I would rather build from scratch; works better for my process.
– JakeGould
Dec 5 '14 at 6:29
|
show 3 more comments
It's worth noting that Microsoft now has ReFS (resilient file system) on Windows 2012+ and Windows 8.1, which does check integrity. Furthermore, if you run ReFS on a mirrored storage space, it can automatically correct those errors by using bits from the other side of the mirror.
ReFS doesn't support all the features of NTFS, so you'll have to decide if any of the things it's missing are important for the files or workloads you need it for.
Good tip! Now, the only problem: Microsoft.
– JakeGould
Dec 5 '14 at 4:48
Well you don't specify your desired operating system and you listed NTFS so I figured it was fair game that Windows might be acceptable! I may be biased as a primarily Windows sysadmin, but Microsoft's stuff is pretty great lately, especially on the server side. Do you have a specific concern?
– briantist
Dec 5 '14 at 4:50
“Well you don't specify your desired operating system…” I am not the original poster. I am simply a site user who gave you a “+1” for this, said “Good tip!” and explained by only personal issue with it: Microsoft. As for whether this is an issue with the original poster, not my position to say.
– JakeGould
Dec 5 '14 at 4:52
Whoops my mistake! I'm on mobile so it's easier to miss. I'm still curious if you have a specific concern, as it would add to the discussion and would probably be useful to the OP.
– briantist
Dec 5 '14 at 4:55
1
My concern is the original poster mentions only Time Machine (an Apple specific Mac OS X tool) as their backup tool and might only be approaching this issue from the Apple side where sometimes—thanks to Fuse—Mac users format drives in NTFS for cross-platform compatibility and will not be mounting—or managing this—in a Microsoft OS environment that would really be able to handle ReFS. If somehow ReFS is cross platform in usability and resiliency without OS concerns, great! But I have a feeling that might be a stumbling block to it being adopted in this scenario as presented.
– JakeGould
Dec 5 '14 at 4:59
|
show 2 more comments
chkbit is a lightweight bitrot detection tool (OS X/Linux/Windows).
chkbit cannot repair bitrot, its job is simply to detect it.
You should
- backup regularly.
- run chkbit before each backup.
- check for bitrot on the backup media.
- in case of bitrot restore from a checked backup.
add a comment |
You can always manually compute checksums with md5sum
and check them periodically, or you can use btrfs, which has an online checksumming feature. On the other hand, it really is kind of redundant and unnecessary since disk drives already have their own error detecting and correcting codes.
And if one day the MD5 checksum doesn't match, then what happens? ZFS allows for snapshots of data so you can recover a valid version of a file. On a normal file system, if the data doesn't check out, the data is gone unless there is a backup.
– JakeGould
Dec 5 '14 at 0:24
@JakeGould, snapshots won't help with that either since all of the snapshots share the same data blocks on disk if they were not modified intentionally.
– psusi
Dec 5 '14 at 4:07
Snapshots would help. You need to investigate how ZFS systems work. They are designed specifically to deal with data rot issues like this and provide a way to alert and restore data.
– JakeGould
Dec 5 '14 at 4:47
@JakeGould, no they would not. The whole point of snapshots is that they do NOT duplicate the data and thus, take up twice the space. ZFS can tell you that the data has become corrupted but unless you configure it with a redundant raid configuration, it can't recover the data. If you are using raid, then that is what provides the backup copy that hopefully is still good, not snapshots.
– psusi
Dec 5 '14 at 14:42
@psusi Running md5sum is not really a good solution since it alone does nothing for recovery which is the whole point of wanting to know where the corruption is. I don't think that checking integrity is redundant because drives have error correction. The world is complicated, hardware breaks all the time. Recovery is the goal.
– hekevintran
Dec 5 '14 at 18:39
|
show 1 more comment
For Windows I've discovered this little program:
"DiskFresh is a simple yet powerful tool that can refresh your hard disk signal without changing its data by reading and writing each sector and hence making your disk more reliable for storage"
It does a full read/write cycle of all the sectors on a disk, so that you can prevent bitrot.
I have some SATA harddisks which I use for archiving purposes, so they are not always connected to the computer. I have them in a plastic enclosure and are kept in a drawer together with some moisture absorbing bags. If and when needed I just slide them in one of those sata disk slots I've installed on the tower. Because they sit for extended periods of time offline, I've some concerns for bitrot on those. Found this utility and tried it out on these disks. Just be prepared that it will take a long time, as it performs a full read-write on the whole disk. I usually use it once a year overnight.
Please read how to recommend software in answers, particularly the bits in bold; then edit your answer to follow the guidelines there. Thanks!
– bertieb
Nov 22 at 18:34
Thanks for the suggestion! edited the post.
– noctrex
Nov 30 at 14:01
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f848711%2fdetecting-and-preventing-hfs-plus-ntfs-bit-rot%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
5 Answers
5
active
oldest
votes
5 Answers
5
active
oldest
votes
active
oldest
votes
active
oldest
votes
I’ve researched dealing with this issue for fairly large—20TB+—enterprise-sized storage systems and the consumer reality is this: ZFS based systems is really the only way to deal with this. If data rot is a real concern, then I would recommend having at least one other hard drive for backups that you backup. Not RAID or anything magical, but simply another external drive that is synced using a tool like rsync
if you are comfortable using the command line or Carbon Copy Cloner which is basically an app that performs the same function as rsync
but has a nice user interface.
I did do some searching right now and found ZFS on Linux which sounds interesting; an open source implementation of ZFS for Linux systems. If you are comfortable rolling up your sleeves and setting up Linux/Unix stuff, this could be a potential solution for a do-it-yourself NAS. But I do not have direct experience with it, so can’t speak for it’s long term usefulness in a “production” environment.
1
I've been using ZFS for years now on my home server, with FreeBSD. It's really great but eats as much RAM as you throw at it. ZFS for Linux should be sufficiently stable by now, I'm using it on one of my dedicated servers. No problems there either.
– Daniel B
Dec 4 '14 at 23:41
@DanielB Very good to know! Now I have a new project to spend time on!
– JakeGould
Dec 4 '14 at 23:46
2
I'd add setting it up right is a bit tricky. There's a SF regular who's a wizard at this, and if I recall correctly he suggests having a fast SSD for ZIL and l2arc. You're also going to want a ton of ram on your storage box as Daniel B mentioned.
– Journeyman Geek♦
Dec 5 '14 at 0:16
It'd probably be easier to use FreeNAS to set up ZFS on FreeBSD than doing it from scratch, since you're looking to build a NAS anyway (unless, of course, you prefer to set it up manually so you understand it).
– Suchipi
Dec 5 '14 at 6:25
@Suchipi “Unless, of course, you prefer to set it up manually so you understand it.” For the original poster, maybe that is the best option. But I do Linux/Unix development, systems administration and security work. Even if I setup something the easy way I assure you at some point I will have to “dig deeper” to really get it to work the way a client desires. So I would rather build from scratch; works better for my process.
– JakeGould
Dec 5 '14 at 6:29
|
show 3 more comments
I’ve researched dealing with this issue for fairly large—20TB+—enterprise-sized storage systems and the consumer reality is this: ZFS based systems is really the only way to deal with this. If data rot is a real concern, then I would recommend having at least one other hard drive for backups that you backup. Not RAID or anything magical, but simply another external drive that is synced using a tool like rsync
if you are comfortable using the command line or Carbon Copy Cloner which is basically an app that performs the same function as rsync
but has a nice user interface.
I did do some searching right now and found ZFS on Linux which sounds interesting; an open source implementation of ZFS for Linux systems. If you are comfortable rolling up your sleeves and setting up Linux/Unix stuff, this could be a potential solution for a do-it-yourself NAS. But I do not have direct experience with it, so can’t speak for it’s long term usefulness in a “production” environment.
1
I've been using ZFS for years now on my home server, with FreeBSD. It's really great but eats as much RAM as you throw at it. ZFS for Linux should be sufficiently stable by now, I'm using it on one of my dedicated servers. No problems there either.
– Daniel B
Dec 4 '14 at 23:41
@DanielB Very good to know! Now I have a new project to spend time on!
– JakeGould
Dec 4 '14 at 23:46
2
I'd add setting it up right is a bit tricky. There's a SF regular who's a wizard at this, and if I recall correctly he suggests having a fast SSD for ZIL and l2arc. You're also going to want a ton of ram on your storage box as Daniel B mentioned.
– Journeyman Geek♦
Dec 5 '14 at 0:16
It'd probably be easier to use FreeNAS to set up ZFS on FreeBSD than doing it from scratch, since you're looking to build a NAS anyway (unless, of course, you prefer to set it up manually so you understand it).
– Suchipi
Dec 5 '14 at 6:25
@Suchipi “Unless, of course, you prefer to set it up manually so you understand it.” For the original poster, maybe that is the best option. But I do Linux/Unix development, systems administration and security work. Even if I setup something the easy way I assure you at some point I will have to “dig deeper” to really get it to work the way a client desires. So I would rather build from scratch; works better for my process.
– JakeGould
Dec 5 '14 at 6:29
|
show 3 more comments
I’ve researched dealing with this issue for fairly large—20TB+—enterprise-sized storage systems and the consumer reality is this: ZFS based systems is really the only way to deal with this. If data rot is a real concern, then I would recommend having at least one other hard drive for backups that you backup. Not RAID or anything magical, but simply another external drive that is synced using a tool like rsync
if you are comfortable using the command line or Carbon Copy Cloner which is basically an app that performs the same function as rsync
but has a nice user interface.
I did do some searching right now and found ZFS on Linux which sounds interesting; an open source implementation of ZFS for Linux systems. If you are comfortable rolling up your sleeves and setting up Linux/Unix stuff, this could be a potential solution for a do-it-yourself NAS. But I do not have direct experience with it, so can’t speak for it’s long term usefulness in a “production” environment.
I’ve researched dealing with this issue for fairly large—20TB+—enterprise-sized storage systems and the consumer reality is this: ZFS based systems is really the only way to deal with this. If data rot is a real concern, then I would recommend having at least one other hard drive for backups that you backup. Not RAID or anything magical, but simply another external drive that is synced using a tool like rsync
if you are comfortable using the command line or Carbon Copy Cloner which is basically an app that performs the same function as rsync
but has a nice user interface.
I did do some searching right now and found ZFS on Linux which sounds interesting; an open source implementation of ZFS for Linux systems. If you are comfortable rolling up your sleeves and setting up Linux/Unix stuff, this could be a potential solution for a do-it-yourself NAS. But I do not have direct experience with it, so can’t speak for it’s long term usefulness in a “production” environment.
edited Dec 5 '14 at 21:38
answered Dec 4 '14 at 23:19
JakeGould
31k1093137
31k1093137
1
I've been using ZFS for years now on my home server, with FreeBSD. It's really great but eats as much RAM as you throw at it. ZFS for Linux should be sufficiently stable by now, I'm using it on one of my dedicated servers. No problems there either.
– Daniel B
Dec 4 '14 at 23:41
@DanielB Very good to know! Now I have a new project to spend time on!
– JakeGould
Dec 4 '14 at 23:46
2
I'd add setting it up right is a bit tricky. There's a SF regular who's a wizard at this, and if I recall correctly he suggests having a fast SSD for ZIL and l2arc. You're also going to want a ton of ram on your storage box as Daniel B mentioned.
– Journeyman Geek♦
Dec 5 '14 at 0:16
It'd probably be easier to use FreeNAS to set up ZFS on FreeBSD than doing it from scratch, since you're looking to build a NAS anyway (unless, of course, you prefer to set it up manually so you understand it).
– Suchipi
Dec 5 '14 at 6:25
@Suchipi “Unless, of course, you prefer to set it up manually so you understand it.” For the original poster, maybe that is the best option. But I do Linux/Unix development, systems administration and security work. Even if I setup something the easy way I assure you at some point I will have to “dig deeper” to really get it to work the way a client desires. So I would rather build from scratch; works better for my process.
– JakeGould
Dec 5 '14 at 6:29
|
show 3 more comments
1
I've been using ZFS for years now on my home server, with FreeBSD. It's really great but eats as much RAM as you throw at it. ZFS for Linux should be sufficiently stable by now, I'm using it on one of my dedicated servers. No problems there either.
– Daniel B
Dec 4 '14 at 23:41
@DanielB Very good to know! Now I have a new project to spend time on!
– JakeGould
Dec 4 '14 at 23:46
2
I'd add setting it up right is a bit tricky. There's a SF regular who's a wizard at this, and if I recall correctly he suggests having a fast SSD for ZIL and l2arc. You're also going to want a ton of ram on your storage box as Daniel B mentioned.
– Journeyman Geek♦
Dec 5 '14 at 0:16
It'd probably be easier to use FreeNAS to set up ZFS on FreeBSD than doing it from scratch, since you're looking to build a NAS anyway (unless, of course, you prefer to set it up manually so you understand it).
– Suchipi
Dec 5 '14 at 6:25
@Suchipi “Unless, of course, you prefer to set it up manually so you understand it.” For the original poster, maybe that is the best option. But I do Linux/Unix development, systems administration and security work. Even if I setup something the easy way I assure you at some point I will have to “dig deeper” to really get it to work the way a client desires. So I would rather build from scratch; works better for my process.
– JakeGould
Dec 5 '14 at 6:29
1
1
I've been using ZFS for years now on my home server, with FreeBSD. It's really great but eats as much RAM as you throw at it. ZFS for Linux should be sufficiently stable by now, I'm using it on one of my dedicated servers. No problems there either.
– Daniel B
Dec 4 '14 at 23:41
I've been using ZFS for years now on my home server, with FreeBSD. It's really great but eats as much RAM as you throw at it. ZFS for Linux should be sufficiently stable by now, I'm using it on one of my dedicated servers. No problems there either.
– Daniel B
Dec 4 '14 at 23:41
@DanielB Very good to know! Now I have a new project to spend time on!
– JakeGould
Dec 4 '14 at 23:46
@DanielB Very good to know! Now I have a new project to spend time on!
– JakeGould
Dec 4 '14 at 23:46
2
2
I'd add setting it up right is a bit tricky. There's a SF regular who's a wizard at this, and if I recall correctly he suggests having a fast SSD for ZIL and l2arc. You're also going to want a ton of ram on your storage box as Daniel B mentioned.
– Journeyman Geek♦
Dec 5 '14 at 0:16
I'd add setting it up right is a bit tricky. There's a SF regular who's a wizard at this, and if I recall correctly he suggests having a fast SSD for ZIL and l2arc. You're also going to want a ton of ram on your storage box as Daniel B mentioned.
– Journeyman Geek♦
Dec 5 '14 at 0:16
It'd probably be easier to use FreeNAS to set up ZFS on FreeBSD than doing it from scratch, since you're looking to build a NAS anyway (unless, of course, you prefer to set it up manually so you understand it).
– Suchipi
Dec 5 '14 at 6:25
It'd probably be easier to use FreeNAS to set up ZFS on FreeBSD than doing it from scratch, since you're looking to build a NAS anyway (unless, of course, you prefer to set it up manually so you understand it).
– Suchipi
Dec 5 '14 at 6:25
@Suchipi “Unless, of course, you prefer to set it up manually so you understand it.” For the original poster, maybe that is the best option. But I do Linux/Unix development, systems administration and security work. Even if I setup something the easy way I assure you at some point I will have to “dig deeper” to really get it to work the way a client desires. So I would rather build from scratch; works better for my process.
– JakeGould
Dec 5 '14 at 6:29
@Suchipi “Unless, of course, you prefer to set it up manually so you understand it.” For the original poster, maybe that is the best option. But I do Linux/Unix development, systems administration and security work. Even if I setup something the easy way I assure you at some point I will have to “dig deeper” to really get it to work the way a client desires. So I would rather build from scratch; works better for my process.
– JakeGould
Dec 5 '14 at 6:29
|
show 3 more comments
It's worth noting that Microsoft now has ReFS (resilient file system) on Windows 2012+ and Windows 8.1, which does check integrity. Furthermore, if you run ReFS on a mirrored storage space, it can automatically correct those errors by using bits from the other side of the mirror.
ReFS doesn't support all the features of NTFS, so you'll have to decide if any of the things it's missing are important for the files or workloads you need it for.
Good tip! Now, the only problem: Microsoft.
– JakeGould
Dec 5 '14 at 4:48
Well you don't specify your desired operating system and you listed NTFS so I figured it was fair game that Windows might be acceptable! I may be biased as a primarily Windows sysadmin, but Microsoft's stuff is pretty great lately, especially on the server side. Do you have a specific concern?
– briantist
Dec 5 '14 at 4:50
“Well you don't specify your desired operating system…” I am not the original poster. I am simply a site user who gave you a “+1” for this, said “Good tip!” and explained by only personal issue with it: Microsoft. As for whether this is an issue with the original poster, not my position to say.
– JakeGould
Dec 5 '14 at 4:52
Whoops my mistake! I'm on mobile so it's easier to miss. I'm still curious if you have a specific concern, as it would add to the discussion and would probably be useful to the OP.
– briantist
Dec 5 '14 at 4:55
1
My concern is the original poster mentions only Time Machine (an Apple specific Mac OS X tool) as their backup tool and might only be approaching this issue from the Apple side where sometimes—thanks to Fuse—Mac users format drives in NTFS for cross-platform compatibility and will not be mounting—or managing this—in a Microsoft OS environment that would really be able to handle ReFS. If somehow ReFS is cross platform in usability and resiliency without OS concerns, great! But I have a feeling that might be a stumbling block to it being adopted in this scenario as presented.
– JakeGould
Dec 5 '14 at 4:59
|
show 2 more comments
It's worth noting that Microsoft now has ReFS (resilient file system) on Windows 2012+ and Windows 8.1, which does check integrity. Furthermore, if you run ReFS on a mirrored storage space, it can automatically correct those errors by using bits from the other side of the mirror.
ReFS doesn't support all the features of NTFS, so you'll have to decide if any of the things it's missing are important for the files or workloads you need it for.
Good tip! Now, the only problem: Microsoft.
– JakeGould
Dec 5 '14 at 4:48
Well you don't specify your desired operating system and you listed NTFS so I figured it was fair game that Windows might be acceptable! I may be biased as a primarily Windows sysadmin, but Microsoft's stuff is pretty great lately, especially on the server side. Do you have a specific concern?
– briantist
Dec 5 '14 at 4:50
“Well you don't specify your desired operating system…” I am not the original poster. I am simply a site user who gave you a “+1” for this, said “Good tip!” and explained by only personal issue with it: Microsoft. As for whether this is an issue with the original poster, not my position to say.
– JakeGould
Dec 5 '14 at 4:52
Whoops my mistake! I'm on mobile so it's easier to miss. I'm still curious if you have a specific concern, as it would add to the discussion and would probably be useful to the OP.
– briantist
Dec 5 '14 at 4:55
1
My concern is the original poster mentions only Time Machine (an Apple specific Mac OS X tool) as their backup tool and might only be approaching this issue from the Apple side where sometimes—thanks to Fuse—Mac users format drives in NTFS for cross-platform compatibility and will not be mounting—or managing this—in a Microsoft OS environment that would really be able to handle ReFS. If somehow ReFS is cross platform in usability and resiliency without OS concerns, great! But I have a feeling that might be a stumbling block to it being adopted in this scenario as presented.
– JakeGould
Dec 5 '14 at 4:59
|
show 2 more comments
It's worth noting that Microsoft now has ReFS (resilient file system) on Windows 2012+ and Windows 8.1, which does check integrity. Furthermore, if you run ReFS on a mirrored storage space, it can automatically correct those errors by using bits from the other side of the mirror.
ReFS doesn't support all the features of NTFS, so you'll have to decide if any of the things it's missing are important for the files or workloads you need it for.
It's worth noting that Microsoft now has ReFS (resilient file system) on Windows 2012+ and Windows 8.1, which does check integrity. Furthermore, if you run ReFS on a mirrored storage space, it can automatically correct those errors by using bits from the other side of the mirror.
ReFS doesn't support all the features of NTFS, so you'll have to decide if any of the things it's missing are important for the files or workloads you need it for.
answered Dec 5 '14 at 4:37
briantist
69059
69059
Good tip! Now, the only problem: Microsoft.
– JakeGould
Dec 5 '14 at 4:48
Well you don't specify your desired operating system and you listed NTFS so I figured it was fair game that Windows might be acceptable! I may be biased as a primarily Windows sysadmin, but Microsoft's stuff is pretty great lately, especially on the server side. Do you have a specific concern?
– briantist
Dec 5 '14 at 4:50
“Well you don't specify your desired operating system…” I am not the original poster. I am simply a site user who gave you a “+1” for this, said “Good tip!” and explained by only personal issue with it: Microsoft. As for whether this is an issue with the original poster, not my position to say.
– JakeGould
Dec 5 '14 at 4:52
Whoops my mistake! I'm on mobile so it's easier to miss. I'm still curious if you have a specific concern, as it would add to the discussion and would probably be useful to the OP.
– briantist
Dec 5 '14 at 4:55
1
My concern is the original poster mentions only Time Machine (an Apple specific Mac OS X tool) as their backup tool and might only be approaching this issue from the Apple side where sometimes—thanks to Fuse—Mac users format drives in NTFS for cross-platform compatibility and will not be mounting—or managing this—in a Microsoft OS environment that would really be able to handle ReFS. If somehow ReFS is cross platform in usability and resiliency without OS concerns, great! But I have a feeling that might be a stumbling block to it being adopted in this scenario as presented.
– JakeGould
Dec 5 '14 at 4:59
|
show 2 more comments
Good tip! Now, the only problem: Microsoft.
– JakeGould
Dec 5 '14 at 4:48
Well you don't specify your desired operating system and you listed NTFS so I figured it was fair game that Windows might be acceptable! I may be biased as a primarily Windows sysadmin, but Microsoft's stuff is pretty great lately, especially on the server side. Do you have a specific concern?
– briantist
Dec 5 '14 at 4:50
“Well you don't specify your desired operating system…” I am not the original poster. I am simply a site user who gave you a “+1” for this, said “Good tip!” and explained by only personal issue with it: Microsoft. As for whether this is an issue with the original poster, not my position to say.
– JakeGould
Dec 5 '14 at 4:52
Whoops my mistake! I'm on mobile so it's easier to miss. I'm still curious if you have a specific concern, as it would add to the discussion and would probably be useful to the OP.
– briantist
Dec 5 '14 at 4:55
1
My concern is the original poster mentions only Time Machine (an Apple specific Mac OS X tool) as their backup tool and might only be approaching this issue from the Apple side where sometimes—thanks to Fuse—Mac users format drives in NTFS for cross-platform compatibility and will not be mounting—or managing this—in a Microsoft OS environment that would really be able to handle ReFS. If somehow ReFS is cross platform in usability and resiliency without OS concerns, great! But I have a feeling that might be a stumbling block to it being adopted in this scenario as presented.
– JakeGould
Dec 5 '14 at 4:59
Good tip! Now, the only problem: Microsoft.
– JakeGould
Dec 5 '14 at 4:48
Good tip! Now, the only problem: Microsoft.
– JakeGould
Dec 5 '14 at 4:48
Well you don't specify your desired operating system and you listed NTFS so I figured it was fair game that Windows might be acceptable! I may be biased as a primarily Windows sysadmin, but Microsoft's stuff is pretty great lately, especially on the server side. Do you have a specific concern?
– briantist
Dec 5 '14 at 4:50
Well you don't specify your desired operating system and you listed NTFS so I figured it was fair game that Windows might be acceptable! I may be biased as a primarily Windows sysadmin, but Microsoft's stuff is pretty great lately, especially on the server side. Do you have a specific concern?
– briantist
Dec 5 '14 at 4:50
“Well you don't specify your desired operating system…” I am not the original poster. I am simply a site user who gave you a “+1” for this, said “Good tip!” and explained by only personal issue with it: Microsoft. As for whether this is an issue with the original poster, not my position to say.
– JakeGould
Dec 5 '14 at 4:52
“Well you don't specify your desired operating system…” I am not the original poster. I am simply a site user who gave you a “+1” for this, said “Good tip!” and explained by only personal issue with it: Microsoft. As for whether this is an issue with the original poster, not my position to say.
– JakeGould
Dec 5 '14 at 4:52
Whoops my mistake! I'm on mobile so it's easier to miss. I'm still curious if you have a specific concern, as it would add to the discussion and would probably be useful to the OP.
– briantist
Dec 5 '14 at 4:55
Whoops my mistake! I'm on mobile so it's easier to miss. I'm still curious if you have a specific concern, as it would add to the discussion and would probably be useful to the OP.
– briantist
Dec 5 '14 at 4:55
1
1
My concern is the original poster mentions only Time Machine (an Apple specific Mac OS X tool) as their backup tool and might only be approaching this issue from the Apple side where sometimes—thanks to Fuse—Mac users format drives in NTFS for cross-platform compatibility and will not be mounting—or managing this—in a Microsoft OS environment that would really be able to handle ReFS. If somehow ReFS is cross platform in usability and resiliency without OS concerns, great! But I have a feeling that might be a stumbling block to it being adopted in this scenario as presented.
– JakeGould
Dec 5 '14 at 4:59
My concern is the original poster mentions only Time Machine (an Apple specific Mac OS X tool) as their backup tool and might only be approaching this issue from the Apple side where sometimes—thanks to Fuse—Mac users format drives in NTFS for cross-platform compatibility and will not be mounting—or managing this—in a Microsoft OS environment that would really be able to handle ReFS. If somehow ReFS is cross platform in usability and resiliency without OS concerns, great! But I have a feeling that might be a stumbling block to it being adopted in this scenario as presented.
– JakeGould
Dec 5 '14 at 4:59
|
show 2 more comments
chkbit is a lightweight bitrot detection tool (OS X/Linux/Windows).
chkbit cannot repair bitrot, its job is simply to detect it.
You should
- backup regularly.
- run chkbit before each backup.
- check for bitrot on the backup media.
- in case of bitrot restore from a checked backup.
add a comment |
chkbit is a lightweight bitrot detection tool (OS X/Linux/Windows).
chkbit cannot repair bitrot, its job is simply to detect it.
You should
- backup regularly.
- run chkbit before each backup.
- check for bitrot on the backup media.
- in case of bitrot restore from a checked backup.
add a comment |
chkbit is a lightweight bitrot detection tool (OS X/Linux/Windows).
chkbit cannot repair bitrot, its job is simply to detect it.
You should
- backup regularly.
- run chkbit before each backup.
- check for bitrot on the backup media.
- in case of bitrot restore from a checked backup.
chkbit is a lightweight bitrot detection tool (OS X/Linux/Windows).
chkbit cannot repair bitrot, its job is simply to detect it.
You should
- backup regularly.
- run chkbit before each backup.
- check for bitrot on the backup media.
- in case of bitrot restore from a checked backup.
answered Jan 2 '15 at 16:24
laktak
1,41362340
1,41362340
add a comment |
add a comment |
You can always manually compute checksums with md5sum
and check them periodically, or you can use btrfs, which has an online checksumming feature. On the other hand, it really is kind of redundant and unnecessary since disk drives already have their own error detecting and correcting codes.
And if one day the MD5 checksum doesn't match, then what happens? ZFS allows for snapshots of data so you can recover a valid version of a file. On a normal file system, if the data doesn't check out, the data is gone unless there is a backup.
– JakeGould
Dec 5 '14 at 0:24
@JakeGould, snapshots won't help with that either since all of the snapshots share the same data blocks on disk if they were not modified intentionally.
– psusi
Dec 5 '14 at 4:07
Snapshots would help. You need to investigate how ZFS systems work. They are designed specifically to deal with data rot issues like this and provide a way to alert and restore data.
– JakeGould
Dec 5 '14 at 4:47
@JakeGould, no they would not. The whole point of snapshots is that they do NOT duplicate the data and thus, take up twice the space. ZFS can tell you that the data has become corrupted but unless you configure it with a redundant raid configuration, it can't recover the data. If you are using raid, then that is what provides the backup copy that hopefully is still good, not snapshots.
– psusi
Dec 5 '14 at 14:42
@psusi Running md5sum is not really a good solution since it alone does nothing for recovery which is the whole point of wanting to know where the corruption is. I don't think that checking integrity is redundant because drives have error correction. The world is complicated, hardware breaks all the time. Recovery is the goal.
– hekevintran
Dec 5 '14 at 18:39
|
show 1 more comment
You can always manually compute checksums with md5sum
and check them periodically, or you can use btrfs, which has an online checksumming feature. On the other hand, it really is kind of redundant and unnecessary since disk drives already have their own error detecting and correcting codes.
And if one day the MD5 checksum doesn't match, then what happens? ZFS allows for snapshots of data so you can recover a valid version of a file. On a normal file system, if the data doesn't check out, the data is gone unless there is a backup.
– JakeGould
Dec 5 '14 at 0:24
@JakeGould, snapshots won't help with that either since all of the snapshots share the same data blocks on disk if they were not modified intentionally.
– psusi
Dec 5 '14 at 4:07
Snapshots would help. You need to investigate how ZFS systems work. They are designed specifically to deal with data rot issues like this and provide a way to alert and restore data.
– JakeGould
Dec 5 '14 at 4:47
@JakeGould, no they would not. The whole point of snapshots is that they do NOT duplicate the data and thus, take up twice the space. ZFS can tell you that the data has become corrupted but unless you configure it with a redundant raid configuration, it can't recover the data. If you are using raid, then that is what provides the backup copy that hopefully is still good, not snapshots.
– psusi
Dec 5 '14 at 14:42
@psusi Running md5sum is not really a good solution since it alone does nothing for recovery which is the whole point of wanting to know where the corruption is. I don't think that checking integrity is redundant because drives have error correction. The world is complicated, hardware breaks all the time. Recovery is the goal.
– hekevintran
Dec 5 '14 at 18:39
|
show 1 more comment
You can always manually compute checksums with md5sum
and check them periodically, or you can use btrfs, which has an online checksumming feature. On the other hand, it really is kind of redundant and unnecessary since disk drives already have their own error detecting and correcting codes.
You can always manually compute checksums with md5sum
and check them periodically, or you can use btrfs, which has an online checksumming feature. On the other hand, it really is kind of redundant and unnecessary since disk drives already have their own error detecting and correcting codes.
answered Dec 5 '14 at 0:05
psusi
6,7811422
6,7811422
And if one day the MD5 checksum doesn't match, then what happens? ZFS allows for snapshots of data so you can recover a valid version of a file. On a normal file system, if the data doesn't check out, the data is gone unless there is a backup.
– JakeGould
Dec 5 '14 at 0:24
@JakeGould, snapshots won't help with that either since all of the snapshots share the same data blocks on disk if they were not modified intentionally.
– psusi
Dec 5 '14 at 4:07
Snapshots would help. You need to investigate how ZFS systems work. They are designed specifically to deal with data rot issues like this and provide a way to alert and restore data.
– JakeGould
Dec 5 '14 at 4:47
@JakeGould, no they would not. The whole point of snapshots is that they do NOT duplicate the data and thus, take up twice the space. ZFS can tell you that the data has become corrupted but unless you configure it with a redundant raid configuration, it can't recover the data. If you are using raid, then that is what provides the backup copy that hopefully is still good, not snapshots.
– psusi
Dec 5 '14 at 14:42
@psusi Running md5sum is not really a good solution since it alone does nothing for recovery which is the whole point of wanting to know where the corruption is. I don't think that checking integrity is redundant because drives have error correction. The world is complicated, hardware breaks all the time. Recovery is the goal.
– hekevintran
Dec 5 '14 at 18:39
|
show 1 more comment
And if one day the MD5 checksum doesn't match, then what happens? ZFS allows for snapshots of data so you can recover a valid version of a file. On a normal file system, if the data doesn't check out, the data is gone unless there is a backup.
– JakeGould
Dec 5 '14 at 0:24
@JakeGould, snapshots won't help with that either since all of the snapshots share the same data blocks on disk if they were not modified intentionally.
– psusi
Dec 5 '14 at 4:07
Snapshots would help. You need to investigate how ZFS systems work. They are designed specifically to deal with data rot issues like this and provide a way to alert and restore data.
– JakeGould
Dec 5 '14 at 4:47
@JakeGould, no they would not. The whole point of snapshots is that they do NOT duplicate the data and thus, take up twice the space. ZFS can tell you that the data has become corrupted but unless you configure it with a redundant raid configuration, it can't recover the data. If you are using raid, then that is what provides the backup copy that hopefully is still good, not snapshots.
– psusi
Dec 5 '14 at 14:42
@psusi Running md5sum is not really a good solution since it alone does nothing for recovery which is the whole point of wanting to know where the corruption is. I don't think that checking integrity is redundant because drives have error correction. The world is complicated, hardware breaks all the time. Recovery is the goal.
– hekevintran
Dec 5 '14 at 18:39
And if one day the MD5 checksum doesn't match, then what happens? ZFS allows for snapshots of data so you can recover a valid version of a file. On a normal file system, if the data doesn't check out, the data is gone unless there is a backup.
– JakeGould
Dec 5 '14 at 0:24
And if one day the MD5 checksum doesn't match, then what happens? ZFS allows for snapshots of data so you can recover a valid version of a file. On a normal file system, if the data doesn't check out, the data is gone unless there is a backup.
– JakeGould
Dec 5 '14 at 0:24
@JakeGould, snapshots won't help with that either since all of the snapshots share the same data blocks on disk if they were not modified intentionally.
– psusi
Dec 5 '14 at 4:07
@JakeGould, snapshots won't help with that either since all of the snapshots share the same data blocks on disk if they were not modified intentionally.
– psusi
Dec 5 '14 at 4:07
Snapshots would help. You need to investigate how ZFS systems work. They are designed specifically to deal with data rot issues like this and provide a way to alert and restore data.
– JakeGould
Dec 5 '14 at 4:47
Snapshots would help. You need to investigate how ZFS systems work. They are designed specifically to deal with data rot issues like this and provide a way to alert and restore data.
– JakeGould
Dec 5 '14 at 4:47
@JakeGould, no they would not. The whole point of snapshots is that they do NOT duplicate the data and thus, take up twice the space. ZFS can tell you that the data has become corrupted but unless you configure it with a redundant raid configuration, it can't recover the data. If you are using raid, then that is what provides the backup copy that hopefully is still good, not snapshots.
– psusi
Dec 5 '14 at 14:42
@JakeGould, no they would not. The whole point of snapshots is that they do NOT duplicate the data and thus, take up twice the space. ZFS can tell you that the data has become corrupted but unless you configure it with a redundant raid configuration, it can't recover the data. If you are using raid, then that is what provides the backup copy that hopefully is still good, not snapshots.
– psusi
Dec 5 '14 at 14:42
@psusi Running md5sum is not really a good solution since it alone does nothing for recovery which is the whole point of wanting to know where the corruption is. I don't think that checking integrity is redundant because drives have error correction. The world is complicated, hardware breaks all the time. Recovery is the goal.
– hekevintran
Dec 5 '14 at 18:39
@psusi Running md5sum is not really a good solution since it alone does nothing for recovery which is the whole point of wanting to know where the corruption is. I don't think that checking integrity is redundant because drives have error correction. The world is complicated, hardware breaks all the time. Recovery is the goal.
– hekevintran
Dec 5 '14 at 18:39
|
show 1 more comment
For Windows I've discovered this little program:
"DiskFresh is a simple yet powerful tool that can refresh your hard disk signal without changing its data by reading and writing each sector and hence making your disk more reliable for storage"
It does a full read/write cycle of all the sectors on a disk, so that you can prevent bitrot.
I have some SATA harddisks which I use for archiving purposes, so they are not always connected to the computer. I have them in a plastic enclosure and are kept in a drawer together with some moisture absorbing bags. If and when needed I just slide them in one of those sata disk slots I've installed on the tower. Because they sit for extended periods of time offline, I've some concerns for bitrot on those. Found this utility and tried it out on these disks. Just be prepared that it will take a long time, as it performs a full read-write on the whole disk. I usually use it once a year overnight.
Please read how to recommend software in answers, particularly the bits in bold; then edit your answer to follow the guidelines there. Thanks!
– bertieb
Nov 22 at 18:34
Thanks for the suggestion! edited the post.
– noctrex
Nov 30 at 14:01
add a comment |
For Windows I've discovered this little program:
"DiskFresh is a simple yet powerful tool that can refresh your hard disk signal without changing its data by reading and writing each sector and hence making your disk more reliable for storage"
It does a full read/write cycle of all the sectors on a disk, so that you can prevent bitrot.
I have some SATA harddisks which I use for archiving purposes, so they are not always connected to the computer. I have them in a plastic enclosure and are kept in a drawer together with some moisture absorbing bags. If and when needed I just slide them in one of those sata disk slots I've installed on the tower. Because they sit for extended periods of time offline, I've some concerns for bitrot on those. Found this utility and tried it out on these disks. Just be prepared that it will take a long time, as it performs a full read-write on the whole disk. I usually use it once a year overnight.
Please read how to recommend software in answers, particularly the bits in bold; then edit your answer to follow the guidelines there. Thanks!
– bertieb
Nov 22 at 18:34
Thanks for the suggestion! edited the post.
– noctrex
Nov 30 at 14:01
add a comment |
For Windows I've discovered this little program:
"DiskFresh is a simple yet powerful tool that can refresh your hard disk signal without changing its data by reading and writing each sector and hence making your disk more reliable for storage"
It does a full read/write cycle of all the sectors on a disk, so that you can prevent bitrot.
I have some SATA harddisks which I use for archiving purposes, so they are not always connected to the computer. I have them in a plastic enclosure and are kept in a drawer together with some moisture absorbing bags. If and when needed I just slide them in one of those sata disk slots I've installed on the tower. Because they sit for extended periods of time offline, I've some concerns for bitrot on those. Found this utility and tried it out on these disks. Just be prepared that it will take a long time, as it performs a full read-write on the whole disk. I usually use it once a year overnight.
For Windows I've discovered this little program:
"DiskFresh is a simple yet powerful tool that can refresh your hard disk signal without changing its data by reading and writing each sector and hence making your disk more reliable for storage"
It does a full read/write cycle of all the sectors on a disk, so that you can prevent bitrot.
I have some SATA harddisks which I use for archiving purposes, so they are not always connected to the computer. I have them in a plastic enclosure and are kept in a drawer together with some moisture absorbing bags. If and when needed I just slide them in one of those sata disk slots I've installed on the tower. Because they sit for extended periods of time offline, I've some concerns for bitrot on those. Found this utility and tried it out on these disks. Just be prepared that it will take a long time, as it performs a full read-write on the whole disk. I usually use it once a year overnight.
edited Nov 30 at 14:00
answered Nov 22 at 13:50
noctrex
11
11
Please read how to recommend software in answers, particularly the bits in bold; then edit your answer to follow the guidelines there. Thanks!
– bertieb
Nov 22 at 18:34
Thanks for the suggestion! edited the post.
– noctrex
Nov 30 at 14:01
add a comment |
Please read how to recommend software in answers, particularly the bits in bold; then edit your answer to follow the guidelines there. Thanks!
– bertieb
Nov 22 at 18:34
Thanks for the suggestion! edited the post.
– noctrex
Nov 30 at 14:01
Please read how to recommend software in answers, particularly the bits in bold; then edit your answer to follow the guidelines there. Thanks!
– bertieb
Nov 22 at 18:34
Please read how to recommend software in answers, particularly the bits in bold; then edit your answer to follow the guidelines there. Thanks!
– bertieb
Nov 22 at 18:34
Thanks for the suggestion! edited the post.
– noctrex
Nov 30 at 14:01
Thanks for the suggestion! edited the post.
– noctrex
Nov 30 at 14:01
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f848711%2fdetecting-and-preventing-hfs-plus-ntfs-bit-rot%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown