Is there a file system that does error correction?
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
|
show 4 more comments
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
|
show 4 more comments
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
backup filesystems file-management error-correction
edited Nov 23 '18 at 1:28
phuclv
8,94563789
8,94563789
asked Oct 21 '12 at 7:32
Alfatau
269135
269135
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
|
show 4 more comments
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
1
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
|
show 4 more comments
3 Answers
3
active
oldest
votes
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
add a comment |
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
add a comment |
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f490527%2fis-there-a-file-system-that-does-error-correction%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
add a comment |
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
add a comment |
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
answered Jul 28 '13 at 14:27
LawrenceC
58.7k10102179
58.7k10102179
add a comment |
add a comment |
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
add a comment |
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
add a comment |
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
answered Feb 22 '17 at 20:01
Jeroen
111
111
add a comment |
add a comment |
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
add a comment |
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
add a comment |
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
answered Nov 22 '18 at 15:44
phuclv
8,94563789
8,94563789
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f490527%2fis-there-a-file-system-that-does-error-correction%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13