Backup: how do I scan beforehand for corrupted or deleted files?

[Davecom3.6]

Hello everyone, I've been pondering a problem that I'm searching for a solution for, and I wonder if anyone would have any clues to an answer.

It's about computer backups, and I know a proper solution exists, it's just about how to get it to work the way I want. You see, as with anybody else, I have personal files that I couldn't suffer losing, such as my collection of pictures on my computer, for instance. Now, what I do regularly is simple backups of these files to external hard drive and DVD+RW's every x months. My point is that I don't keep different series of backups; I just keep overwriting my external drive and DVD's with the current backup, just to have the security of having a double of my data in a different geographic location in case of fire, flood, theft, etc. (knock on wood)

It occurs to me, however, that this system may not be entirely secure. What happens if, between the last backup and the current one, files on my system are modified in a way I didn't want? Or if someone else uses my computer and deletes files? Or if a file were accidentaly deleted, or even corrupted? In any of these cases, when I'd go do a backup, I'd be backing up a snapshot of My Documents that I don't quite want, because, unbeknownst to me, some of the files are corrupted or deleted, and it's not like I can go back in time and pull out a previous backup from (say) 2002 to recover a lost or damaged file. (The reason I won't keep previous backups is because I don't want to end up with an ever-expanding collection of dozens, and then hundreds of disks to keep in storage somewhere ... and even if I did, how would I know that the oldest DVD's in that pile weren't corroding and becoming unreadable over time...)

So what I want is this: a backup program, or separate application I'd run before the backup, which, when I come to do a backup, remembers the files from the last backup, and somehow compares the last backup to my current system (i.e. compares the previously backed-up files to what I'm about to backup), and tells me, before I proceed to overwrite the last backup with the current one, what has changed, in terms of which files are new since last time (obviously there will be new ones ... pictures, mp3's, docs), which ones are no longer there or corrupted (at which point I can make sure that the absent ones have intentionally been deleted since the last backup, and, if not, I can pull them back out of the previous backup and put them back on my system), before proceeding with the overwriting backup.

I looked at the different typical options, like incremental and differential backup, but they all involve constantly adding the new or modified files to an older initial backup, which always increases the number of discs you end up with. Also, those options will always keep the initial full backup on an old disc that's busy rotting away in the humidity. The way I'm talking about makes you always refresh your backup medium, and if your discs are no longer good, then you just replace them. In short, you would always have a fresh, full backup, and are sure that you're not copying corrupted files or that you're missing any, never to retrieve them again, because it tells you what the problems are before you proceed.

You see, normally, an incremental backup would be what I need, because it keeps an original version of the file, and then it keeps all incremental changes to that file over time, such as modifications. The problem with the incremental backup, however, is the principle that you're always needing more and more space to keep updating the original backup. If I'm backup up to DVD's, I don't want to have a new disc every time I incrementally backup my files. Also, how do you know that the original (1st) base backup, made years ago, doesn't now contain files that have become corrupted? Or the same for some of the earlier increments? Furthermore, there's the problem where, if you need to retrieve a file, you'd have to go back to the beginning, retrieve the original version, and then retrieve all the modifications across all the increments to get back to the version you want to get.

The way I'm talking about would be a rewrite your backup onto the medium (external drive, DVD's, solid state, whatever) every time you backup. That way, you always have a "fresh" set, and are not relying on an original base backup from 14 years ago + monthly increments, and don't have to worry about the original base backup itself having become corrupt after all these years.

Also, an incremental backup won't tell you what's missing or what's corrupt. Let's say you have your My Pictures. One of the folders is pictures from a Florida trip 5 years ago, and in it is a picture of when you had just caught a shark while fishing. That's a pretty important picture, and you want to keep it for all time. Well, meanwhile, you go on 20 more trips since that time, with 20 more folders full of pictures. 5 years later, you don't notice it, because you don't review old pictures very often, but that shark picture has become corrupt and irretrievable in your computer. Or, while viewing the pics from that trip one day, you accidentally delete it and don't notice. Or your friend or family member (for instance, a child) goes through your photos and starts deleting pictures. Or what if they open the picture in MS Paint and vandalise it by drawing a male member on your forehead. Now you come to do your backup of your whole My Pictures. If you simply overwrite the last backup with the current one, you're replacing a good copy of the shark picture with (potentially) a bad one ... or if it's been deleted, you're replacing the folder that had that picture with a folder that's missing it, and you'll never retrieve it again. If, on the other hand, you do incremental backups, then your picture is probably in the original base backup from 5 years ago, or else it's in one of the increments from 5 years ago, but who knows what the state is of the original base backup? Those original discs might be unreadable by now, and then there's the whole hassle of going up through the chain to get the pic. Not to mention that under this scenario, you have to notice yourself that a file is missing, and then take steps to retrieve it, as opposed to an application simply scanning and telling you so whenever you want.

Instead, there MUST be a way to do the full backup each time you back up (so you only have 1 set of discs to go into to find your files, and so that it's always relatively new, and not a rotten, corroded set of DVD's from 14 years ago), and that, before it actually does the backup, it tells you first what's missing, modified, added, corrupt between what you had the last time in your backup, and what you're about to overwrite it with now in your current backup.

I understand that the difficulty in indentifying corrupt files of any kind is that there are many different file types, and no program is so complete as to do that scan. But I guess I'm essentially worried about pictures, since mp3's and documents are less of a problem for me.

So really all I need is any application (not necessarily the backup program) to scan particular folders on my drive (only those I want to back up ... essentially My Documents, and not the whole drive) to check only 3 things:

1) which files have been modified (regular Windows xp search can do this if you advanced search for "files modified between ____ and _____", the 2 given dates being, say, the date of my last backup and the date of the current one);
2) which files are missing since the last backup (all this would involve is checking the files against a list ... seems simple ... and manually looking in Recycle Bin is obviously not enough because you may have emptied the Recycle Bin since last time); and
3) which files are corrupted

With the knowledge of these 3 things, I can take action to replace lost or corrupted files by taking them back out of my last backup. Then, I can safely proceed with the current backup.

I'm running XP MCE 2005. Does anyone have a solution to this? THANK YOU!!!

 

[Dr. Chauncey]

Hello Davecom3.6,

First and foremost, welcome to the Tech Support Guy forums! If I understand correctly from your rather lengthy post, which is well written and leaves little room for misinterpretation, you require a fail-proof method to conduct data backups on a regular basis. I candidly admit that I conduct my own backups in such a method as you have described and will be glad to offer the knowledge that I have acquired through the years.

I whole-heartedly agree that personal electronic memories are of irreplaceable value and by their nature, capture timeless moments in a way physical pictures can not. I know I've lost my share of electronic data in unfortunate accidents, catastophic system failures, FBI raids, etc. and would leap at the opportunity to prevent such loss from visiting somebody else's life. After the last time such an event occured, which cost me dearly, I vowed to conduct timely backups and never be deprived of the files I hold most dear again. To this day, the files lost fill my memory with an empty void for which there is no consolation and I would give anything to have them back.

Early on, I puzzled over the very same quandry you have presented here, unknowingly entering into a world of fear, uncertainty, and paranoia. With great worry I wondered about these questions and more: Would my data ever be truly and undeniably safe? In this universe does there really exist a method to secure this data from destruction worthy of my trust? But with my questions came answers and after countless minutes, infinite hours, sleepless days, and relatively few years analyzing the inner workings of the magnificent data machine, so many of us take for granted everyday, I have the results I will present to you now.

My search began with finding the most suitable containment device to store this data on, far, far away from the instability of any computer. Like you, I decided a Hard disk, external and separate from the others, was the most suitable way and I settled upon a manufacturer that produces these disks of the most robust design. Professional courtesy dictates that I don't repeat the name here but I'm sure that like me, you already know. I purchased several of these monstrous book-like devices reaching a terabyte in capacity (1,073,741,824KB) and transferred my files immediately.

A few months passed before I began to have doubts. I started asking myself; "What if?" Coming to the same conclusions as your post above, I delved deeper into the information. I looked at my precious data until I saw the megabytes, I meditated until the megabytes were kilobytes, I forced myself to stay awake for days eating colorful mushrooms until those kilobytes had turned to bytes, until finally they became bits and I came face-to-face with true enlightenment. I could see the data, not only as a graphic manifestation on my computer monitor, but as an endless stream of 1's and 0's. I could see the binary signature of my data and it was the most beautiful sight ever to befall my eyes!

It was also at this moment that it came to me: What if there was a way to compare the 1's and 0's as they were being copied to my external disk, a way that would ensure the file I copied was identical to the one on my computer? As luck would have it, I already possessed the software to conduct such a comparison in Windows XP. Microsoft calls it Cyclic Redundancy Check (CRC), a method as I had imagined before that matched the binary signature of both the copy and the original file to ensure no corruption occured during the transfer.

Then I devised a method to keep these backups from reaching unfathomable size. I would rewrite the backups from my computer to the exact same files on the external disk. By doing so, I would have a constant up-to-date copy of everything on my computer. But what to do if the original was vandalized by the tempermental machine? How could I stop this abomination from overwriting my perfect copy? By copying the files to the exact same directory, I was presented with this message which gave me two choices: This folder already contains a file named 'Photoshop-edited Picture of me with a Shark'. Would you like to replace the existing file... with this one... Yes or No? It's then that it all came together; I could click 'No' everytime as I manually observed the transfer progress and only my new data would be copied.

Now I'm tired so I'm going to retire, but I sincerely hope this helps.

 

[The_Oracle]

I could see the binary signature of my data and it was the most beautiful sight ever to befall my eyes!

well, aren't you the poet

as for corrupted data, i recommend to use MD5, a widely accepted cryptographic hash function, to verify the integrity of your precious files/archives. here's a free hash calculator supporting MD5 and many other algorithms:

http://www.slavasoft.com/hashcalc/index.htm

 

[Davecom3.6]

Guys (Dr. Chauncey and The_Oracle), I would like to offer my deepest and most profuse thanks for taking the time to read my post and answer with very helpful advice (I was worried I'd have to start over with a shorter post to entice people to listen to my problem).

So, just thinking out loud about the CRC, here are a few points I'm still wondering about:

1) It appears the system you suggest would actually preclude using a backup program, since the CRC needs to do a file-by-file check as it's copying, whereas a backup program wants to basically dump a continuous lump stream of data. And therefore, this system also precludes backing up to optical media, as without a backup program that will split files to fully fill up DVD's for you, it would probably be inconvenient or impossible for the CRC to keep asking you to switch discs to check the files you're copying. But that's fine; I could do one simple-copy CRC-type backup, and 1 backup using a backup program for the DVD+RW set.

2) Do I have the following information right?: a) for files that have remained exactly the same from the last backup to the current one, no action is taken, b) for files added since the last backup, the new files get added to the external drive, c) for files modified since the last backup, the system asks you if you want to overwrite them or not, then you can go check the source file to see if it was a modification you actually did intentionally or not, and decide whether to overwrite the file on the external drive with the current one, d) for files that are missing on the source drive but present on the external drive, it lets you know about the discrepancy, and you can decide if it was a file you actually wanted to delete or not, in which case it would keep the version on the external drive intact (actually, what DOES it do in this case?), and e) for files that have become corrupted since the last backup, the same as c).

3) Also, not sure if I'm thinking straight about this, but do we need to worry about the external drive going bust? I know we've accomplished one of my goals, i.e. that we have only 1, full backup in total, and not many series of backups, but what about my other goal, i.e. that that 1 full backup be completely fresh? In other words, in 2 a) above, for the files that are to remain the same, if the external drive's version of the file is not going to be overwritten, and remain completely unchanged, then wouldn't there be a risk that eventually it would become corrupt just sitting there unchanged and unchecked? Or no... maybe the CRC, in checking every single file against the source version, will tell you there's a difference, and when you go check it out, if the source file is fine, then the external drive's version of the file must be the one that's corrupt. (Unless there's the extremely unlikely coincidence of both the source and external versions becoming independantly corrupt in exactly the same way, and the CRC would not flag them as being different, but I'm not too worried about this.)

4) Ok, so now, how does it work? How would I activate the CRC as I'm overwriting the external version with my source one?

5) Oracle, I'm going to give the app a try, as an additional scan to check for corruption. Is there any danger to scanning files with so many different applications, like this one, virus scan, adware scan, etc.?

6) Chauncey, were you making a Matrix reference in your post? If so, cool!

THANKS AGAIN!

 

[Dr. Chauncey]

Davecom3.6,

Welcome back! It was not the slightest inconvenience to respond to your post and you are to be commended yet again for your superior skill and mastery of the written english language. On to business.

1) It's true that I have presented you with the most cost-effective method for backing up data, whereby no extra software would be a neccessity. However, there are many programs on the market for conducting these backups. I'm sure many of these are superior to the methods I use, however, I cannot bring myself to place my precious memories in the hands of a program written to make a buck. I'm sure some of these were carefully written and function extremely well, but I simply can't take that chance.

2) The majority of this section was, for the most part, correct, with some slight variances: a) For files that remained the same since the last backup, it will ask you whether or not you wish to replace them. Furthermore, the computer makes no distinction, in this case, between the files you have described in "c)" or "e)" either. b) You are, in whole, correct on this point; files added since the last backup will be copied to your backup media without question or interference. c) See "a)" d) The computer will not inform you of any discrepancies of that sort. If the file is missing from your computer, but is present on the destination drive, no action will be taken. It will not notify you of any missing files. I apologize for this, as it seems in my original post I failed to address all of your backup needs. e) Again, for files existing on the computer and also on the backup media, it will ask you whether or not you wish to replace them regardless of any changes or corruption that may have occured since the last backup.

3) You are correct, also in that the harddrive has now become the weak point in this operation. Over time, this disks do start to decompose, although only in a completely electronic fashion. The platters within the disk will begin to accumulate "bad sectors" and the information in these sectors will be rendered completely unreadable. The best and most effectual way to prevent this is to replace the disk every two to three years, although there is no way of telling when such an error is likely to occur. In addition to what I've mentioned before, there are a few other advantages of using a hard disk over CD's or DVD's. For instance, if both were to become damaged in some way, there are a lot more options for recovering the files off of a harddrive than from optical media.

4) One of the beautiful things about CRC is that it runs all the time. It is not a simple process that must be started and stopped, it is a built in feature of the finest operating system Microsoft has produced to date, Windows XP.

5) I commend you on your decision to use MD5, and eagerly wait to hear of your results. Please post back after what you feel has been an appropriate trial period and share your experience with us.

6) I may have. Which pill would you have chosen if put in Neo's position?

You're most welcome and we hope to hear from you again soon!

 

[Davecom3.6]

Dr. Chauncey, thank you very much for the compliment; I work with the written English language on a daily basis, which may explain any positive or negative abnormal written ease with it. Thanks again for the reply. If you don't mind, I have a couple more wrap-up questions to put the final period at the end of the sentence.

1) No problem; I'll just use both methods, simply to have an additional backup on an optical medium that I can store offsite more easily than the external hard drive.

2) + 4) So if I understand correctly, this isn't an extraordinary procedure, but simply dragging folders from the source drive and dropping them in the destination drive, where they will automatically try to overwrite themselves on the same folders in the destination drive, and ask you, file by file, whether you want to replace files you already have on the destination drive. You also seemed to allude to the fact that it didn't matter whether the files on the source drive are the same or different (modified or corrupted) as the ones already on the destination drive; in all cases, it will ask you whether you want to replace them or not. I'm not sure if this system properly responds to what my idea is, because it seems now like it won't make a distinction between files that have changed and those that haven't. My idea was that if a file hasn't changed, the system would tell me so, and I'd leave it be. If a file has changed since last time (and there shouldn't be many of these; there should mostly only be NEW files as opposed to modified ones in My Documents), I could manually go check the source file, see if it is a modification I made intentionally, and if not, fix the problem or keep the backed up version intact, and if the source file is fine, that would be an indication that the backed up version is somehow corrupt, and take steps to remedy the issue.

I therefore also understand that the system doesn't account for missing files. That's the thing: what if you or someone accidentally deleted a file on your source drive? Isn't there some way for an application to compare the source to the destination to tell you so? On the other hand, maybe you intentionally deleted a file, and really do want your backup to reflect the fact that that file is not part of your collection. Without something to check for missing files, it would never be brought to your attention, and you could never synchronize your source to your destination so that both versions are the total, real, "official" version of your collection. Wouldn't it be such a simple function to check file lists and flag missing files? I can't believe there's nothing like this out there. Would you happen to know of something that will do the trick?

3) This point is actually ok; by asking you whether you want to replace files, it's actually verifying, as a side benefit, that those files are present on the external drive. You'd be able to notice, at the first instance of doing a backup where the copying were going haywire, that the drive is now bad, and take steps to replace it.

5) I'd run the scan before going forward with the backup, as an extra indicator of corrupt files.

Thank you ever so much for your patience with this issue.

 

[Davecom3.6]

.... anyone?

To recap, I'm looking to do a proper backup of my system, but beforehand, I need to scan for 2 things:

- which files have gone missing since the last backup (accidental or intentional delete) ... need some app that would scan the current list of files and compare it to the list of files at a previous moment, or the list of files in the previous backup and flag the missing ones

and

- which files have become modified (accidental, intentional, vandalism by a third party, etc. ... and this is also a method to pick up files that have become corrupted)

Thank you to anyone who can help

 

[stantley]

Try SyncBack, it can do a simulation run and create a report before you do the actual backup. They call where you're backing from the source and where you're backing up to the destination.

So new files show up as 'Not in Destination, Source Copied', files that have been deleted show up as 'Not in Source, Skipping'. Files that have a date or size change show up as 'Source copied'. Files that are the same don't show on the report. So you can look through the report to double check things and then do the actual backup.

File attributes can tell you when a file has changed, but not why. Only you are going to know the difference between intentional and accidental. File corruption is such a rare event that it's not worth worrying about, keep several backups and one should have a good file.

And if you have 'vandalism by a third party' you got bigger problems that go beyond backup files.

 

[Davecom3.6]

Oh my Non-Denominational Higher Power, that sounds AWESOME, exactly what I need. As for corruption, the download website mentions it has optional MD5 checksum, which might pretty well cover scanning for corrupt media files. I will let you know in a few days (hectic weekend ahead...)!

fingers crossed, thanks a bunch

 

[Davecom3.6]

So far so good! I downloaded SyncBack and installed it (very small application, very nice and clean interface, very non-invasive, does not add registry commands to startup), and created a profile. Immediately I realized that this software would respond to EVERY SINGLE ONE OF MY CONCERNS IN THIS POST. By comparing a previous backup of your files to the current one, you can set it to copy new files, and prompt you for every file that has been modified AND for ones that are missing from the source, but present on the destination. So one would be able to pick up files accidentally deleted. Very neat!

Thanks a million!