1. Computer problem? Tech Support Guy is completely free -- paid for by advertisers and donations. Click here to join today! If you're new to Tech Support Guy, we highly recommend that you visit our Guide for New Members.

WTD: Advanced De-duping and backup strategies

Discussion in 'All Other Software' started by bsacco, Aug 30, 2017.

Thread Status:
Not open for further replies.
Advertisement
  1. bsacco

    bsacco Thread Starter

    Joined:
    Jun 11, 2003
    Messages:
    846
    I have TBs of data sitting on several different External HDs.

    Does anyone have a link or know of the best way to approach de-duping all my files from all these different HDs into a sustainable and cohesive backup solution?
     
  2. zx10guy

    zx10guy Trusted Advisor Spam Fighter

    Joined:
    Mar 30, 2008
    Messages:
    5,955
    Is this for personal or business?
     
  3. bsacco

    bsacco Thread Starter

    Joined:
    Jun 11, 2003
    Messages:
    846
    personal
     
  4. bsacco

    bsacco Thread Starter

    Joined:
    Jun 11, 2003
    Messages:
    846
    Here are my initial thoughts:
    1) I use Crashplan Pro. So, I was thinking that I could leverage the service to create a backup plan in the cloud? I believe they de-dupe upon upload. Though, it really doesn't solve the hierarchy of how my various External HDs are organized. I would first have to come up with a scheme for MAJOR parts of the data, i.e. PHOTOS, VIDEOS, PC BACKUPS, MUSIC, etc.... I suppose once I create specific folders in the cloud using Crashplan I could mirror that on my External HDs? AM I going in the right direction here?
     
  5. zx10guy

    zx10guy Trusted Advisor Spam Fighter

    Joined:
    Mar 30, 2008
    Messages:
    5,955
    You have to find out where they do the de-duplication. There are number of places where they can do it. At the source (you), inline, or post processing after receiving the data. I'm going to assume the de dup is only going to happen some where along the data path between you and Crashplan. Which means when you pull data from Crashplan to load on your local or external hard drive, the data will no longer be de dup'd. When data is de dup'd, the data is not directly readable. It requires the same de dup mechanism to translate the de dup'd data into something you can work with.
     
  6. bsacco

    bsacco Thread Starter

    Joined:
    Jun 11, 2003
    Messages:
    846
    Very interesting. I did not know that. I guess it encrypts to save then it de-encrypts to release the data. Hummmmmmm. I guess I'm going to have to find a de-duping software solution locally on my PC. Can you recommend good de-duping software? Secondly, I'm going to have to figure out the correct sequence of how to de-dupe all my drives. And finally, I need to come up with a hierarchy strategy of how to organize the DATA, i.e. Photos, videos, backups, music, etc.... Do you have any suggestions of best practice?
     
  7. zx10guy

    zx10guy Trusted Advisor Spam Fighter

    Joined:
    Mar 30, 2008
    Messages:
    5,955
    I'm not aware of any consumer grade de-duping products out there. De-duping has been an offering in the enterprise business space. Not all de-duping technology is created equal. Some do a better job than others. One of the best on the market at de-duplication is an EMC acquired company called Data Domain. De-duplication is also included in some SAN arrays from EMC, NetApp, Compellent, etc.

    I almost think you're better off just compressing the data either through native Windows or with compression software such as WinZip.

    Also a point of clarification, deduplication is not the same as encryption.
     
  8. bsacco

    bsacco Thread Starter

    Joined:
    Jun 11, 2003
    Messages:
    846
  9. zx10guy

    zx10guy Trusted Advisor Spam Fighter

    Joined:
    Mar 30, 2008
    Messages:
    5,955
    I've said from the beginning dedup is in reference decreasing the amount of data required to be stored. You didn't specify finding duplication of file. In a home situation, I can't imagine any duplication or multiples of the same file being a significant problem where removal of those duplicated files would create a substantial savings in the amount of storage space.

    I dealt with a client who wanted file auditing/intelligence because they had petabytes of data out there spread across multiple geographical locations across servers, laptops, and desktops. They wanted to identify where files are being duplicated and then be able to make a decision on if the duplicated files would be deleted. I suggested they use a tool from NTP Software to accomplish this task. But again, this organization has petabytes of data with thousands of users where there is a strong likely of significant number of sizeable duplicated files.
     
  10. bsacco

    bsacco Thread Starter

    Joined:
    Jun 11, 2003
    Messages:
    846
    Why do you keep pointing me towards a commercial enterprise-wide solution? Dude......stop....please. I just need to delete duplicate files in the most efficient FREE manner. Very simple. I need smart, thoughtful and creative thinking....Not big $ commercial solutions.
     
  11. Sponsor

As Seen On
As Seen On...

Welcome to Tech Support Guy!

Are you looking for the solution to your computer problem? Join our site today to ask your question. This site is completely free -- paid for by advertisers and donations.

If you're not already familiar with forums, watch our Welcome Guide to get started.

Join over 733,556 other people just like you!

Loading...
Thread Status:
Not open for further replies.

Short URL to this thread: https://techguy.org/1195557

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice