This program takes a directory as an argument, then walks through the directory looking for duplicate and partially duplicate files. If it an exact duplicate, it deletes one and creates a hardlink to the other, minimizing disk usage. If it finds a pair of files where they contain the same data up to the size of the smaller file, it will prompt if you want to combine them. If you say yes, it will delete the smaller of the files and hardlink to the larger.
This is version 0.1, the ‘beta’ version. It seems to work for me, confirmed on linux and fbsd. It doesn’t work recursively, it doesn’t work on solaris yet, haven’t confirmed MacosX AIX HPUX etc. There are some cosmetic issues, and some fancy command line parameters might be nice. That’s why it’s beta. Give it a try, let me know what should be fixed, added, etc.
WARNING: This software has not been as well tested as it should be, and does a lot of linking and unlinking. It’s also abusive on ram. It may screw up your archive, or even OOM your machine. Use with caution, on a copy of your archive if possible.