I have a fellow data hoarder friend who was notified recently that G Suite storage would no longer offer unlimited storage next year so we’re looking at replicating our video collections together so we have some form of redundancy if we either have a total loss (we both run UNRAID and have setup a VPN). We can read and write to each other’s collection, but, the challenge is, there’s a good bit of overlap among our collections and we have over 120TBs to filter through. Manually, we’ve dumped our file info to a spreadsheet that we’re sorting through, but it’s so bloody time consuming. Any suggestions on apps that we could run to index our collections and pick the best version of duplicate movie files (work smart over working hard on this)?
We can run a windows VM if needed based on the app recommendation, but would prefer to run something native to Linux in either a VM or as a docker container. What would you do in our shoes? Also, thought about doing a local data transfer of our collections, but we don’t have the available storage to do that yet. Hoping to snag some more 16TB WD easy store drives for black Friday! 🙂