New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
[Looking for]Best method for DeDupe on media server
Howdily doodily,
I have a media server and a FreeNAS storage box to supply a few friends/family with VOD content. We share the responsibility to keep the library updated which often leads to content duplication, <10% I would guess but I'm interested in learning new things, as we all should, and see this as a perfect opportunity to learn how to setup a good deduplication platform or what not.
Any of you had a good success in configuring any platform with dedupe?
Thanks!
Comments
Enforce some basic file naming convention, use ffmpeg to strip out formats and sizes, and script the rest from there? On the whole, streaming content is difficult for this process.
I assume you've read this...?
http://doc.freenas.org/9.10/storage.html#deduplication
It links to this article which is also good:
http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe
Windows server 2012 is good for dedup
Can you reasonably expect the duplicate content to be bit-for-bit identical? If they've been transcoded, rescaled, etc., then dedup won't be useful. If they're DVD rips or whatnot, the encoding settings need to be identical. If they're all "Linux ISOs" / torrents / etc., then duplicates probably will be from the same source, and dedup will work.
DVD rip mainly, with default settings and we all use the same app so that should work.
Unless your friends and you have the exact same taste in 'release' groups, I don't see how dedupe can reduce storage utilization, for encoded multimedia.
Just my armchair understanding of dedup.
I'd love to hear from someone who's tried dedup on multimedia. Maybe check with /r/datahoarder as well.
Check out https://github.com/adrianlopezroche/fdupes
It should do what you want without having to do anything fancy in the filesystem. If the files really are identical, then it'll just setup hard links.
Francisco