DskDitto: Ultra-fast, parallel duplicate-file detector

(github.com)

19 points | by ingve 4 days ago

4 comments

  • jdefr89 21 minutes ago
    Wow. Wasn’t expecting to see this here… I am the author…
  • jauntywundrkind 1 hour ago
    imo fclones is the one to beat, https://github.com/pkolaczk/fclones#benchmarks

    nice that there are actually some ok interfaces here. with fclones, i tend to generate a file of candidates then do a little review to make super certain everything is square / as expected. it's very built for intermediary files a core pattern, which is very unix, very convenient. but it did take me a little while to settle on this, and it felt like i wasn't being offered a ton of options for management out of box. looks like some real attempts to be more user friendly here.

  • akoboldfrying 1 hour ago
    Interested to know what algorithm you use for --fuzzy. Presumably not any of the worst-case-quadratic diff algorithms... MinHash on n-grams?
    • jdefr89 19 minutes ago
      I implemented it quite quickly I use SimHash for now but I need to refactor and possibly replace…