SudokuSolver Forum

What is a good way to find duplicates in a file of 81-character puzzle strings, or between two such files? There are too many puzzles for a visual check to be feasible.

If you have access to a Linux (or Unix) installation, and there are not expected to be many duplicates, then you could try:


where <filelist> is a whitespace-delimited list of one or more names of files containing 1 puzzle per line.

This would output a list of duplicates to stdout, which you could then search for (in the original files) and prune manually.

If you've only got access to Windows, then you'll have to look for another way.

_________________
Cheers,
Mike

I might be tempted to read them into Excel and then sort. Scrolling down through the sort, one could spot dupes.

There shouldn't be, but. . .


Hmmm. . . I've been needing to reinstall Red Hat anyway. . .

That's the way I've been doing it, but I have too many to do it that way--just takes too long. That's why I was hoping for something more automated.

In Excel you can use an Advanced Filter (Data > Filter > Advanced Filter) to remove duplicates from a sorted or unsorted column:

_________________
Quis custodiet ipsos custodes?

Normal: [D  Y-m-d,  G:i]     PM->email: [D, d M Y H:i:s]

I tried that out, but evidently I'm not understanding how it works. From the Help file, it appears that it makes a listing somewhere of all the unique ones, but I couldn't figure out where Excel puts the listing or what it does with it.

For example, if you put the column list [1,2,3,9,9,4,2,5,7,3] in the cells A1:A10. Now click menu Data -> Filter -> Advanced Filter. Check both "Copy to another location" and "Unique records only". Put "A1:A10" into the "List range:" entry and put "B1:B10" into the "Copy to:" entry. Click "OK". Voila!

_________________
ADYFNC HJPLI BVSM GgK Oa m

OK, thanks! Can it check for dupes through more than one sheet (presumably by highlighting multiple sheets)?

I don't see a way, though, to check just one column for duplicates but copy the entire row with the unique record to, say, a second sheet. I want to keep the data in the other columns (puzzle number, for instance) from becoming detached from the column with the puzzle data itself. I had hoped that "Criteria range" could be used to do that, but apparently not.

Is 65536 the maximum number of lines Excel can handle in a worksheet? Is the FUNCTIONAL limit smaller?

Just try it with a test workbook and you will find it out.
Even if this is possible I would strongly advice against doing so, unless you are capable of remembering how far you have processed X sheets. See next answer.

DO NOT check "Copy to another location" and leave "Copy to:" empty. Excel then hides all rows having duplicates. Now select the complete sheet (Ctrl+A) and copy (Ctr+C) and paste it to a new blank one. The hidden rows are not included, i.e. selected and copied

Yes, for all versions prior to Excel 2007. Excel 2007 can have 1,048,576 lines.

I do not understand your question!

_________________
Quis custodiet ipsos custodes?

Normal: [D  Y-m-d,  G:i]     PM->email: [D, d M Y H:i:s]