How do you find duplicate images that are not identical?

Use a perceptual hash rather than a checksum — but ship it at a Hamming threshold of 2, not the 10 everyone hardcodes: at 10 a 63-bit pHash matched 67.4% of 6,000 known duplicate pairs and returned 7.7 false positives for every true match, while SHA-256 matched 0 of them. Not 0.1%. Zero, across all fifteen transformations.

The corpus is synthetic, and that has to be said before any number below. I generated 400 images and applied fifteen transformations I chose myself, so every recall figure describes those transformations rather than whatever your CMS, your users and your scrapers do to a file. The transferable finding is the relative pattern by transformation — which hash survives a re-encode, a crop, a rotation — and that pattern is a property of what each hash measures, not of my test set. No absolute threshold travels.

Hardware: Apple M3, 16 GB, macOS 26.4.1. Python 3.14.6, Pillow 12.3.0, no numpy, no imagehash, no model, no network. Every hash is implemented below in about ten lines so you can see exactly what is being compared. Timings are the best of three runs, single core, in process.

The short answer

• A checksum finds byte-identical files and nothing else. SHA-256 matched 0 of 6,000 (original, variant) pairs. Re-saving a PNG as a JPEG at quality 95 — visually indistinguishable — changes every byte.
• pHash is the most precise per bit and the least robust geometrically. At Hamming ≤ 10 it recovered 95.6% of photometric changes (re-encode, resize, brightness, greyscale) and 25.2% of geometric ones (crop, rotate, flip, watermark). dHash recovered 93.6% and 43.0% of the same two buckets.
• The default threshold of 10 is far too loose. Sweeping it, pHash peaked at F1 0.4486 at a threshold of 2, where precision was 0.4502; at 10 precision had fallen to 0.1144 — 31,325 false positive pairs against 4,046 true ones.
• Nothing survives a rotation, and no better hash will fix that. aHash, dHash and pHash recovered 3.7%, 10.2% and 0.0% of 90-degree rotations. Indexing all eight dihedral orientations of each image took rot90 and horizontal-flip recall from 0.000 to 1.000, for 8× the storage and 6.6× the false positives.
• pHash is not the slow one — the JPEG decoder is. pHash ran at 2,594 images/sec against dHash's 9,469, but decoding the PNG cost 0.924 ms against pHash's 0.386, so end to end pHash is only 27% slower than dHash.

This is the image counterpart to near-duplicate detection over text, where MinHash plus LSH cleared 100,000 documents in 1.09 seconds. The conclusions rhyme; the failure modes do not, because text edits are local and image transformations are global.

What exactly is in the corpus?

400 originals at 384×384, in four families of 100: gradient (a rotated linear ramp between two random colours, with a circle outline), shapes (6–14 random ellipses, rectangles and triangles on a light ground), noise (three channels of Image.effect_noise, blurred 0–2 px, blended 35% toward a random colour) and text (random syllable words in the default font on white). Each was written as a PNG and then transformed fifteen ways:

That is 6,400 images and 6,000 known duplicate pairs. Comparing everything with everything gives 20,476,800 pairs, of which 20,428,800 join images from different originals and count as negatives.

Does a checksum ever work?

Only if the bytes never change, which in practice means only if nothing has touched the file. Across all 6,000 pairs SHA-256 matched zero. Every one of the fifteen transformations scored 0/400, including jpeg95, which is a re-encode you would struggle to see.

A cryptographic hash is designed so that a one-bit input change scatters the output — exactly what you want for integrity and exactly what you do not want for similarity.

What are the four fingerprints?

def ahash(im):                                # 8x8 grey, bit = pixel > mean
    p = list(im.convert("L").resize((8, 8), Image.Resampling.LANCZOS).tobytes())
    avg, v = sum(p) / 64.0, 0
    for i, x in enumerate(p):
        if x > avg: v |= 1 << i
    return v

def dhash(im):                     # 9x8 grey, bit = pixel > its right neighbour
    p = list(im.convert("L").resize((9, 8), Image.Resampling.LANCZOS).tobytes())
    v = i = 0
    for y in range(8):
        r = y * 9
        for x in range(8):
            if p[r + x] > p[r + x + 1]: v |= 1 << i
            i += 1
    return v

pHash resizes to 32×32, takes a two-dimensional DCT-II, keeps the low-frequency 8×8 block, drops the DC term and sets each bit on whether that coefficient beats the median of the remaining 63. The usual complaint is that the DCT is expensive. It need not be: only the first eight frequency columns are ever read, so the separable transform does 10,240 multiplies rather than 2·32³ = 65,536.

_NP, _NK = 32, 8
_COS = [[math.cos(math.pi * (2*n + 1) * k / (2*_NP)) for n in range(_NP)]
        for k in range(_NK)]

def phash(im):
    p = list(im.convert("L").resize((_NP, _NP), Image.Resampling.LANCZOS).tobytes())
    tmp = [[0.0] * _NK for _ in range(_NP)]
    for y in range(_NP):                          # rows -> first 8 freq columns
        row, t = p[y*_NP:(y+1)*_NP], tmp[y]
        for k in range(_NK):
            ck = _COS[k]; s = 0.0
            for n in range(_NP): s += row[n] * ck[n]
            t[k] = s
    c = [0.0] * 64
    for k in range(_NK):                          # columns -> first 8 freq rows
        ck = _COS[k]
        for j in range(_NK):
            s = 0.0
            for n in range(_NP): s += tmp[n][j] * ck[n]
            c[k*8 + j] = s
    rest = sorted(c[1:])                          # drop DC
    med, v = (rest[30] + rest[31]) / 2.0, 0
    for i in range(1, 64):
        if c[i] > med: v |= 1 << i
    return v

The colour histogram is 16 bins per channel over a 64×64 copy, normalised, and compared with half the L1 distance so it lands in [0, 1].

Which hash survives which transformation?

This is the table. Each row is scored on its own sub-corpus of 800 images — the 400 originals plus the 400 variants of that one transformation — giving 319,600 pairs of which 400 are true. Bit hashes are thresholded at Hamming ≤ 10, the histogram at ≤ 0.05. Cells are precision / recall / F1.

Four things in there are worth stating on their own.

pHash is not uniformly the best hash. It is the best on every photometric change and the worst on every geometric one. dHash beat it 0.73 to 0.33 under a watermark and 0.91 to 0.56 under a 3% crop; aHash beat both under a 10% crop, 0.40 to 0.12 and 0.00. The folklore that pHash dominates comes from benchmarks made of re-encodes.

A 3% crop is survivable and a 10% crop is not. All three hashes downscale to a fixed grid, so trimming the edges shifts every sample point. Rotating by two degrees also costs more than you would guess — pHash 0.63, aHash 0.72 — and a 90-degree rotation is total, at 0 of 400.

The colour histogram is the only rotation- and flip-invariant fingerprint here, at 1.00 recall on both, because it throws away all spatial information. It is correspondingly useless under a brightness change: 0.00.

Precision is bad everywhere. Even pHash's best cell is 0.42. That is not noise; it is the real result, and the next two sections are about it.

What threshold should you use?

Everyone hardcodes 5 or 10. Here is the sweep, scored on the full 6,400-image corpus: 6,000 true (original, variant) pairs against 20,428,800 cross-original pairs.

Best F1: aHash 0.3655 at threshold 1, dHash 0.3880 at threshold 1, pHash 0.4486 at threshold 2. Every optimum is at 1 or 2. Going from 2 to 10 bought pHash 23 points of recall and cost it 34 points of precision — 4,046 true pairs against 31,325 false ones.

Note also what the sweep does not do between consecutive odd and even rows for pHash: threshold 0 and 1 are identical, 2 and 3 are identical, and so on all the way up. Every pHash has exactly 32 of its 63 bits set, because the bits are set by comparison against the median of those same 63 coefficients. Two hashes with equal popcount always differ in an even number of positions, so pHash Hamming distances are always even and half your thresholds are dead. Setting 5 instead of 4 does nothing at all. I have never seen this mentioned.

Which images collide when they should not?

Precision never gets good, and the reason is specific. Of the 3,275 false positive pairs pHash produced at threshold 2, 1,991 were gradient × gradient and 1,219 were noise × noise — 98% from two of the four families. Images with almost no mid-frequency structure all look alike once you downscale to 32×32, take the low-frequency block and throw away colour. The text family, which has real structure, contributed 10.

The classic version is the near-empty image, and it is worse than its reputation. I generated 80 genuinely different low-detail images — 40 near-solid pastel grounds each with one small square somewhere, 40 white pages with a few lines of different text — giving 3,160 pairs that are all true negatives:

On the 40 text-page scans alone aHash called 32.2% of the pairs duplicates at ≤ 5 and 86.5% at ≤ 10. aHash keeps 64 bits of "is this pixel above the average pixel", and on a white page with a little text nearly every answer is the same. If your corpus is scanned documents, receipts or screenshots, aHash will merge them.

The fix is a second stage. Take pHash ≤ 2 as a candidate generator and verify each survivor with something that measures what pHash discarded — colour:

A colour check takes precision from 0.45 to 0.99 — 13 false pairs out of 20.4 million — and costs 9 points of recall, almost all of it on grey (1.000 to 0.362) and bright (0.510 to 0.155). That is the right trade if a recoloured copy is not a duplicate to you, and the wrong one if it is: the same deliberate choice as picking a level in quantising embeddings, where the accuracy you give up is the point rather than an accident.

Can anything survive a rotation?

Not a hash — an index. Store the pHash of all eight dihedral orientations of every image and match against the minimum:

Perfect on both, at 8× the index and 6.6× the false positives — the same trade you make when packing more into a fixed budget, as with the wasted cells in a sprite atlas. Rotation invariance is bought, not computed.

How fast are they, and what breaks at 100,000 images?

400 pre-decoded 384×384 images, best of three:

pHash is 3.6× dHash in isolation. It is also 2.4× faster than decoding the image it runs on, so the honest end-to-end comparison is 1.310 ms against 1.030 ms — 27%. Choosing dHash to save time is optimising the wrong term.

Comparing everything with everything is the real wall. Measured, pHash at Hamming ≤ 2:

Both indexes lost no recall at all. Splitting the 63-bit hash into four bands and requiring one band to match exactly is not an approximation here: by the pigeonhole principle any pair within Hamming 3 must agree on at least one of four bands, so at a threshold of 2 the band index is exact, and the measured recall against brute force was 1.0000 at every size. The BK-tree is exact by construction and visited 82 nodes per query at n = 6,400 against 6,399 for a linear scan — but it is slower than the band index at every size, and it is a pointer structure rather than a GROUP BY.

Brute force ran at 14.29 M pairs/sec. From that rate — calculated, not measured — 10,000 images is 3.5 seconds, 100,000 images is 5.8 minutes, and a million images is 500 billion pairs and 9.7 hours. Ten times the images is a hundred times the work, so the index is not an optimisation you add later.

What I would ship

pHash at Hamming ≤ 2, looked up through a four-band prefix index, with all eight orientations stored if your users rotate things, and a colour-histogram check on every candidate before anything is merged or deleted. Threshold 2, not 10 — and not 5, which on pHash is silently the same as 4. Expect to catch re-encodes, resizes and greyscale conversions almost perfectly, most of a small rotation, and none of a 10% crop.

And treat any published threshold, including this one, as a starting point to be re-swept on your own images. The sweep took 2.1 seconds.

Check it yourself

imgdup.py needs Python 3.11+ and Pillow, nothing else. It generates 150 originals and 15 transformations of each in a temp directory, computes all five fingerprints, prints the recall table and the threshold sweep, then deletes every image it made. About 35 seconds.

python3 -m pip install Pillow
python3 imgdup.py

#!/usr/bin/env python3
"""imgdup.py -- perceptual image hashing measured against KNOWN duplicates.
Python 3.11+ and Pillow only. No network, no models, ~35 s, cleans up after itself.
The corpus is SYNTHETIC: the transformations are mine, so the recall figures
describe THESE transformations. The relative pattern is what transfers."""
import hashlib, math, os, random, shutil, tempfile, time
from PIL import Image, ImageDraw, ImageEnhance, ImageFilter, ImageFont

N, S, SEED = 150, 384, 20260831
TR = ["jpeg95", "jpeg75", "jpeg40", "jpeg15", "resize50", "resize200", "crop3",
      "crop10", "rot90", "rot2", "bright", "contrast", "watermark", "hflip", "grey"]

# ---------------------------------------------------------------- fingerprints
def sha256_file(p):
    h = hashlib.sha256()
    with open(p, "rb") as f:
        for b in iter(lambda: f.read(1 << 16), b""):
            h.update(b)
    return h.hexdigest()

def ahash(im):                                   # 8x8 grey, bit = pixel > mean
    p = list(im.convert("L").resize((8, 8), Image.Resampling.LANCZOS).tobytes())
    avg, v = sum(p) / 64.0, 0
    for i, x in enumerate(p):
        if x > avg: v |= 1 << i
    return v

def dhash(im):                        # 9x8 grey, bit = pixel > right neighbour
    p = list(im.convert("L").resize((9, 8), Image.Resampling.LANCZOS).tobytes())
    v = i = 0
    for y in range(8):
        r = y * 9
        for x in range(8):
            if p[r + x] > p[r + x + 1]: v |= 1 << i
            i += 1
    return v

_NP, _NK = 32, 8
_COS = [[math.cos(math.pi * (2 * n + 1) * k / (2 * _NP)) for n in range(_NP)]
        for k in range(_NK)]

def phash(im):     # 32x32 grey -> 2-D DCT-II -> low-frequency 8x8 block vs median
    p = list(im.convert("L").resize((_NP, _NP), Image.Resampling.LANCZOS).tobytes())
    tmp = [[0.0] * _NK for _ in range(_NP)]
    for y in range(_NP):                         # rows -> first 8 freq columns
        row, t = p[y * _NP:(y + 1) * _NP], tmp[y]
        for k in range(_NK):
            ck = _COS[k]; s = 0.0
            for n in range(_NP): s += row[n] * ck[n]
            t[k] = s
    c = [0.0] * 64
    for k in range(_NK):                         # columns -> first 8 freq rows
        ck = _COS[k]
        for j in range(_NK):
            s = 0.0
            for n in range(_NP): s += tmp[n][j] * ck[n]
            c[k * 8 + j] = s
    rest = sorted(c[1:])                         # drop DC
    med, v = (rest[30] + rest[31]) / 2.0, 0
    for i in range(1, 64):
        if c[i] > med: v |= 1 << i               # exactly 32 of 63 bits are set
    return v

def chist(im):                                   # 3 channels x 16 bins, L1/2
    h = im.convert("RGB").resize((64, 64), Image.Resampling.BILINEAR).histogram()
    o = [sum(h[c * 256 + b * 16: c * 256 + b * 16 + 16])
         for c in range(3) for b in range(16)]
    t = float(sum(o)) or 1.0
    return [x / t for x in o]

cdist = lambda a, b: 0.5 * sum(abs(x - y) for x, y in zip(a, b))

# ------------------------------------------------------------------- the corpus
rnd = random.Random(SEED)
FONT, BIG = ImageFont.load_default(size=15), ImageFont.load_default(size=44)
SYL = "ka ro min ta vel sur dan qui lo pex tra nis gol mer ub za fen cri thal ond".split()
word = lambda: "".join(rnd.choice(SYL) for _ in range(rnd.randint(1, 3)))
col = lambda: (rnd.randrange(256), rnd.randrange(256), rnd.randrange(256))

def make(i):
    f = i % 4
    if f == 0:                                                       # gradient
        g = Image.linear_gradient("L").resize((S, S)).rotate(rnd.randrange(360))
        a, b = col(), col()
        lut = [tuple(int(a[c] + (b[c] - a[c]) * n / 255) for c in range(3))
               for n in range(256)]
        im = Image.new("RGB", (S, S)); px, gp = im.load(), g.load()
        for y in range(S):
            for x in range(S): px[x, y] = lut[gp[x, y]]
        r = rnd.randrange(60, 140)
        ImageDraw.Draw(im).ellipse([S//2-r, S//2-r, S//2+r, S//2+r],
                                   outline=col(), width=rnd.randrange(2, 9))
        return im
    if f == 1:                                                         # shapes
        im = Image.new("RGB", (S, S), (rnd.randrange(200, 256),) * 3)
        d = ImageDraw.Draw(im)
        for _ in range(rnd.randint(6, 14)):
            x0, y0 = rnd.randrange(S - 40), rnd.randrange(S - 40)
            x1, y1 = x0 + rnd.randint(30, 160), y0 + rnd.randint(30, 160)
            k = rnd.random()
            if k < .4:   d.ellipse([x0, y0, x1, y1], fill=col())
            elif k < .8: d.rectangle([x0, y0, x1, y1], fill=col())
            else:        d.polygon([(x0, y0), (x1, y0), ((x0+x1)//2, y1)], fill=col())
        return im
    if f == 2:                                                          # noise
        im = Image.merge("RGB", [Image.effect_noise((S, S), rnd.randint(24, 64))
                                 for _ in range(3)])
        im = im.filter(ImageFilter.GaussianBlur(rnd.uniform(0, 2)))
        return Image.blend(im, Image.new("RGB", (S, S), col()), 0.35)
    im = Image.new("RGB", (S, S), (rnd.randrange(240, 256),) * 3)        # text
    d = ImageDraw.Draw(im)
    d.rectangle([20, 18, 20 + rnd.randint(120, 260), 42], fill=col())
    y = 56
    while y < S - 20:
        d.text((20, y), " ".join(word() for _ in range(rnd.randint(4, 9))),
               font=FONT, fill=(rnd.randrange(60),) * 3)
        y += 20
    return im

def apply(name, im):
    T = Image.Transpose; R = Image.Resampling
    if name == "resize50":  return im.resize((S // 2, S // 2), R.LANCZOS)
    if name == "resize200": return im.resize((S * 2, S * 2), R.BICUBIC)
    if name == "crop3":     c = round(S * .03); return im.crop((c, c, S - c, S - c))
    if name == "crop10":    c = round(S * .10); return im.crop((c, c, S - c, S - c))
    if name == "rot90":     return im.transpose(T.ROTATE_90)
    if name == "rot2":      return im.rotate(2, resample=R.BICUBIC, fillcolor=(255,)*3)
    if name == "bright":    return ImageEnhance.Brightness(im).enhance(1.25)
    if name == "contrast":  return ImageEnhance.Contrast(im).enhance(1.40)
    if name == "hflip":     return im.transpose(T.FLIP_LEFT_RIGHT)
    if name == "grey":      return im.convert("L").convert("RGB")
    if name == "watermark":
        b = im.convert("RGBA"); lay = Image.new("RGBA", b.size, (0, 0, 0, 0))
        d = ImageDraw.Draw(lay)
        d.text((28, S // 2 - 24), "SAMPLE 2026", font=BIG, fill=(255, 255, 255, 102))
        d.rectangle([S - 150, S - 46, S - 12, S - 12], fill=(0, 0, 0, 153))
        return Image.alpha_composite(b, lay).convert("RGB")
    return im                                            # jpeg* differ at save

root = tempfile.mkdtemp(prefix="imgdup-")
try:
    t0 = time.time()
    H = {d: {} for d in ["orig"] + TR}
    for i in range(N):
        im = make(i)
        for d in ["orig"] + TR:
            out = im if d == "orig" else apply(d, im)
            if d.startswith("jpeg"):
                p = os.path.join(root, f"{d}-{i}.jpg"); out.save(p, quality=int(d[4:]))
            else:
                p = os.path.join(root, f"{d}-{i}.png"); out.save(p, compress_level=1)
            q = Image.open(p); q.load()
            H[d][i] = (sha256_file(p), ahash(q), dhash(q), phash(q), chist(q))
            q.close()
        if (i + 1) % 50 == 0:
            print(f"  {i+1}/{N} originals + variants ({time.time()-t0:.0f}s)", flush=True)

    ham = lambda a, b: (a ^ b).bit_count()
    print(f"\nSYNTHETIC corpus: {N} originals x {len(TR)} transformations "
          f"= {N*len(TR)} known duplicate pairs\n")
    print("recall per transformation, Hamming <= 10 (cHist <= 0.05)")
    print(f"{'transform':11s} {'sha256':>7s} {'aHash':>7s} {'dHash':>7s} "
          f"{'pHash':>7s} {'cHist':>7s}")
    for t in TR:
        r = [sum(1 for i in range(N) if H["orig"][i][0] == H[t][i][0]) / N]
        for k in (1, 2, 3):
            r.append(sum(1 for i in range(N)
                         if ham(H["orig"][i][k], H[t][i][k]) <= 10) / N)
        r.append(sum(1 for i in range(N)
                     if cdist(H["orig"][i][4], H[t][i][4]) <= .05) / N)
        print(f"{t:11s} " + " ".join(f"{x:7.3f}" for x in r))

    print(f"\n{'thr':>4s} {'aHash P/R':>18s} {'dHash P/R':>18s} {'pHash P/R':>18s}")
    for thr in (0, 1, 2, 4, 6, 8, 10, 14, 20):
        cells = []
        for k in (1, 2, 3):
            tp = sum(1 for t in TR for i in range(N)
                     if ham(H["orig"][i][k], H[t][i][k]) <= thr)
            fp = sum(1 for i in range(N) for j in range(i + 1, N)
                     for t in ["orig"] + TR
                     if ham(H["orig"][i][k], H[t][j][k]) <= thr)
            p = tp / (tp + fp) if tp + fp else 1.0
            cells.append(f"{p:.3f} / {tp/(N*len(TR)):.3f}")
        print(f"{thr:4d} " + " ".join(f"{c:>18s}" for c in cells))
    print("\npHash distances are always EVEN: every pHash has exactly 32 of its "
          "63 bits set,\nso a threshold of 5 is identical to a threshold of 4.")
finally:
    shutil.rmtree(root, ignore_errors=True)
    print(f"\ndeleted {root}")

Its precision column reads higher than the article's because it scores against a smaller pool of negatives — 150 originals rather than 6,400 images. The shape is the same, and the shape is the finding.