Detect duplicate uploads without a manual audit

Before you start

Two things decide how much of this you need.

First, the mechanism. The Duplicate Image Detection add-on assigns each image a perceptual hash derived from what the image looks like, and two assets are duplicates when their hashes are close. That is different from a checksum. A file re-exported from Photoshop, a resized copy, or the same photo saved as WebP has a different checksum and an unchanged appearance, so a hash-based dedupe misses exactly the cases that fill a real library.

Second, the bill. Add-ons are billed separately from the base plan’s credits and appear as their own invoice line, and the Free plan reaches only the free add-on tiers. So a full-library sweep is a budgeted operation, not something you switch on in the background and forget. Count the images you intend to fingerprint before you start, and decide whether you are sweeping everything or only what arrives from now on.

Why bother: duplicates cost you twice. Storage is metered per gigabyte held, and it is a current total rather than a rolling window, per the credits FAQ — which also means removing a copy lowers the number the same day. Worse than the storage is the split metadata: tags, alt text and structured fields end up on whichever copy someone edited last. In practice the second upload is the one people find in search, and the first is the one every page links to.

Steps

1. Stop the cheap case at upload. Before you detect anything, remove the reason for the most common duplicate: the same file uploaded twice. Set the public_id on upload deterministically — from a content ID in your CMS, a SKU, or a normalised filename — as the upload API allows. A second upload with the same public ID lands on the same asset instead of beside it. This is a line in the upload preset or the ingestion flow, not an add-on, and if you are running bulk ingestion from a spreadsheet it is the column that matters most. It does nothing for a resize or a re-export, which is what the remaining steps are for.

2. Fingerprint new uploads as they arrive. Enable the duplicate detection add-on and request the perceptual hash on upload, so every incoming image carries its fingerprint from day one. Doing this on ingest keeps the add-on spend proportional to upload volume, which is the number you already track.

3. Decide whether to sweep the backlog. Existing assets have no hash until you ask for one, and that is the budgeted part. Either fingerprint the whole library once, or fingerprint only the folders where duplicates actually hurt — product imagery, brand assets — and leave the archive alone. Multiply the asset count by the add-on’s tier before choosing.

4. Compare and flag, in the automation, without deleting. In MediaFlows or the automation you already run for asset expiry, compare each new hash against the store and, on a close match, tag both assets and put them in front of a person — the same approval routing pattern used for other judgement calls. Do not let the flow delete. Which of two near-identical images is canonical is not a property of the pixels; it is a question of what references them. The one embedded in forty pages is canonical even if it is the smaller, older, worse-tagged copy. A flow cannot see that; a person with the usage report can.

5. Resolve by redirecting, then removing. For each flagged pair, the reviewer picks the canonical asset, moves the metadata onto it, repoints or leaves the references, and only then removes the other. Storage drops immediately, because it is a total rather than a window.

Done looks like

Every image uploaded from now on carries a perceptual hash. Same-file re-uploads collapse onto one public ID without any detection running. Near-identical images — resizes, re-exports, format changes — arrive tagged as suspected duplicates with a pointer to the match, in a review queue rather than a trash folder. The add-on shows up as its own line on the invoice, sized by the sweep you chose, and the storage line goes down as reviewers merge pairs. Nobody has audited a folder by eye.