Reconciling What a Per-Event Media Automation Missed

Before you start

You need a per-event automation that already works most of the time — a MediaFlows flow, a hand-written handler behind a Cloudinary notification, or something in between — and a definition of “done” for an asset that can be read back from the asset itself. That second requirement is the one people skip. If the only record that an asset was processed is a log line in the automation, there is nothing to query against and no reconciliation is possible. Done has to be visible on the asset: a tag, a structured metadata field, a context value, a derived version that exists, or a moderation status. Decide that marker now, before writing anything, and make the live path set it as its last step.

You also need somewhere to run a scheduled job. Trigger surfaces differ between hosted automation products; the available trigger types page covers what a hosted flow can start from and where a schedule has to live outside it.

Steps

1. Accept that the live path misses events, and list how.

   Every per-event automation misses events. This is not a defect in a particular product; it is what “per-event” means. Three failure shapes cover almost everything you will see:

   • A webhook that never arrived. Notification delivery from Cloudinary is a network call to your endpoint; an endpoint that was deploying, timing out, or returning a 5xx at the wrong moment does not get a second chance from the asset’s point of view — the asset is uploaded and sitting there whether or not you were told.
   • A run that failed mid-branch. The flow started, did steps one and two, and died at step three — a rate limit, a malformed response, a bad expression. Retries and dead-letter handling, covered on error handling and retries in a hosted automation, reduce this class but never eliminate it, and a retry budget that runs out is still a miss.
   • An asset that arrived by a route that does not trigger. A colleague uploaded through the console, a migration script pushed ten thousand assets with notifications disabled, an integration used a different upload preset. The event you were listening for never fired because, from the automation’s perspective, nothing happened.

   Write these down for your specific flow. Each one tells you what “wrong state” looks like for the assets it leaves behind, and that is what the next step queries for.

2. Write the reconciliation as a query for wrong-state assets, not as an event replay.

   The tempting design is to keep a log of received events, find the gaps, and replay them. Do not do this. It only catches the first failure shape — you cannot replay an event that was never sent, and you cannot tell from a delivery log that a run failed halfway through.

   Instead, query for assets that are not in the done state. Cloudinary’s Search API takes an expression over asset attributes — tags, context, structured metadata, folder, resource type, upload date — and returns matching assets in pages. So if the live path’s last step adds a tag such as processed, the reconciliation asks for assets uploaded before some safety margin ago that lack that tag. If done means a metadata field is populated, ask for assets where it is empty. If done means a moderation status, ask for assets still pending.

   This is what makes the reconciliation robust to whatever caused the miss. It does not know or care whether the webhook was lost, the branch failed, or the upload came in sideways; it sees an asset that should have been handled and was not. New failure modes you have not thought of yet are covered for free, as long as they leave the asset in a queryable wrong state.

   The safety margin matters. An asset uploaded thirty seconds ago may simply not have been reached by the live path yet. Exclude anything newer than the longest time the live path legitimately takes, with room to spare, or the reconciliation and the live path race each other on fresh assets.

3. Make the reconciliation share logic with the live path.

   The reconciliation and the live path should share logic wherever the tooling allows, because two implementations of “process this asset” drift. Someone tightens the alt-text prompt in the flow and forgets the nightly job; someone changes the tag name in one place; six weeks later the overnight run is producing different metadata from the real-time one and nobody notices until a customer does.

   In MediaFlows the practical shape is one flow whose processing steps do the work, reached from two entry points: the event trigger for the live path, and whatever fires the scheduled query for the reconciliation. The PowerFlows catalogue shows the pattern in its prebuilt flows — a moderation or alt-text flow is a sequence of blocks over an asset, and nothing in that sequence needs to know how the asset reached it. If your product cannot share a flow between two triggers, the fallback is a single deterministic transformation — a named transformation, an upload preset, an eager derivation — that both paths invoke, so at least the expensive part cannot diverge. If neither is possible and you are maintaining two copies of the same processing by hand, that is one of the signals on when no-code stops being enough.

4. Make it idempotent by construction.

   The reconciliation will process assets the live path already handled. This is guaranteed, not occasional: the safety margin is a guess, the live path sometimes finishes late, and a query that runs while the live path is mid-asset returns that asset. So the processing must be safe to run twice on the same asset and produce the same end state.

   “By construction” means you do not achieve this by checking whether it already ran and skipping — that check is itself a race. You achieve it by making every step overwrite rather than append. Set the tag, do not add a duplicate; write the metadata field to a value, do not concatenate to it; generate the derived asset under a deterministic name so a second run replaces rather than duplicates. Then run the whole path twice on a test asset and confirm the second run changes nothing. If it changes something — a second tag, a doubled context string, a second copy of a rendition — fix the step, not the schedule.

   The same property protects the live path. A notification delivered twice, which can happen, hits the same idempotent processing and does no harm.

5. Record how many assets each run repairs, and treat that number as the metric.

   Each reconciliation run should emit one number: how many wrong-state assets it found and fixed. This count is a health metric for the live path, and it is more useful than most of what a dashboard shows you.

   • A reconciliation that repairs nothing, run after run, is telling you the live path is sound. You can relax the schedule — hourly to nightly, nightly to weekly — and spend fewer credits on the query and on re-touching assets.
   • A steady small count is normal background loss: the occasional missed webhook, the odd console upload. Leave the schedule where it is.
   • A count that is growing is not a reconciliation problem. It is reporting that the live path is broken — a trigger that stopped firing, an endpoint that started returning errors, a new upload route nobody wired in — and the reconciliation is now doing the live path’s job on a delay. Investigate the live path; do not just tighten the schedule and let the backlog hide.

   Store the count somewhere you will look at it. A metric nobody reads is a log line.

Expected result

When this is done, the system has two paths that converge on one state. The live path handles most assets within seconds of the event. A scheduled query finds every asset that slipped past it — for any reason, including reasons you have not met yet — and runs the same processing over it, and running that processing on an already-handled asset leaves it unchanged. Each run reports how many assets it repaired, and that number sits close to zero. When it starts climbing, you find out from the metric before you find out from a customer, and the thing you fix is the live path, not the reconciliation.