Driving content moderation from an agent via an MCP server

Before you start

You need three things in place before an agent can moderate anything usefully.

• A Cloudinary product environment with the Analysis MCP server reachable. The remote endpoint is https://analysis.mcp.cloudinary.com/sse and it authenticates with OAuth; the local variant needs credentials by hand. Both are described in Cloudinary’s MCP server documentation, and MCP server access is included on every plan, Free included (checked 2026-08-18). If you have not decided which servers the project needs, choosing MCP servers per project covers that; for this task the Analysis server is mandatory and the Asset Management server is nearly so, because the agent has to change delivery state as well as read verdicts.
• A moderation add-on enabled in the environment. Cloudinary’s moderation overview lists which automated add-ons and manual moderation are available and how an asset carries a moderation status; the automated image path is the AI content analysis add-on. Add-ons are billed separately from base-plan credits, so check the add-on tier before pointing a bulk queue at it. Free reaches only the free add-on tiers.
• Somewhere to write a decision record that outlives the agent’s context window. A row per asset in whatever store the application already has is enough; do not make this a new system.

What the reader should not expect: this page does not tell you which threshold values to use. Those are properties of your content mix, and step 5 is how you find them.

Steps

1. Upload every asset so that it is not publicly deliverable until moderation says so.

   This is the step people skip, and skipping it makes everything after it decorative. If an asset lands at a public delivery URL at upload time and moderation runs a few seconds later, the asset was public for those seconds — and for as long afterwards as it takes your pipeline to act on a rejection, which under load can be minutes. Anyone who fetched the URL in that window has the file; any cache that saw it may keep serving it.

   The mechanism is to upload with the asset’s access restricted (the moderation docs describe how moderated and restricted assets behave on delivery — read that section for your add-on, because behaviour differs between pending and rejected states) and to lift the restriction only as the last act of an approval. Do it in that order: the agent’s approve tool call should be “record decision, then make deliverable”, never the reverse. If the agent crashes between the two, the failure mode is an approved-but-hidden asset, which someone will notice and fix, rather than a rejected-but-visible one, which nobody notices until it is a complaint.

   For the upload itself the Asset Management MCP server is the natural surface if the agent is doing the uploading; if uploads come from a web form, the same restriction is set server-side and the agent only handles what follows.

2. Request the verdict and confidence, not just the verdict.

   Automated moderation returns a category label plus a confidence score. The label alone is not enough to build a policy on, because the add-on’s own approve/reject cut-off is a single line, and a single line is the wrong shape for a queue that a person is going to work. Ask the Analysis MCP server for the full result and keep the score. What the Analysis MCP server exposes for moderation, tagging and detection, and how the moderation result is shaped, is on that page; the AI content analysis add-on documentation lists the categories the model reports against.

3. Configure two thresholds, not one, and give the middle band to a person.

   The useful configuration is:

   • above the upper threshold: auto-approve, make deliverable, record;
   • below the lower threshold: auto-reject, keep hidden, record;
   • between the two: hold, keep hidden, put in a review queue for a human, record the hold.

   Why two lines rather than one: a single cut-off forces you to choose between letting borderline material through and rejecting legitimate uploads, and both errors are expensive in different currencies. The band between the lines is where you spend human attention, and its width is a budget decision — wider band, more review work, fewer automated mistakes. Start with a band you can actually staff. A band that generates two hundred items a day for a reviewer who has an hour is a band that gets rubber-stamped, which is worse than no band, because it looks like review in the log.

   Implement the thresholds in your own code around the MCP call rather than relying on the add-on’s default line, so the values are versioned with the application and the agent cannot drift them.

4. Route contextual harm to a person regardless of score.

   An automated moderator catches the categories it was trained on — the AI content analysis add-on documentation lists them — and it evaluates pixels. It does not know that this particular photo of a house is the reviewer’s home address, that this innocuous image is being posted to harass a named person, or that this screenshot contains someone else’s private message. None of that is visual; all of it is harm. So the score is only a decision input for the classes the model was built for. Anything where the harm depends on who posted it, where, about whom, or alongside what text needs a human path that does not consult the score at all: a report button that puts the asset straight into the review queue, and a rule that certain contexts (new accounts, assets attached to a named third party, anything reported once) go to review before publication whatever the model says.

   State this in the agent’s instructions explicitly. An agent given only “approve above 0.9” will approve above 0.9.

5. Measure the false negative rate on approved content — by sampling what passed.

   The number your review dashboard shows you by default is how many items were flagged and what happened to them. That is the wrong number. Flagged items are already in front of a person; the risk lives in the auto-approved pile, which nobody looks at, and the metric that describes it is the false negative rate: of the assets the pipeline let through, how many should it not have.

   You cannot get that from the flagged set. Get it by sampling the approved set — a fixed random fraction per day, or a fixed count if volume is uneven — and having a reviewer judge the sample blind, without seeing the model’s score. The sample review is what tells you whether the upper threshold is set right, and it is the only feedback loop that will ever tell you the model has drifted or your users have found something it does not see. Sample size is a trade-off between confidence and reviewer time; a small daily sample you actually do beats a large weekly one that gets skipped. This is a place where a person is doing the checking; if the sampling is itself driven by the agent, seeing what the agent actually did matters, because a sample the agent silently stopped drawing looks identical to a clean one.

6. Record every decision against the asset.

   The question asked later is never “what is your moderation policy”; it is “why was this specific item allowed through”. To answer it you need, per asset: the add-on and model that scored it, the raw category and confidence, the thresholds in force at that moment, which path it took (auto-approve, auto-reject, human), who reviewed it if a person did, and when delivery was enabled. Store this against the asset — Cloudinary’s structured metadata or context is one place, your own database is another; the point is that it is keyed by asset and survives the agent session.

   Thresholds change over time, so recording “approved” without recording the threshold that approved it makes the log unanswerable a month later. Recording the model output also lets you re-run the sample from step 5 retrospectively when you move a line.

Done looks like this

An uploaded asset is unreachable at any public URL until a decision has been made and recorded. Every asset has a decision record naming the model output, the thresholds applied, the path taken and, where a person acted, who. The review queue holds the middle band plus anything reported or flagged as contextual, and it is small enough that the reviewer working it is making judgements rather than clearing it. A daily blind sample of auto-approved assets exists, and the false negative rate from it is the number that decides whether the upper threshold moves. Nothing on the site depends on the score alone.