Image research and guides/AI detectors·4 min read

How to test whether an AI image remover actually works

A practical test design with positive controls, unchanged exports, exact files, quality checks and honest reporting of failures.

By Deprint

Reviewed by Dimitry Apollonsky

Published and reviewed

Start with a precise claim, a verified baseline and a saved output. Then test detection and image quality separately. A tool’s “complete” message proves that it finished a workflow. It does not prove that a watermark disappeared, a classifier changed its label or the picture remained acceptable.

A small, well-controlled comparison can be useful. A large folder of related exports can be misleading if the study hides its selection process or treats every variant as an independent image.

Define “works” before running the tool

Choose one primary outcome. Examples include removing specified metadata fields, receiving a negative SynthID response from a named verifier, or changing a named visual classifier’s label. Then choose quality constraints, such as preserving text, faces, color and required dimensions.

Do not move the goal after seeing results. If an output misses the watermark goal but receives a lower unrelated AI score, that is a different observation. It may be interesting, but it does not rescue the original claim.

A useful test starts before the treatment

Illustrative study design, with no invented result.

Baseline

Original + known positive control
Run the named checks
Confirm the starting condition
What this means

If the verifier cannot detect the original, that pair cannot demonstrate removal.

Treatment

A saved copy of the same original
Apply one documented operation
Keep the exact output
What this means

Record settings, dimensions and a hash. Keep failed attempts too.

Evaluation

Output + unchanged export control
Check detection and visual quality
Report both outcomes
What this means

A low detector score and an unacceptable visual change are different facts, and both belong in the report.

Build a minimal comparison set

File or check Purpose Failure it can expose
Untouched original Establish the starting result The original never triggered the detector
Unchanged or matched export Isolate ordinary format and size effects Exporting alone explains the change
Treated output Measure the proposed operation The operation does not meet the target
Known positive control Confirm the relevant check is functioning A broken or unsuitable verifier looks like success
Known camera-photo controls Examine false positives in visual classification A detector flags real photos too
Held-out scenes Test beyond the images used for tuning The method only works on selected examples

Not every personal check needs a full research dataset. But each control answers a specific question. If you omit one, narrow the conclusion accordingly.

Identify exact files, not just scene names

“Portrait after processing” is ambiguous when a folder contains a native PNG, a restored intermediate, a resized JPEG and a screenshot. Record filename, dimensions, format, byte count and a SHA-256 hash for each evaluated file.

A hash identifies bytes. It does not certify camera origin or visual equivalence. Its role is to prevent later confusion about which artifact received which verdict.

Also record the detector interface, date, model or version when available, score, label and raw response. Some hosted tools do not expose all implementation details. State that limitation instead of filling it in from assumptions.

Set quality gates that match the use

For a product image, a changed label or color may be unacceptable. For a portrait, identity and expression may dominate. For a decorative background, some texture change may be tolerable.

Inspect the output at its intended display size and compare important local regions at matching scale. Similarity metrics can help, but they do not decide whether a logo is spelled correctly. A full-frame score can hide a small, consequential error.

Choose these gates before selecting the best detector result. Otherwise the study rewards whatever changes produce a favorable score, even if those changes ruin the image’s purpose.

Keep tuning and evaluation separate

If you adjust a method repeatedly against one detector, improvement on that detector is expected to be easier to obtain than improvement on an untouched system. Use held-out images and detectors when making broader claims.

Deprint’s targeted local experiments illustrate the issue. Two scenes passed the local optimization targets but remained at 99.9% in Hive. The local result was real within its measurement, yet it did not transfer to the external classifier. See why detectors disagree.

This is also why twenty variants of one portrait are not twenty independent examples of reliability. Group related files by their underlying scene when describing the sample.

Report denominators and missing checks

If five of ten independently selected images meet a predefined outcome, say which ten, what the outcome was and whether quality gates also passed. If the sample was hand-picked, say that too. A selected example is a demonstration, not an estimated population success rate.

Timeouts, quota failures and ambiguous replies belong in a separate category. They should not inflate the number of negative detections. Preserve failed attempts so readers can understand what was actually evaluated.

When publishing a promising case, publish a counterexample if the method failed on other tested scenes. That helps a reader judge whether their own image resembles the favorable case or the failure.

A clear result statement

A useful conclusion identifies the exact output, the named checks, the observation date, the positive baseline and the quality tradeoff. It also states the limits of the sample.

For example, “this selected output received negative responses in these completed checks” can be supported by a small experiment. “This tool makes any image undetectable” needs evidence far beyond it.

Use the Deprint methods and observations CSV as an example of keeping distinct measurements visible. This guide proposes a test design; it reports no new remover benchmark.

Sources

  1. Deprint: methods, controls and dated observations
  2. ImageDetectBench research paper
  3. Google Help: verify AI-generated images, videos and audio