Start with a precise claim, a verified baseline and a saved output. Then test detection and image quality separately. A tool’s “complete” message proves that it finished a workflow. It does not prove that a watermark disappeared, a classifier changed its label or the picture remained acceptable.
A small, well-controlled comparison can be useful. A large folder of related exports can be misleading if the study hides its selection process or treats every variant as an independent image.
Define “works” before running the tool
Choose one primary outcome. Examples include removing specified metadata fields, receiving a negative SynthID response from a named verifier, or changing a named visual classifier’s label. Then choose quality constraints, such as preserving text, faces, color and required dimensions.
Do not move the goal after seeing results. If an output misses the watermark goal but receives a lower unrelated AI score, that is a different observation. It may be interesting, but it does not rescue the original claim.
Illustrative study design, with no invented result.
Baseline
What this means
If the verifier cannot detect the original, that pair cannot demonstrate removal.
Treatment
What this means
Record settings, dimensions and a hash. Keep failed attempts too.
Evaluation
What this means
A low detector score and an unacceptable visual change are different facts, and both belong in the report.
Build a minimal comparison set
| File or check | Purpose | Failure it can expose |
|---|---|---|
| Untouched original | Establish the starting result | The original never triggered the detector |
| Unchanged or matched export | Isolate ordinary format and size effects | Exporting alone explains the change |
| Treated output | Measure the proposed operation | The operation does not meet the target |
| Known positive control | Confirm the relevant check is functioning | A broken or unsuitable verifier looks like success |
| Known camera-photo controls | Examine false positives in visual classification | A detector flags real photos too |
| Held-out scenes | Test beyond the images used for tuning | The method only works on selected examples |
Not every personal check needs a full research dataset. But each control answers a specific question. If you omit one, narrow the conclusion accordingly.
Identify exact files, not just scene names
“Portrait after processing” is ambiguous when a folder contains a native PNG, a restored intermediate, a resized JPEG and a screenshot. Record filename, dimensions, format, byte count and a SHA-256 hash for each evaluated file.
A hash identifies bytes. It does not certify camera origin or visual equivalence. Its role is to prevent later confusion about which artifact received which verdict.
Also record the detector interface, date, model or version when available, score, label and raw response. Some hosted tools do not expose all implementation details. State that limitation instead of filling it in from assumptions.
Set quality gates that match the use
For a product image, a changed label or color may be unacceptable. For a portrait, identity and expression may dominate. For a decorative background, some texture change may be tolerable.
Inspect the output at its intended display size and compare important local regions at matching scale. Similarity metrics can help, but they do not decide whether a logo is spelled correctly. A full-frame score can hide a small, consequential error.
Choose these gates before selecting the best detector result. Otherwise the study rewards whatever changes produce a favorable score, even if those changes ruin the image’s purpose.
Keep tuning and evaluation separate
If you adjust a method repeatedly against one detector, improvement on that detector is expected to be easier to obtain than improvement on an untouched system. Use held-out images and detectors when making broader claims.
Deprint’s targeted local experiments illustrate the issue. Two scenes passed the local optimization targets but remained at 99.9% in Hive. The local result was real within its measurement, yet it did not transfer to the external classifier. See why detectors disagree.
This is also why twenty variants of one portrait are not twenty independent examples of reliability. Group related files by their underlying scene when describing the sample.
Report denominators and missing checks
If five of ten independently selected images meet a predefined outcome, say which ten, what the outcome was and whether quality gates also passed. If the sample was hand-picked, say that too. A selected example is a demonstration, not an estimated population success rate.
Timeouts, quota failures and ambiguous replies belong in a separate category. They should not inflate the number of negative detections. Preserve failed attempts so readers can understand what was actually evaluated.
When publishing a promising case, publish a counterexample if the method failed on other tested scenes. That helps a reader judge whether their own image resembles the favorable case or the failure.
A clear result statement
A useful conclusion identifies the exact output, the named checks, the observation date, the positive baseline and the quality tradeoff. It also states the limits of the sample.
For example, “this selected output received negative responses in these completed checks” can be supported by a small experiment. “This tool makes any image undetectable” needs evidence far beyond it.
Use the Deprint methods and observations CSV as an example of keeping distinct measurements visible. This guide proposes a test design; it reports no new remover benchmark.