image generation prompt test set: Design a reusable evaluation set before optimizing prompts

An image generation prompt test set is a collection of defined tasks and acceptance criteria, not merely a folder of attractive examples. Its value is that future prompt changes face the same kinds of problems. The procedure here is an experiment design; it does not report a completed model comparison.

Select tasks from actual delivery needs

Start with the assets your team must deliver. A small product workflow might need a plain-background image, a scene with room for a headline, a detail view, and an image containing approved text. Write the task requirement before choosing a model or composing an elaborate prompt.

For each case, save a stable case identifier, intended use, subject brief, required details, prohibited changes, target ratio, and acceptance criteria. If a reference is required, retain its approved source and content hash. A URL alone can change its bytes later.

Avoid making every task a minor variation of the same easy scene. Include a difficult but relevant case, such as several distinct objects or a partially obscured product, when your real workflow needs it. Do not add exotic cases merely to make another model look weak.

Separate development from acceptance

Use one portion of the set to revise prompts and another portion to evaluate the frozen revision. Once authors repeatedly inspect and optimize against a case, that case becomes development evidence. Calling it an untouched acceptance test would overstate independence.

Freeze the acceptance briefs, request settings, and rubric before reviewing new outputs. If a requirement was missing, revise the set explicitly and keep the previous version. Do not quietly change the pass rule after seeing an appealing result.

The current image endpoint expects prompt as a string and maps selected standard fields. Keep your test case metadata outside that request body. Compile the approved case into an explicit model, prompt, ratio, image count, and optional seed or first reference.

Preserve each attempted sample

Record the planned sample count, actual submissions, returned job identifiers, saved outputs, and failed or unresolved attempts. A missing result is part of the operational record, not a reason to delete the case.

When humans choose the best image from several candidates, report that selection procedure. Comparing a selected best-of-four result with a single unselected result changes the experiment and its cost. Keep both requested work and accepted deliverables visible.

A seed can help document a request but does not guarantee repeated bytes across versions or time. Archive the actual evaluation artifacts.

Acceptance review

A second reviewer should be able to reconstruct a case, identify its frozen criteria, and distinguish prompt-development examples from held-out acceptance cases. Leave score fields empty until a real review occurs.

Continue with human evaluation rubrics, prompt revision diffs, and cost per accepted asset when turning this design into an internal evaluation.

Continue with quality and testing

• generation model regression checklist: Maintain a release scorecard with real evidence
• generation prompt revision diff: Review the change as a field-level hypothesis
• AI image color drift test: Control viewing conditions before judging color
• Browse all quality and testing guides