Choose tasks that represent your actual work
Use a small set of shots rather than one spectacular prompt. A practical starter set is a tabletop product move, a person interacting with an object and a landscape camera move. Each exposes different failure modes: geometry, contact and spatial consistency.
Write a minimum acceptable outcome for each. For the product shot, the product must keep its silhouette and remain readable at the delivery size. For the person shot, the action must complete without impossible contact. For the landscape, the horizon and fixed structures must remain stable.
Hold requirements constant while respecting tool differences
Match the subject, action, duration target and final frame shape. Use the same authorized image when comparing an image-led task. Keep resolution and upscaling separate in the log. If one model cannot support a required input, record that as a workflow limitation rather than silently changing the test.
Do not insist on identical API parameter names or identical prompt syntax. Tools expose different controls. Adapt the expression of the instruction while keeping the required scene intact. Document the adaptation so a reviewer can see whether the comparison remains fair.
Set acceptance criteria before viewing the outputs
| Criterion | Pass condition | Typical rejection |
|---|---|---|
| Required action | Main action completes in the specified order. | The action never happens or reverses. |
| Subject continuity | Defining features remain recognizable. | Identity or product geometry changes. |
| Temporal stability | Important objects remain coherent through motion. | Objects pop in, vanish or melt. |
| Camera behavior | Viewpoint follows the planned shot. | Unexpected cuts or a contradictory move. |
| Delivery fitness | Crop, duration and export satisfy the brief. | Critical subject cropped or file unsuitable. |
Add audio synchronization only when your project needs it and the exact model mode supports it. Weighting is a project decision. Publish the weights if you publish results, and avoid presenting your preferences as an objective universal ranking.
Repeat attempts and retain failures
A small pilot could use three attempts per task and three tasks per model. That is nine attempts per model. It is enough to discover obvious workflow problems, but too small to establish a robust population-level ranking. Describe it as a pilot if you publish it.
Save the full output list, not just favorites. Use neutral filenames for a blind review when practical. Record the reviewer, pass/fail reason, charge and elapsed time for each attempt. If an attempt fails for infrastructure reasons, identify that separately from a creative rejection.
Report cost per accepted shot
Divide total charged generation and processing cost by the number of shots you accept. Include rejected attempts in the numerator. Track editing time separately, because a cheap generated file can require expensive repair work.
Download our blank model test log to record attempts. The template contains field names and illustrative guidance, not prefilled benchmark results. Use the calculator to explore how acceptance rate changes the budget before you start spending credits.
Sources and verification
Reviewed October 25, 2026. Provider documentation supports specifications; it is not evidence of an independent performance test.