Chapter 36. Image-to-Video and Motion Design#

36.1 Motion must serve the state change#

Once the keyframe holds, video only has to complete the change this shot requires. Moving hair, background, camera and every character at once adds drift without adding story.

Split motion into the character's primary action, their micro-motion, environmental motion, and camera motion. One primary action per shot; keep the rest at low amplitude.

36.2 The motion brief#

motion_request:
  shot: E001_S09
  duration: 3.5s
  start_frame: KF_E001_S09_START@v04
  character_action: right hand slides the authorization forward 12cm, then stops
  micro_action: fingers relax once, naturally
  environment: background figures stay defocused and still
  camera: very slight push
  end_state: title and signature area hold steady for 0.6s
  forbidden:
    - change in finger count
    - the document rotating
    - background figures walking
    - generated text changing

Distances express scale rather than promising pixel-level control. What matters is that the action is short, the direction unambiguous, and the ending cuttable.

36.3 Two to five seconds is not absolute, and it is useful#

Short shots reduce identity and physical drift and make it easy to select a stable middle section. Longer shots should be supported by genuine stillness, simple motion, or real necessity. Do not stuff several beats into one shot merely because the generation length allows it.

Generating five seconds is not using five seconds. Record the final usable interval.

36.4 Timeline prompts#

Describe slightly more complex actions by second or by stage: the eyeline resting on the document with the body still; the right hand sliding it steadily forward; the document stopping and the fingers releasing; then holding still while the camera pushes slightly.

A timeline schedules only a small number of changes that can happen together. Even when a model cannot execute it strictly, it still conveys priority and the order of actions.

36.5 Motion amplitude#

Micro-motion is the most stable: blinking, breathing, a slight head turn, a finger tapping once, small fabric movement. Medium actions such as standing, handing something over or taking two steps need more state control. Large actions, contact and multi-person movement are better split into shots or layers.

Tag risk as low, medium or high amplitude and set the attempt budget accordingly.

36.6 Do not let camera and character compete#

While a character completes an important action, keep the camera static or on a simple push. When camera movement is complex, simplify the character's action. Orbits, fast pushes and handheld change background, face and occlusion simultaneously, which is high risk.

A camera move must deliver new information by the time it ends. If nothing is revealed, do not move.

36.7 First frame, last frame and handles#

The first frame must be an approved keyframe or a verified variant. The last frame must be reachable, identity-stable, in the correct state, and hold briefly. Handles before and after give the edit choices and avoid the model's start-up jitter.

Where first/last frame control is available, check first that character, hands, props and space agree across the two frames. Too much difference makes the middle melt.

36.8 No lip sync, lip sync, and off-screen lines#

Ordinary motion shots default to producing no speaking mouth shapes. Dialogue can be delivered off screen, from behind, over the shoulder, in reaction, or with a short frontal lip-sync shot. Avoid characters whose mouths move continuously with no lines.

Lip sync is a separate high-risk route. It uses final audio and requires identity and performance checks, and it should not run alongside complex action.

36.9 Environmental motion#

Curtains, rain, screen light and background crowds all obey the environmental state. Wind direction, rain intensity and crowd behavior carry across shots. Keep background motion at a lower amplitude than the subject so it does not steal attention.

A completely still environment can stay alive through sound. There is no need to force every part of the frame to move.

36.10 Choosing candidates#

Record each candidate's deviation from the first frame, how completely the action finished, identity drift, physical errors, camera stability, and the usable interval. Choose the most cuttable, which is not necessarily the one with the most movement.

Different intervals of one clip can serve the episode and an ad, and every use records the parent.

36.11 Loops, slow motion and speed changes#

Loops suit repetitive environments or micro-motion; looping an emotional action makes a character look mechanical. Slow motion cannot repair wrong physics — it magnifies it. Slight speed changes can match an edit, and lip sync, rain and walking will expose the change.

When you need a hold, prefer generating a stable end state or an extremely subtle living frame, rather than freezing a face indefinitely.

36.12 Motion QC#

Check identity at first, middle and last; hands and props; direction of action; camera; background; end state; and cuttable handles. Watch at normal speed for perception first, then go frame by frame for blocking errors. Lay in scratch audio to confirm the action's rhythm can carry the dialogue and effects.

36.13 Use action phases instead of vague verbs#

"She hands the document over" contains at least five phases: preparation, approach, handoff, release and completion. If the shot only needs to express applying pressure, generate approach through completion. If the story point is that the other party refuses to take it, you must keep the approach and the hover — and must not let the model complete the handoff on its own.

The motion brief states which phases the shot covers and where it cuts:

action_phases:
  action_id: ACT_AUTHORIZATION_HANDOFF
  covered: [approach, hold]
  omitted: [grasp, release]
  start: document flat on the table, Lin Yun's right hand on the top-right corner
  cut_point: the document stops after crossing the table's midline
  state_out: the document is still controlled by Lin Yun; Zhou Lan has not touched it
  dramatic_meaning: this is a challenge, not a polite handover

Phase design protects continuity and dramatic meaning at once. If the next shot begins with Zhou Lan's hand already gripping the document, there must be a separate handoff shot or an explicitly permitted elision — an object cannot change owner across a cut without cause.

36.14 Previsualize motion before spending generation budget#

Medium-amplitude actions and above can be previsualized with a phone, a real person walking it, crude 3D, still-frame pushes, or an edit animatic. Previsualization does not chase visual quality; it verifies duration, direction, occlusion, cut points and audio rhythm. Testing whether three seconds is enough to turn and sit is cheaper done once with a person than twenty times through generation.

Previsualization also exposes shots that do not need to move. If the power shift already holds once dialogue, reaction and effects are laid over still frames, you can generate only breath, eyeline or a micro push. Motion is not value in itself; it is a carrier for changing information.

36.15 Usable intervals and edit handles#

A generated file should record three regions: the start-up contamination zone, the usable zone, and the decay zone. Many results jump from a still into life in the first 0.3 seconds and begin changing the face in the final 0.8. Writing only "this one works" makes the editor hunt repeatedly for the real interval.

usable_segment:
  source: GEN_E001_S09_006
  source_duration: 5.0
  in: 0.62
  out: 3.18
  preferred_cut_in: 0.84
  preferred_cut_out: 2.92
  state_at_in: hand_begins_push
  state_at_out: document_stable
  defects_outside_range: [startup_warp, late_identity_drift]

Try to keep 6–12 frames of handle beyond the preferred cut points, so sound, reaction and rhythm can be adjusted. If the final interval has no handles, mark the shot as fragile in the edit ledger; a later change in line or music length may force it to be remade.

36.16 Handoff testing across shots#

Once each clip passes on its own, run a three-way check: the previous shot's last 8–12 frames, the whole current shot, and the next shot's first 8–12 frames played continuously. Check whether the speed of the action changes abruptly, whether the character's center of gravity jumps, whether the eyeline switches target, whether environmental motion restarts, and whether the camera makes an unintended jump cut.

Record action speed in three registers: wind-up, execution, aftermath. If the outgoing shot is fast execution, the reverse can enter a still aftermath. If the reverse starts winding up again, the audience perceives the action happening twice. The generation station must deliver a state change, not merely a video file.

36.17 Entering the boardroom in Backlit Takeover#

The first draft wrote "Lin Yun pushes the door, walks to the table, produces the authorization and addresses the room" as one seven-second shot. In testing, her face changed with every step, the authorization appeared from a handbag out of nowhere, and the boardroom geometry stretched as the camera retreated.

The rebuild split it into five controllable tasks: the handle turning and the door opening; Lin Yun stopping after crossing the threshold; the room turning to look; her folder landing on the table; and a short frontal line — from now on she represents the acquirer. Sound glued the five together with the door, footsteps and boardroom room tone, leaving half a second of quiet before the line. The shot count went up while total generation attempts went down, and every cut now reinforces her taking the space.

36.18 SOP for image-to-video#

First, compile the motion brief from the approved keyframe. Second, separate primary action, micro-motion, environment and camera. Third, constrain duration and amplitude. Fourth, write a timeline where needed. Fifth, generate a small number of candidates and tag failures. Sixth, select the usable interval rather than the whole clip. Seventh, check first, middle, last and the end state. Eighth, update state out and the generation ledger. Ninth, on failure, simplify the action or change the shot design.

36.19 Fault tree#

Symptom: the frame is lively and the story is unclear. Motion has no hierarchy. Keep one primary state change.

Symptom: the face changes near the end. Duration or action amplitude is too large. Cut earlier, shorten, or regenerate.

Symptom: the document's shape and text flow. The model was asked to maintain detail during motion. Reduce motion and composite tracked graphics.

Symptom: a character with no lines keeps moving their mouth. Lip sync was never prohibited. Add a silence constraint or regenerate.

Symptom: the camera move judders. Character and camera are both moving in complex ways. Lock the camera or split the shot.

36.20 Checklist, exercises and deliverables#

Check that the primary action is singular; that environment and camera are subordinate; that the duration is necessary; that the timeline is executable; that first and last frames agree; that handles exist; that lip sync is intentional; that candidates record usable intervals; and that failures are simplified first.

Exercise one: split "she walks in, raises the document and denounces them" into four motion briefs. Exercise two: design still, micro-motion and medium-amplitude versions from one keyframe. Exercise three: mark a two-second usable interval in a five-second candidate.

Deliverables for this chapter: the motion brief, timeline prompts, candidate records, usable intervals, motion QC, state out, and the retry-or-restage decision.

36.21 The motion contract and time windows#

A motion contract defines subject, action phases, target, camera, environment, what must be preserved, and prohibited events. Express time as windows rather than a precise illusion: preparation from 0 to 0.6 seconds, sliding the document from 0.5 to 1.8, settling and reacting from 1.6 to 2.6. Overlapping windows give the edit handles.

Figure 36-1 The keyframe locks identity, pose, props, world position and lighting. The green region describes permitted body and camera motion; the red region protects the invariants. Only the reviewed usable interval is exported from the result, and the end state is written back for the next shot.

The start and end frames are not the only checkpoints. Middle frames still need sampling for identity, hands, props and background. If an error lands exactly on the peak of the main action, recovering by the last frame does not make it pass.

motion_contract:
  subject: CH_LINYUN
  action: push_authorization
  phases: [prepare, slide, settle]
  preserve: [identity, right_hand, document_face_up, axis]
  camera: static
  forbidden: [left_hand_swap, page_flip, background_people_move]
  usable_target: 1.4_seconds

When output exceeds the contract, first judge whether a compliant interval can be extracted, then decide on regeneration. The finished cut needs only a usable stretch of seconds — it does not require the whole generated file to be perfect.

36.22 Temporal stability and reviewing the middle frames#

Correct ends do not mean a correct middle. Identity, fingers, text, background and lighting can drift mid-clip and recover. Motion QC samples first, middle, last, the action peak, and fixed intervals according to shot risk, while also watching at normal speed for perceptual problems.

For key close-ups, generate a contact strip and a difference view marking the usable start and end. A brief error outside the cuttable region is acceptable; one crossing the main action or an expression peak blocks. After frame interpolation, slow motion or stabilization, review again — intermediate frames can introduce new blending.

36.23 The motion de-escalation ladder after failure#

Try in order: shorten the duration, reduce simultaneous motion, lock the camera, split into two shots, use an insert and sound to elide, composite in layers, or convert to a still brought to life. Each step down preserves the narrative goal; a failed shot is never silently deleted.

Hero actions justify higher cost; ordinary transitions are simplified first. After three failures with the same root cause, stop rolling and return to the keyframe, the action design or the model routing. De-escalation decisions go into the shot ledger, so a later postmortem knows why the finished cut uses that grammar.

A note on sources#

In practice, stable 2–5 second clips, a single action and timeline descriptions are usually more reliable than completing a whole scene in one shot. The edit is what organizes short movements into continuous performance.