How does image-to-video AI work?

How it differs from text-to-video

In text-to-video, the model invents everything: the face, the clothes, the room, the light, and the motion. In image-to-video, you supply the first frame, and the model only has to invent what happens next. Google's Veo documentation describes this as using an existing image as the first frame of a new video; OpenAI's Sora guide calls it anchoring the first frame.

This changes what the prompt is for. The image already answers who, where and what style. The prompt answers what moves, in which direction, how fast, and what the camera does.

Inside the model, the start image is encoded and fed in as a condition, alongside the text. Stable Video Diffusion, an open model, was released with both text-to-video and image-to-video variants built on the same latent video diffusion base.

Choosing the start image

The start image decides more than the prompt does. Most failed image-to-video clips can be traced to a start image that did not leave room for the motion.

• Match the output aspect ratio. If the tool outputs 16:9 or 9:16, give it that shape; other sizes may be cropped or resized. Google's Veo documentation recommends at least 720p input in 16:9 or 9:16.
• Leave room for the motion. If the person will walk right, leave space on the right. If the camera will push in, the face should not already fill the frame.
• Pose for the start of the action. For 'she turns to look at the camera', start with her looking away.
• Clear, sharp subject. Blurry or tiny faces become worse when animated.
• Simple hands. Hands that are hidden, relaxed, or holding nothing animate more cleanly.
• No text you need to keep. Logos and signs may warp once the frame starts moving.

If you make the start image with an image model, generate it for the shot, not as a finished poster: a natural pose, a believable light source, and space around the subject.

Writing the motion prompt

Describe only what changes. Repeating the description of the image adds words the model does not need and can make it hold still. The examples in Google's image-to-video documentation are short motion lines, such as an animal moving naturally or flowers swaying gently.

Weak:
A beautiful woman with long red hair in a green dress standing in a field
of flowers at sunset, highly detailed.

Strong:
She slowly turns her head toward the camera and smiles. A light breeze
moves her hair and the flowers. Slow dolly in. Audio: wind, distant birds.

1. Subject motion: one action, with speed and end point.
2. Secondary motion: small things that make the scene alive, such as hair, leaves, steam, water.
3. Camera: one move or static. See the camera movement terms guide.
4. Audio, if the model generates sound.

Keep the motion physically possible from the start pose. A person seated in the image cannot cleanly be running in two seconds. If the action is large, use a first-and-last-frame tool and give the end pose too.

The 2.5D parallax method: depth map plus camera move

There is a second, much simpler way to make a photo move, and it is not generative video at all. It is often called the 3D Ken Burns effect, after a 2019 paper by Niklaus and colleagues.

1. Estimate depth. A depth model, such as Depth Anything V2, predicts for every pixel how far it is from the camera.
2. Turn the photo into a 3D layer. Each pixel is pushed back by its depth, so the photo becomes a shallow relief rather than a flat sheet.
3. Move a virtual camera. The camera pushes in, pulls out, orbits or pans. Near pixels shift more than far ones, which is parallax, and the brain reads it as depth.
4. Fill the gaps. When the camera moves, it can see areas that were hidden behind the subject. Simple tools stretch nearby pixels or zoom in slightly to hide the edges; research systems such as the 3D Ken Burns paper inpaint the hidden areas.

This is the method cv.cm's v tool uses. It runs Depth Anything V2 in your browser, renders the parallax with the GPU along a push, pull, orbit, pan or rise path, and exports an MP4. The camera moves; the subject does not blink, walk or speak. You can also look at the depth map itself first with cv.cm's depth-map tool.

Parallax works best on photos with clear depth layers: a person in front of a distant background, a product on a table, a street receding into the distance. It works poorly on flat subjects, such as a document or a wall, because there is no depth for the camera to reveal.

Common problems and fixes