What is a depth map and how does AI make one from a photo?

A depth map in one picture

Take a photo of a person standing in front of a building. The depth map of that photo has the same width and height, but each pixel holds one number: how far that point is from the camera. Shown as an image, the person is one shade, the building another, and the sky the most extreme shade.

There is no single rule for which shade means near. Check the convention of every tool you use:

If a 3D effect looks inside out, with the background popping toward you, the map is in the opposite convention. Inverting the image (white becomes black) usually fixes it.

How AI estimates depth from one photo

A phone with two cameras or a LiDAR sensor can measure depth. A single ordinary photo cannot: the same picture could show a small toy car close up or a real car far away. Monocular depth estimation solves this the way people do, by using learned clues.

• Perspective: parallel lines such as roads and rails meet in the distance.
• Known sizes: a face, a door or a car has a typical size, so its size in the image hints at distance.
• Occlusion: an object that covers another is in front of it.
• Texture density and haze: grass and tiles look finer, and colors look paler, further away.
• Shading and focus blur, which give hints about surface shape.

A depth network is trained on millions of images where the answer is known or estimated. Depth Anything V2, used by many tools at the time of writing, learned from precise synthetic (rendered) images first, then from a very large set of real photos labeled by a larger teacher model. Its authors chose synthetic data partly because real depth sensors cannot measure transparent and reflective surfaces correctly, so real labels are noisy exactly where photos are hard.

The model works on a resized copy of the image (Depth Anything V2 was trained with the shorter side at 518 pixels) and outputs one value per pixel. The tool then scales it back to the photo size and maps the values to 0 to 255 for an 8-bit image.

Relative depth vs metric depth

Most general-purpose depth models predict relative depth. Depth Anything V2's base models, for example, predict affine-invariant inverse depth. In plain terms, the order and the proportions are meaningful, but there is an unknown scale and offset. The map can tell you the person is in front of the building; it cannot tell you the person is 3.2 meters away.

This design came from training on many datasets with different cameras and units. The MiDaS work showed that a loss which ignores scale lets a model learn from all of them at once, which made it much more robust on random internet photos.

Metric depth models are fine-tuned versions trained for meters, usually for one kind of scene (indoor rooms or street driving). Use them for measurement or robotics. For creative work, relative depth is what you want, because it generalizes better and the effect only needs correct ordering.

What you can do with a depth map

For example, cv.cm's depth-map tool turns a photo into a grayscale map (near is bright) or a color preview in the browser, and cv.cm's photo-to-video tool uses the same kind of map to move a virtual camera through a still photo. In that second case the camera moves and the subject stays still: it is parallax, not a model that animates people.

ControlNet showed that a diffusion model can be steered by conditions such as edges, pose, segmentation and depth. A depth condition keeps the composition (where the foreground and background are) while the prompt changes style, lighting or materials.

Where AI depth goes wrong

• Glass, mirrors and water: the model may give the depth of the reflection or of what is behind the glass.
• Sky and very far areas: all of it lands at the far end, so parallax there is flat.
• Thin structures: hair, fences, branches and wires blur into the background at the model's working resolution.
• Paintings, posters and screens: a photo of a picture can be given false 3D depth, because the content inside looks like a scene.
• Unusual views: straight-down drone shots, macro photos and fish-eye images are less common in training data.
• Edges between near and far: the map is smooth there, so a parallax effect stretches pixels into a rubbery band.

Getting a better depth map

1. Choose photos with a clear foreground, middle ground and background. Flat scenes, such as a face filling the frame against a wall, give little depth to work with.
2. Avoid strong reflections and large windows in the area you care about.
3. Make sure your next tool reads the same convention. If near is dark where it should be bright, invert the map.
4. Keep the map at the same size and crop as the photo. A depth map that is cropped or resized differently will not line up.
5. Blur or expand the map slightly before parallax to reduce tearing at edges, and keep camera moves gentle.