A point appears at different horizontal positions in two photographs. That shift can tell us how far it is from the cameras—provided the cameras are calibrated and the images rectified. The shift is called disparity. For an ideal parallel-camera pair, depth follows Z = fB/d, where f is focal length in pixels, B is the distance between camera centers, and d is disparity in pixels. The arithmetic is easy. Finding the same point in both photographs is the hard part.
That is the central task of stereoreconstruction. Stereo matching identifies corresponding points in two views; reconstruction uses those matches and the camera geometry to locate points in three-dimensional space. A pair of convincing-looking photographs offers no guarantee of a sound result. One bad match can produce a precise-looking 3D point in the wrong place.
Geometry before matching
Stereo geometry has roots in photogrammetry, which used overlapping photographs to measure terrain and structures. Digital computer vision made pixel matching and shape recovery algorithmic. The epipolar geometry reference explains a useful constraint: a point in one camera view limits its counterpart in the other to an epipolar line. Instead of searching the whole image, an algorithm can search along that line.
The constraint relies on knowing how the cameras form images. Intrinsic calibration describes properties such as focal length and principal point; lens-distortion parameters account for bending introduced by the optics. Extrinsic calibration describes the cameras’ relative rotation and translation. Together, these measurements define a ray extending from each camera through an image point. A reconstructed point lies where two matching rays intersect, or where they come closest when measurement error prevents an exact intersection.
Rectification makes corresponding epipolar lines run along the same image rows. It simplifies the search and makes disparity easier to interpret, but it cannot discover matches or repair faulty calibration. If the assumed camera positions are wrong, plausible-looking disparities can still yield biased depths.
What counts as a match?
A stereo algorithm needs evidence that two image regions show the same surface. Early digital systems often compared small windows of intensity values. They shifted a window from the left image across candidate positions in the right, then chose the shift with the lowest difference. Sum of absolute differences and sum of squared differences were common measures because they were straightforward to compute.
Object boundaries expose a weakness in this approach. A window may include a nearby object and a distant background; no single shift aligns both, so the estimated disparity smears the boundary. A flat wall presents the opposite difficulty: if neighboring pixels look alike, many shifts score almost equally well. Repeating patterns, such as railings or tiled floors, can produce confident-looking matches at the wrong position.
From local costs to spatial consistency
To resolve these ambiguities, researchers combined image similarity with assumptions about neighboring pixels. Points on a piecewise-smooth surface tend to have similar depths, while a sharp change in disparity may mark an object edge. Methods balanced those observations in different ways:
- Local matching pooled evidence within a window. It was relatively fast and simple but prone to errors near boundaries.
- Dynamic programming sought consistent matches along scanlines and could represent gaps caused by occlusion, though separately processed rows might disagree.
- Global optimization minimized an energy combining match costs with smoothness penalties. Graph-cut and belief-propagation approaches explored this trade-off at greater computational expense.
- Feature-based matching paired distinctive corners or textured patches, producing fewer but often more reliable 3D points than dense pixel matching.
The choice changes what kind of reconstruction is useful. A sparse set of reliable points may be enough to estimate camera position; obstacle avoidance or surface modeling often calls for a denser account of the scene. Even a dense disparity map can have holes where the images offer little evidence.
Why some surfaces cannot be recovered
An occlusion leaves a surface visible to one camera but hidden from the other. There is no genuine stereo match, and forcing one invents geometry. Shiny objects pose another problem: the cameras may see different reflections rather than stable markings on the surface. Transparent materials, low light, motion between captures, and nearly textureless areas also make the two views harder to compare.
A practical system needs permission to leave a match unresolved. One test computes disparity in both directions: a left-to-right match should return approximately to its starting point when matched right-to-left. Disagreement may signal an occlusion or a bad match. A confidence score can indicate how much better the selected candidate is than the alternatives. Neither test proves a match correct, but both can stop doubtful pixels from passing as measured surface.
Depth precision sets another limit. In Z = fB/d, a small disparity error has a larger effect on distant objects because their disparities are small to begin with. A larger baseline B improves depth sensitivity at a given distance, but cameras farther apart see more dissimilar regions and encounter more occlusions. Higher image resolution can help resolve small shifts, provided the optics, calibration, and surface texture support that precision.
From disparity maps to usable shape
Once disparity is estimated, each valid pixel can be back-projected into 3D using the calibrated camera model. The immediate result is usually a point cloud, not a complete solid object. Points are missing behind foreground surfaces and wherever matching failed. Noise may make a planar wall look rippled; isolated bad matches may float far from any plausible surface.
Making a mesh or taking measurements from that cloud requires further decisions. Outlier removal can discard inconsistent points, while surface fitting can replace noisy measurements with an approximate plane or curve. Additional stereo pairs may reveal surfaces hidden in the first pair, but their measurements must share a coordinate system. With a moving camera, errors in estimated pose can leave otherwise reasonable local reconstructions misaligned.
What a result should report
A polished rendering can hide systematic error. An interpretable evaluation should ask how many relevant pixels received a depth estimate, how accurate those estimates were, and where the errors occurred. A method that rejects every difficult region may score well on accepted pixels while offering poor coverage. One that fills every gap may look complete while inventing depth at occlusions.
Ground-truth depth, when available, permits numerical comparison. Without it, calibration checks, independently measured distances, and left-right consistency provide narrower evidence. Error maps can say more than a single average: failures around thin structures or reflective surfaces have different implications from a small bias across a flat plane.
Its place in mid-2000s vision research
By the mid-2000s, stereo was both a geometric problem and a testbed for broader computer-vision methods. Researchers compared matching costs, ways to preserve depth discontinuities, treatments of occlusion, and optimization under limited computing resources. Faster local methods appealed to robotics applications; slower global methods could yield cleaner maps in controlled evaluations. Outcomes depended on image quality, calibration, parameter choices, and test scenes—not just the optimization method.
Stereo brought a familiar pattern-recognition problem into sharp focus: how much should a system trust noisy observations, and how much should it rely on assumptions about structure? The role of academic venues in exchanging and testing such methods is discussed in How Conferences Shaped Pattern-Recognition Research in the 2000s. For stereo, the question stayed concrete: does the proposed 3D structure agree with what both cameras could see?
Applications included surveying, cultural-heritage documentation, robot navigation, and inspection. When the physical camera baseline is known, stereo can recover scale—an advantage over reconstruction from an unscaled monocular sequence. Its blind spots shaped system design too: an autonomous platform cannot safely treat a dark patch, a reflection, or an unobserved region as known free space. That uncertainty belongs in the 3D representation, not merely in a note beside it.
Take rectified images with f = 800 pixels and a baseline of 0.10 metres. A disparity of 40 pixels implies a depth of 2 metres; 20 pixels implies 4 metres. At the latter distance, an error of one pixel changes the inferred depth by roughly 0.2 metres. Before using that point to measure a doorway or plan a robot’s path, check the match and the calibration.