Two cameras do not produce depth on their own. The central problem is finding the point in the right image that depicts the same physical point as one in the left. Once that correspondence is established, depth follows from disparity: nearby features shift more between the two views than distant ones. When the match is wrong, a reconstruction may still look plausible numerically while placing surfaces in the wrong position.
This compact idea brought together geometry, image processing, optimization, and hardware design in mid-2000s computer vision. Academic stereo systems were often tested on rectified image pairs with ground-truth disparity maps, yet the intended uses were wider: robot navigation, industrial inspection, mapping, driver assistance, cultural heritage documentation, and three-dimensional measurement. Moving from a conference benchmark to a working system depended less on one winning algorithm than on calibration, texture, occlusion, computation, and uncertainty.
Disparity turns image shift into depth
In a calibrated, rectified stereo rig, corresponding scene points lie on the same horizontal scanline. Instead of searching across a two-dimensional image, the algorithm searches along a single row. If the horizontal coordinates of a matched point are xL and xR, disparity is commonly written as d = xL − xR. With focal length f and baseline B, depth is approximated by:
Z = fB / d
The units must be consistent. If focal length is expressed in pixels, the calibration must support the corresponding conversion. The formula is especially sensitive at long range, where disparity becomes small: a one-pixel matching error can then produce a large depth error. A wider baseline can improve depth resolution at distance, but it also makes correspondence harder because nearby objects look more different to the two cameras.
Why rectification became standard practice
Real camera pairs are seldom aligned perfectly. Lens distortion bends apparent straight lines, slight orientation errors tilt epipolar lines, and manufacturing tolerances have consequences. Stereo calibration estimates intrinsic camera properties, including focal length and distortion, along with the cameras’ relative pose. Rectification then warps both images so epipolar lines become horizontal and corresponding points can be searched for on matching rows.
Rectification is more than a programming convenience; it is also a useful diagnostic. Overlay a few corresponding features after rectification: they should align vertically. Persistent vertical mismatch may point to calibration drift, poor corner detection during calibration, focus changes, an altered camera mount, or an unsuitable distortion model.
The correspondence problem
A basic local method compares a small patch around each left-image pixel with candidate patches along the relevant row in the right image. The candidate with the lowest matching cost is selected. Common costs include absolute intensity differences, squared differences, and normalized cross-correlation. These methods are fast and easy to inspect, but the size of the image window introduces a difficult trade-off.
- Small windows preserve boundaries and fine structures, but are more susceptible to noise and ambiguous texture.
- Large windows stabilize matches in textured regions, but blur disparity changes at object edges.
- Uniform regions, such as plain walls, often provide too little visual information for a unique match.
- Repeated patterns, including railings, tiles, and windows, can produce confident-looking yet incorrect matches.
These limits explain why research moved beyond independent pixel decisions. Stereo is not simply a set of local comparisons: neighbouring pixels often belong to continuous surfaces, apart from genuine depth boundaries. Algorithms therefore combined a data term—how well two image locations match—with a smoothness term that penalizes abrupt disparity changes where the image offers no reason to expect them.
From matching costs to coherent surfaces
Global and semi-global methods sought a disparity field that balanced image evidence with spatial consistency. Dynamic programming provided an efficient way to optimize along scanlines, while graph-cut formulations and belief propagation treated stereo as a broader optimization problem over a pixel grid. These approaches often handled low-texture regions better than simple window matching, although they required more memory, computation, and parameter tuning.
By the middle of the 2000s, comparisons between local and global stereo methods had become a recurring feature of the literature. Benchmark culture exposed errors in difficult areas: depth discontinuities, slanted surfaces, reflective objects, and occlusions. A broader discussion of how conference papers exposed such methodological trade-offs appears in What Mid-2000s Conference Papers Reveal About Recognition Research.
Cost aggregation and edge awareness
A practical middle ground appeared in methods that first computed local matching costs, then aggregated them more selectively. Rather than averaging within a fixed square window, an edge-aware method gives greater weight to pixels likely to belong to the same surface. Colour similarity, image gradients, segmentation, or adaptive support regions can reduce smoothing across a strong object boundary.
A depth map should therefore not be judged by smoothness alone. Excessive smoothing can erase a narrow pole, merge an object with its background, or pull foreground depth across an edge. In measurement work, preserving the position of discontinuities can matter more than producing a visually pleasing dense map.
Occlusion is a geometric fact, not just an error
Some points visible to one camera are hidden from the other. A foreground edge may block part of the background in the left view but not in the right, leaving no valid correspondence. No matching rule can recover a true stereo match for a point that is absent from one image.
Classical pipelines addressed this with left-right consistency checking. A disparity is first estimated from the left image to the right, then estimated again in reverse. If a left pixel projects to a right pixel whose reverse disparity does not return to the original position, the match is marked unreliable. The same test can expose mismatches caused by repeated textures or illumination differences.
Invalid pixels are best handled explicitly rather than silently filled. Depending on the application, an occluded region may remain unknown, be interpolated only within a likely surface, or be excluded from later measurement. For obstacle avoidance, a conservative unknown label may be safer than an invented depth value. For visualization, carefully constrained filling can help, provided it is not mistaken for observed geometry.
What made laboratory stereo fail outdoors
Benchmark imagery is valuable because it enables controlled comparison, but operational scenes introduce conditions that are easy to underestimate. Outdoors, systems encounter shadows, specular highlights, moving foliage, exposure differences, weather, and low-texture surfaces. Indoors, glossy machinery, transparent guards, repetitive shelving, and fluorescent lighting create different failure modes. Passive stereo depends on visible appearance and performs poorly in darkness unless illumination or active sensing is added.
| Practical issue | Why it affects stereo | Typical response |
|---|---|---|
| Low texture | Several candidate matches look equally plausible | Use structured illumination, larger contextual support, or confidence masking |
| Reflective or transparent surfaces | Brightness does not represent stable surface appearance | Treat estimates as uncertain; combine with other sensing where needed |
| Camera vibration | Calibration and rectification no longer match the physical rig | Use rigid mounting and validate calibration regularly |
| Exposure mismatch | Patch similarity no longer reflects correspondence well | Synchronize cameras and apply radiometric normalization carefully |
| Motion between views | Corresponding pixels may depict different object positions | Capture simultaneously and identify moving regions |
Synchronization deserves particular attention. Cameras that expose at different times can produce false disparity on moving objects even when calibration is excellent. This mattered especially for mobile platforms and traffic scenes, where a moving vehicle or pedestrian could be reconstructed as a distorted surface. Hardware triggering, short matched exposures, and attention to rolling-shutter effects became practical requirements rather than optional refinements.
Evaluation: accuracy is not one number
Academic reports often measured the proportion of pixels whose disparity error exceeded a specified threshold. That made methods easier to compare, but a single score conceals distinctions that matter in use. An error near an object boundary may be more serious than the same error on a flat background. A method can produce dense disparities while failing precisely where safety or measurement depends on reliable values.
A useful evaluation separates at least four concerns:
- Geometric accuracy: compare reconstructed depth or disparity against measured reference geometry.
- Completeness: report how much of the scene receives a valid estimate instead of assuming every pixel should be filled.
- Boundary behavior: inspect depth discontinuities, thin objects, and occluded zones separately.
- Runtime and stability: test on the intended hardware under changes in light, range, vibration, and scene texture.
For industrial measurement, the useful result may be a stable plane estimate or the distance to a known component rather than an attractive point cloud. For a robot, the important question may be whether nearby free space and obstacles are separated conservatively. Such uses require task-level tests alongside standard disparity metrics.
Building a practical stereo pipeline
A reliable system is usually built as a sequence of independently testable stages, rather than treated as a single depth-producing algorithm:
- Mount cameras on a rigid baseline suited to the working range.
- Calibrate intrinsics, distortion, and relative pose using images captured at the focus and resolution intended for operation.
- Rectify paired frames and check vertical alignment using real scene features.
- Capture synchronized images with controlled or monitored exposure.
- Compute matching costs and disparities over a range consistent with expected scene distances.
- Apply consistency checks, confidence estimation, and explicit invalid-pixel handling.
- Convert disparity to depth only after retaining calibration metadata and units.
- Validate the resulting measurements against known distances and representative operating scenes.
This staged view captures an important historical lesson. Better optimization could improve a disparity benchmark, but it could not compensate for a loose mount, a poorly characterized lens, unsynchronized cameras, or a scene with no usable visual texture. Practical success depended on making the whole measurement chain observable and testable.
Confidence should travel with the depth map
Early stereo demonstrations often emphasized dense, coloured disparity images. Operational systems need a second output: an indication of where depth is trustworthy. Confidence can be estimated from the gap between the best and second-best match, left-right agreement, local texture, proximity to occlusions, or stability under small parameter changes. No single cue is decisive, but together they help downstream components avoid treating every depth value as equally reliable.
For example, a system inspecting a planar panel can fit a plane only to high-confidence points, then report residuals separately near shiny edges and holes. That is more defensible than fitting every available disparity value. A final practical check is equally concrete: place a calibration object at several known distances across the intended field of view, record both depth error and invalid-pixel rate, and retain those results with the camera configuration that produced them.