A Depth Sensor is a hardware device or sensor modality that measures the distance between the sensor and objects in a scene, producing per-pixel or per-point depth information as its primary output. Operating principles include structured light projection, time-of-flight measurement, stereo vision correlation, and LiDAR pulse ranging. Depth sensors are fundamental components in robotics, augmented and mixed reality, autonomous vehicles, and industrial inspection, providing the three-dimensional scene understanding that colour cameras alone cannot supply. The output is typically represented as a depth map, disparity map, or three-dimensional Point Cloud, forming the input to downstream perception and scene reconstruction pipelines.
Content
- Depth sensors can be categorised by their operating principle. Structured-light sensors, such as the original Microsoft Kinect, project a known infrared pattern onto the scene and use a separate infrared camera to observe the pattern’s deformation; the degree of deformation encodes depth. This approach works well indoors but degrades in bright sunlight and at long ranges. Time-of-flight sensors emit a modulated light signal and measure the phase shift or time delay of the returning reflection, yielding dense depth maps at video frame rates and tolerating more varied lighting conditions.
- Stereo vision systems use two or more spatially separated cameras to compute depth from disparity: points that are close to the cameras appear at different horizontal positions in the left and right images, and the magnitude of that difference is inversely proportional to distance. Stereo depth is passive (requiring no active emitter), making it suitable for outdoor and long-range scenarios, but requires textured surfaces for reliable matching. Deep learning-based stereo matching has substantially improved accuracy over classical block-matching approaches.
- LiDAR (Light Detection and Ranging) sensors use pulsed or frequency-modulated laser light to achieve high-accuracy, long-range depth measurement. Rotating mechanical LiDAR units produce 360-degree Point Cloud scans and have been the primary sensor modality for autonomous vehicle outdoor perception. Solid-state LiDAR using OPA (optical phased arrays) or MEMS mirrors is reducing size, cost, and moving parts, accelerating deployment in consumer robotics and advanced driver assistance systems.
- In augmented and mixed reality headsets, depth sensors enable real-time mesh reconstruction of the user’s environment, supporting occlusion rendering, surface interaction, and spatial anchoring of virtual content. The Microsoft HoloLens and Meta Quest Pro both incorporate time-of-flight depth sensors for this purpose, while the Apple Vision Pro uses a multi-camera stereo approach combined with structured light for face tracking.
- Calibration is a critical process for depth sensors: intrinsic parameters (focal length, principal point, lens distortion) must be accurately estimated, and if a depth sensor is used alongside a colour camera, extrinsic calibration aligns the two coordinate frames. Camera Calibration procedures for depth sensors must account for the specific properties of the depth modality, including depth-dependent noise characteristics and edge blurring artefacts at depth discontinuities.