
Computer vision in robotics is what lets a robot find an object, work out its exact position and orientation, and move to it without a human placing everything in a fixed spot first. Cameras capture the scene, models detect and locate what matters, and the robot's controller turns that into a path or a grasp.
Most explainers stop at "robots can see now." This one follows the full chain from pixels to motion, compares the cameras robots use, explains 2D vs 3D robot vision, walks through real applications from bin picking to humanoids, and covers the objects that still trip robots up.
Computer vision in robotics is the use of cameras and AI models to give a robot an understanding of its surroundings: what objects are present, where they are in 3D space, and how the robot itself is positioned. That information drives decisions like where to grasp, where to drive and when to stop.
The difference from computer vision elsewhere is the closed loop. A photo-tagging model can be wrong and nothing happens. A robot that misjudges a part's position by two centimeters drops it, crashes into it or stops the line. Robot vision has to be accurate in 3D, fast enough for motion, and dependable on the hundredth shift as well as the first.
Every vision-guided robot runs some version of the same pipeline. The details change with the task, but the steps don't.
First the robot has to find things. Object detection draws a box around each item and names it. Instance segmentation goes further and marks the exact pixels of each object, which matters when parts touch or overlap in a bin.
For manipulation, a box is not enough. The robot needs the object's 6D pose: its position on three axes plus its rotation around three axes. A gripper approaching a bracket that's rotated 30 degrees needs to rotate too. Pose estimation is usually the hardest perception step, and it's where depth data earns its keep.
Mobile robots also need to know where they are. Visual SLAM (simultaneous localization and mapping) tracks features across camera frames to build a map of the space and locate the robot inside it at the same time. Warehouse robots, delivery robots and the Mars rovers all use variants of it. Some combine cameras with LiDAR or wheel odometry, because cameras alone drift in long, featureless corridors.
A camera reports positions in its own coordinate frame. The robot moves in a different one. Hand-eye calibration computes the transform between them, and if it's off by a few millimeters, every grasp is off by the same amount.
There are two setups. Eye-in-hand mounts the camera on the robot's wrist, so it sees the target close up and from any angle, but the robot has to stop to look. Eye-to-hand fixes the camera above the work cell, which lets the robot plan while it moves but gives a single, more distant view. Bin picking often uses the first; conveyor tracking usually uses the second.
The choice of sensor decides what the robot can perceive and what the model has to infer. Here are the main options.
| Sensor | What it measures | Strengths | Weak spots | Typical robot use |
|---|---|---|---|---|
| RGB camera (2D) | Color image | Cheap, high resolution, reads labels and texture | No direct depth, sensitive to lighting | Inspection, sorting, barcode and label reading |
| Stereo camera | Depth from two views | Works outdoors, passive | Struggles on plain surfaces with no texture | Mobile robots, agriculture |
| Structured light | Depth from a projected pattern | Sub-millimeter accuracy at short range | Washed out by sunlight, slower | Bin picking, precision assembly |
| Time-of-flight (ToF) | Depth from light travel time | Fast, compact, full depth frame at once | Lower resolution, multipath errors near corners | Palletizing, people detection, AMRs |
| LiDAR | 3D point cloud from laser pulses | Long range, accurate geometry | Sparse, costly, no color | Navigation, outdoor mobile robots |
Many robots fuse two or more of these. A common pairing is an RGB camera for recognition and a depth sensor for geometry, aligned so every pixel has both a color and a distance.
2D vision works from a flat image. It's enough when parts lie flat in a known plane, like labels on a conveyor or components on a tray, and the robot only needs X, Y and one rotation. It's cheaper, faster and easier to set up.
3D vision adds depth. You need it when objects are stacked, tilted or piled at random, when height varies, or when the robot must grasp from an angle. Random bin picking, depalletizing mixed boxes and assembly on curved surfaces are 3D problems. A rough rule: if a human doing the task would need to judge how far away something is, the robot needs 3D vision too.
Vision-guided robotics (VGR) is the industrial term for arms that use cameras to find parts instead of relying on fixtures. Random bin picking is the classic case: parts lie jumbled in a container, and the robot must segment one, estimate its pose, check it can reach it without hitting the bin wall, and pick. It's one of the main reasons factories buy 3D cameras.
Amazon deployed its one millionth robot in July 2025, across more than 300 facilities. Its Proteus mobile robot moves carts autonomously around people, and robotic arms like Sparrow pick individual items out of totes, which means recognizing products from a huge and constantly changing catalog using camera images alone. Parcel sorting, depalletizing and truck unloading run on the same perception stack.
A camera on a robot arm can inspect a part from dozens of angles that a fixed station can't reach. Car body plants use robot-mounted cameras to find paint defects and check weld seams. The vision model classifies defects, and the robot's position data records exactly where each one sits on the part.
Carbon Robotics' LaserWeeder is a strong example of vision on a deadline. According to the manufacturer's spec, the pull-behind model uses 42 high-resolution cameras to tell weeds from crops and fires lasers at the weeds, up to 300,000 per hour. Fruit-picking robots face a related problem: they must judge ripeness from color and find stems hidden behind leaves.
Boston Dynamics' Spot walks industrial sites and reads gauges, spots leaks and checks equipment on a set route. It combines stereo cameras for terrain and obstacle detection with an optional pan-tilt-zoom camera for close inspection. Autonomous mobile robots (AMRs) in factories use the same mix of SLAM and obstacle detection to share aisles with forklifts and people.
The newest shift is robots that take an image and a spoken instruction and output motor actions directly. These are vision-language-action (VLA) models. Google DeepMind's RT-2 showed the idea in 2023. NVIDIA's GR00T N1, released in March 2025 as an open model for humanoids, pairs a vision-language model for reasoning with a separate module for motor control. Instead of a hand-built pipeline of detector, pose estimator and planner, one model learns the whole chain from data. That makes the training data, not the pipeline design, the main limit on what these robots can do.
| Robot task | CV task | Labels needed for training | Usual sensor |
|---|---|---|---|
| Pick sorted parts from a conveyor | Detection, 2D pose | Bounding boxes, orientation | RGB, fixed above |
| Random bin picking | Instance segmentation, 6D pose | Masks, 6D pose, depth | Structured light or ToF |
| Depalletizing mixed boxes | Segmentation, plane fitting | Masks, depth | ToF or stereo |
| Weld and paint inspection | Defect classification, segmentation | Defect masks and classes | High-resolution RGB on the arm |
| Mobile navigation | Visual SLAM, obstacle detection | Boxes, depth, camera poses | Stereo, LiDAR |
| Instruction following (VLA) | End-to-end perception and control | Images paired with actions and text | RGB, often several views |
Demos hide the hard cases. These are the ones that show up in production.
Transparent and shiny objects. Glass, clear plastic packaging and polished metal confuse depth sensors. Structured light and ToF beams pass through glass or bounce off chrome, so the depth map shows holes or phantom surfaces exactly where the object is.
Lighting changes. A model trained under the integrator's lab lights meets skylights, night shifts and a welding arc next door. 2D systems suffer most.
Clutter and occlusion. In a full bin, most parts are partly hidden. The model has to estimate the pose of an object it can see only a third of.
New objects. A logistics robot sees products it has never seen before every day. A model trained on a fixed catalog needs either retraining or the ability to grasp unknown items by shape alone.
Motion blur. On fast conveyors and moving robots, frames blur. Exposure, frame rate and robot speed have to be tuned together.
Each of these is mostly a data problem. The model fails on cases it didn't see enough of in training.
Labels for robot vision are harder to collect than for most computer vision. A bounding box takes seconds to draw. An accurate 6D pose for a part in a cluttered bin can't be drawn by hand at all; it needs CAD alignment or special capture rigs. Depth ground truth needs calibrated equipment. And the failure cases above, like glass parts under changing light, are exactly the ones that are rare in collected footage.
That's why robotics teams lean on simulation. Rendering 3D models of parts in virtual cells gives perfect 6D pose, masks and depth for every frame, and lets you generate the transparent, occluded and badly lit cases on purpose. The remaining challenge is the sim-to-real gap, which teams close with photorealistic rendering, wide randomization and a smaller set of real images for fine-tuning. We cover the method in detail in our guide to synthetic data for robotics training, and the specific technique of generating edge cases for computer vision models.
Step five decides how far the system gets. Vivid 3D generates labeled datasets for computer vision in robotics from 3D models: RGB, depth, point clouds, segmentation masks and 6D pose, exported in COCO, YOLO or KITTI format. If your team is building perception for robotics and physical AI, that's the fastest way to cover the cases your real data doesn't. If you need the model itself built and evaluated, Vivid 3D also offers custom computer vision development.
Machine vision is the industrial subset: cameras on production lines that inspect, measure and guide machines, often with rule-based software and controlled lighting. Computer vision is the broader field of getting computers to interpret images, including deep learning models that handle uncontrolled scenes. Modern robot vision draws on both.
Yes. Computer vision is a branch of artificial intelligence focused on interpreting images and video. Most robot vision today relies on deep learning, a type of machine learning, although some industrial systems still use classic rule-based image processing for simple, well-lit tasks.
In industrial robotics, the big four are usually named as FANUC, ABB, KUKA and Yaskawa. All four sell robot arms with integrated vision options for tasks like bin picking and inspection.
Robotics uses C++ heavily for real-time control and performance-critical perception code, and ROS 2, the most common robotics framework, supports C++ and Python. Python dominates model training and prototyping. Plain C appears mostly in embedded firmware on microcontrollers.
