Blog

Computer Vision in Robotics: How Robots See, Grasp and Navigate

Synthetic Data & Simulation
9 min

Computer vision in robotics is what lets a robot find an object, work out its exact position and orientation, and move to it without a human placing everything in a fixed spot first. Cameras capture the scene, models detect and locate what matters, and the robot's controller turns that into a path or a grasp.

Most explainers stop at "robots can see now." This one follows the full chain from pixels to motion, compares the cameras robots use, explains 2D vs 3D robot vision, walks through real applications from bin picking to humanoids, and covers the objects that still trip robots up.

What Is Computer Vision in Robotics?

Computer vision in robotics is the use of cameras and AI models to give a robot an understanding of its surroundings: what objects are present, where they are in 3D space, and how the robot itself is positioned. That information drives decisions like where to grasp, where to drive and when to stop.

The difference from computer vision elsewhere is the closed loop. A photo-tagging model can be wrong and nothing happens. A robot that misjudges a part's position by two centimeters drops it, crashes into it or stops the line. Robot vision has to be accurate in 3D, fast enough for motion, and dependable on the hundredth shift as well as the first.

How a Robot Turns Pixels Into Actions

Every vision-guided robot runs some version of the same pipeline. The details change with the task, but the steps don't.

Perception: Detection, Segmentation and 6D Pose

First the robot has to find things. Object detection draws a box around each item and names it. Instance segmentation goes further and marks the exact pixels of each object, which matters when parts touch or overlap in a bin.

For manipulation, a box is not enough. The robot needs the object's 6D pose: its position on three axes plus its rotation around three axes. A gripper approaching a bracket that's rotated 30 degrees needs to rotate too. Pose estimation is usually the hardest perception step, and it's where depth data earns its keep.

Localization and Mapping (Visual SLAM)

Mobile robots also need to know where they are. Visual SLAM (simultaneous localization and mapping) tracks features across camera frames to build a map of the space and locate the robot inside it at the same time. Warehouse robots, delivery robots and the Mars rovers all use variants of it. Some combine cameras with LiDAR or wheel odometry, because cameras alone drift in long, featureless corridors.

Camera-to-Robot Calibration

A camera reports positions in its own coordinate frame. The robot moves in a different one. Hand-eye calibration computes the transform between them, and if it's off by a few millimeters, every grasp is off by the same amount.

There are two setups. Eye-in-hand mounts the camera on the robot's wrist, so it sees the target close up and from any angle, but the robot has to stop to look. Eye-to-hand fixes the camera above the work cell, which lets the robot plan while it moves but gives a single, more distant view. Bin picking often uses the first; conveyor tracking usually uses the second.

Robot Vision Sensors Compared

The choice of sensor decides what the robot can perceive and what the model has to infer. Here are the main options.

SensorWhat it measuresStrengthsWeak spotsTypical robot use
RGB camera (2D)Color imageCheap, high resolution, reads labels and textureNo direct depth, sensitive to lightingInspection, sorting, barcode and label reading
Stereo cameraDepth from two viewsWorks outdoors, passiveStruggles on plain surfaces with no textureMobile robots, agriculture
Structured lightDepth from a projected patternSub-millimeter accuracy at short rangeWashed out by sunlight, slowerBin picking, precision assembly
Time-of-flight (ToF)Depth from light travel timeFast, compact, full depth frame at onceLower resolution, multipath errors near cornersPalletizing, people detection, AMRs
LiDAR3D point cloud from laser pulsesLong range, accurate geometrySparse, costly, no colorNavigation, outdoor mobile robots

Many robots fuse two or more of these. A common pairing is an RGB camera for recognition and a depth sensor for geometry, aligned so every pixel has both a color and a distance.

2D vs 3D Robot Vision

2D vision works from a flat image. It's enough when parts lie flat in a known plane, like labels on a conveyor or components on a tray, and the robot only needs X, Y and one rotation. It's cheaper, faster and easier to set up.

3D vision adds depth. You need it when objects are stacked, tilted or piled at random, when height varies, or when the robot must grasp from an angle. Random bin picking, depalletizing mixed boxes and assembly on curved surfaces are 3D problems. A rough rule: if a human doing the task would need to judge how far away something is, the robot needs 3D vision too.

Computer Vision Applications in Robotics

Bin Picking and Vision-Guided Robotics

Vision-guided robotics (VGR) is the industrial term for arms that use cameras to find parts instead of relying on fixtures. Random bin picking is the classic case: parts lie jumbled in a container, and the robot must segment one, estimate its pose, check it can reach it without hitting the bin wall, and pick. It's one of the main reasons factories buy 3D cameras.

Warehouse and Logistics Robots

Amazon deployed its one millionth robot in July 2025, across more than 300 facilities. Its Proteus mobile robot moves carts autonomously around people, and robotic arms like Sparrow pick individual items out of totes, which means recognizing products from a huge and constantly changing catalog using camera images alone. Parcel sorting, depalletizing and truck unloading run on the same perception stack.

Inspection and Quality Control

A camera on a robot arm can inspect a part from dozens of angles that a fixed station can't reach. Car body plants use robot-mounted cameras to find paint defects and check weld seams. The vision model classifies defects, and the robot's position data records exactly where each one sits on the part.

Agricultural Robots

Carbon Robotics' LaserWeeder is a strong example of vision on a deadline. According to the manufacturer's spec, the pull-behind model uses 42 high-resolution cameras to tell weeds from crops and fires lasers at the weeds, up to 300,000 per hour. Fruit-picking robots face a related problem: they must judge ripeness from color and find stems hidden behind leaves.

Mobile Robots and Quadrupeds

Boston Dynamics' Spot walks industrial sites and reads gauges, spots leaks and checks equipment on a set route. It combines stereo cameras for terrain and obstacle detection with an optional pan-tilt-zoom camera for close inspection. Autonomous mobile robots (AMRs) in factories use the same mix of SLAM and obstacle detection to share aisles with forklifts and people.

Humanoids and Vision-Language-Action Models

The newest shift is robots that take an image and a spoken instruction and output motor actions directly. These are vision-language-action (VLA) models. Google DeepMind's RT-2 showed the idea in 2023. NVIDIA's GR00T N1, released in March 2025 as an open model for humanoids, pairs a vision-language model for reasoning with a separate module for motor control. Instead of a hand-built pipeline of detector, pose estimator and planner, one model learns the whole chain from data. That makes the training data, not the pipeline design, the main limit on what these robots can do.

Robot Tasks, Vision Models and Labels at a Glance

Robot taskCV taskLabels needed for trainingUsual sensor
Pick sorted parts from a conveyorDetection, 2D poseBounding boxes, orientationRGB, fixed above
Random bin pickingInstance segmentation, 6D poseMasks, 6D pose, depthStructured light or ToF
Depalletizing mixed boxesSegmentation, plane fittingMasks, depthToF or stereo
Weld and paint inspectionDefect classification, segmentationDefect masks and classesHigh-resolution RGB on the arm
Mobile navigationVisual SLAM, obstacle detectionBoxes, depth, camera posesStereo, LiDAR
Instruction following (VLA)End-to-end perception and controlImages paired with actions and textRGB, often several views

Where Robot Vision Still Fails

Demos hide the hard cases. These are the ones that show up in production.

Transparent and shiny objects. Glass, clear plastic packaging and polished metal confuse depth sensors. Structured light and ToF beams pass through glass or bounce off chrome, so the depth map shows holes or phantom surfaces exactly where the object is.

Lighting changes. A model trained under the integrator's lab lights meets skylights, night shifts and a welding arc next door. 2D systems suffer most.

Clutter and occlusion. In a full bin, most parts are partly hidden. The model has to estimate the pose of an object it can see only a third of.

New objects. A logistics robot sees products it has never seen before every day. A model trained on a fixed catalog needs either retraining or the ability to grasp unknown items by shape alone.

Motion blur. On fast conveyors and moving robots, frames blur. Exposure, frame rate and robot speed have to be tuned together.

Each of these is mostly a data problem. The model fails on cases it didn't see enough of in training.

Training Data for Robot Vision

Labels for robot vision are harder to collect than for most computer vision. A bounding box takes seconds to draw. An accurate 6D pose for a part in a cluttered bin can't be drawn by hand at all; it needs CAD alignment or special capture rigs. Depth ground truth needs calibrated equipment. And the failure cases above, like glass parts under changing light, are exactly the ones that are rare in collected footage.

That's why robotics teams lean on simulation. Rendering 3D models of parts in virtual cells gives perfect 6D pose, masks and depth for every frame, and lets you generate the transparent, occluded and badly lit cases on purpose. The remaining challenge is the sim-to-real gap, which teams close with photorealistic rendering, wide randomization and a smaller set of real images for fine-tuning. We cover the method in detail in our guide to synthetic data for robotics training, and the specific technique of generating edge cases for computer vision models.

How to Add Vision to a Robot Project

  1. Write down what the robot must know: object identity only, 2D position, or full 6D pose.
  2. Choose 2D or 3D based on how parts are presented, then pick the sensor from the table above.
  3. Decide eye-in-hand or eye-to-hand, and budget time for hand-eye calibration.
  4. List the hard cases early: reflective parts, mixed lighting, overlap, new SKUs.
  5. Build a dataset that covers those cases, with synthetic data for pose, depth and rare conditions and real images for validation.
  6. Test in the actual cell under real lighting before trusting lab accuracy numbers.

Step five decides how far the system gets. Vivid 3D generates labeled datasets for computer vision in robotics from 3D models: RGB, depth, point clouds, segmentation masks and 6D pose, exported in COCO, YOLO or KITTI format. If your team is building perception for robotics and physical AI, that's the fastest way to cover the cases your real data doesn't. If you need the model itself built and evaluated, Vivid 3D also offers custom computer vision development.

Frequently Asked Questions

What is the difference between computer vision and machine vision?

Machine vision is the industrial subset: cameras on production lines that inspect, measure and guide machines, often with rule-based software and controlled lighting. Computer vision is the broader field of getting computers to interpret images, including deep learning models that handle uncontrolled scenes. Modern robot vision draws on both.

Is computer vision part of AI?

Yes. Computer vision is a branch of artificial intelligence focused on interpreting images and video. Most robot vision today relies on deep learning, a type of machine learning, although some industrial systems still use classic rule-based image processing for simple, well-lit tasks.

Who are the big 4 in robotics?

In industrial robotics, the big four are usually named as FANUC, ABB, KUKA and Yaskawa. All four sell robot arms with integrated vision options for tasks like bin picking and inspection.

Does robotics use C or C++?

Robotics uses C++ heavily for real-time control and performance-critical perception code, and ROS 2, the most common robotics framework, supports C++ and Python. Python dominates model training and prototyping. Plain C appears mostly in embedded firmware on microcontrollers.

Table of contents
9 min
Share

Recommended