Unitree B2-W: Learned Navigation among People for a Wheeled Quadruped

Safety shield

On the robot every command passes a shield that is independent of the policy (g1_patrol.shield). A command is admissible if the robot, executing it for 0.35 s (sensing, policy and controller latency) and then braking at 0.8 m/s², with 12% more speed than commanded and 8 cm of drift per meter, can stop before any sensor return comes within a margin of its footprint: 0.10 m to obstacles, 0.25 m to drops, and to tall things (people, but also machines and walls) from 0.10 m below 0.25 m/s to 0.45 m above 0.75 m/s, so that doors are passed slowly and people with room. Commands are scaled down by bisection until they are admissible. Returns where the current scan sees something tall and the previous one saw floor are assumed to be people walking at the robot at 1.4 m/s while it closes in on them, and the closing speed towards tall things is held to 0.25 m/s within 0.8 m. Moving away is never limited. A guard around the shield replaces a command that is nearly blocked by a safe one near it (the translation or the turn alone, a nudge sideways, a step back) when the robot is at rest, and backs the robot off after 1.5 s of being blocked.

The shield alone lowers structure contacts from 10.3% to 3.2% of the episodes and contacts in which the robot moves into a pedestrian from 6.9% to 0.7%; the guard raises the goal rate from 73.5% to 76.0% and lowers structure contacts to 2.2% (Table 2). The alternatives must be checked against the same returns as the command, those closest to the footprint: checked against the returns closest to the robot's center, a wall alongside crowded out what lay ahead, and structure contacts rose to 9%. Crawling lowers contacts in which the robot moves into a pedestrian to 0.6% and costs 19 points of goal rate: the returns that appear where the previous scan saw floor come also from static things coming into view as the robot moves. The robot crawls by default and drives without crawling where nobody walks (crawl:=false). Most remaining pedestrian contacts are pedestrians walking into a robot that stands or crawls (22.6% of the episodes); a quarter of the simulated pedestrians are distracted and give way only at the last moment.

On the robot

One ROS 2 launch file starts everything on the robot's Jetson Orin NX: the lidar driver and the depth cameras, odometry from the sport-mode state, SLAM (slam_toolbox) on a slice of the scan, a route planner on the map being built (unknown space counts as free, so the robot can be sent into unmapped rooms), the policy (ONNX) with its height map and the shield, and the output to the sport-mode controller as Move(vx, vy, wz). The policy drives only while a dead-man button on the remote is held; moving a stick takes over, and missing scans, odometry or remote messages stop the robot. An install script sets up ROS 2 Foxy or Humble, Unitree's messages and the lidar driver; the deployment guide goes from the network to a shadow run (the policy drives nothing while the robot is steered by hand and the map is built) to the first drive.

Before the robot, the whole stack was run against a simulated B2-W on ROS 2 (deploy/b2w/sim) that speaks like the robot, ray-casts the lidar's point cloud in a section of the lab hall with walking people and moves with the controller model of the training (Table 4).

Results

All evaluations are on the lab mixture (half the tiles sections of the lab hall, half street and indoor tiles), with robots spread over all ten difficulty levels and up to 20 pedestrians, a quarter of whom are distracted and give way only at the last moment; 1,024 robots for 300 s. With the shield, the speed cap is applied before the shield, as on the robot.

The fused student reaches 72.5% of its goals and the lidar-only student 73.1%, against 69.4% for the privileged teacher, which touches structures and pedestrians more often. A built-in controller with more latency, a slower response, weaker tracking or more drift, failed depth cameras and a degraded lidar map each change the goal rate by at most 1.3 points (Table 6). With the shield at the speed caps of the first trials (0.5 and 1.0 m/s) the goal rate is 53.9% and 58.7%, and with a laggy controller 57.1%.

Civil engineering lab

On a mixture in which the civil engineering lab takes half of the lab tiles, the released policy reaches 71.9% of its goals there without the shield and 57.0% with it, against 80.7% and 71.4% in the lab hall (Table 7). The lab leaves passages of at least 0.9 m between the things on its strong floor, so that no route passes within 0.5 m of a structure, but it is denser: the privileged teacher, which observes the distances to structures, touched structures in 15% of the easiest civil-lab episodes (10% in the lab hall). Fine-tuning the lab teacher and distilling the students on that mixture did not improve on it: 72.2% of the goals in the civil engineering lab against 71.9% without the shield and 58.7% against 57.0% with it, while the goals in the lab hall fell to 66.9% with the shield (from 71.4%) and contacts in which the robot moved towards a pedestrian rose to 10.2% of the episodes without it (from 8.8%). The released policy is the one trained without the civil engineering lab.

Camera student

A third student also sees the image of the B2-W's front color camera: the class token and 2 × 3 average-pooled patch tokens of a frozen DINOv2-small (2,688 features) of a 336 × 196 image, besides the fused student's inputs. During training the camera sees the environment drawn as in the demo: textured tiles, indoor rooms under ceilings with light fixtures, furniture drawn as NVIDIA SimReady props with tools, monitors and boxes on the work surfaces and boards and signs on the walls, and every pedestrian as an animated character that only its own robot's camera sees; the geometry the robot and its sensors interact with is unchanged. Started from the released fused student with the camera's weights at zero and distilled for 300 iterations with 256 robots, it reaches 70.5% of its goals in simulation (57.1% with the shield), against 73.5% (60.3%) for the fused student (Table 8): the camera does not improve the policy in simulation.

Its encoder is aligned to realistic images after training rather than trained on them. The 61 indoor replays of the camera environment (of 126) are transferred to realistic video by NVIDIA Cosmos 3 Nano, the render's edges and depth keeping the layout and a text prompt per replay drawing wall paint, floor, ceiling, light, equipment colors and clothing at random (Figure 3). The last 8 of the encoder's 12 blocks are fine-tuned so that a transferred frame gives the patch tokens the original encoder gives its render; the policy is unchanged. On held-out replays, realistic images change the policy's actions by 0.110 with the original encoder and by 0.058 with the aligned one, against 0.141 without the camera's information; the cosine similarity of the camera features to the renders' rises from 0.77 to 0.93. The camera student is not part of the robot stack yet.

Pedestrians as people

In training, the lidar and the depth cameras see every pedestrian as a capsule 0.5 m wide and 1.7 m tall. To measure what this costs, they can see Isaac Sim characters in walking poses instead: the 10 characters of the demo, each skinned at 8 phases of its gait cycle and decimated to 4,000 triangles, facing the way the pedestrian walks; contacts remain those of the capsule. A person gives the lidar about two thirds of a capsule's returns at every distance (Figure 4). Seeing people, the released fused student moves into pedestrians in 9.6% of the episodes instead of 6.9% and the lidar-only student in 9.9% instead of 7.0%, at goal rates 0.8 and 2.2 points lower (Table 9); the shield, which acts on the sensor returns themselves, keeps these contacts at 0.4% (0.6% with capsules).

Fine-tuning the students for 500 iterations with their sensors seeing people (DAgger with the same teacher, from the released students) closes the gap: with people, the fused student moves into pedestrians in 6.5% of the episodes and reaches 73.4% of its goals, and 6.3% and 73.7% with 20 poses not used in the fine-tuning, while with capsules it does as before (7.0%, 72.5%). The lidar-only student improves alike with people (7.7%, 72.7%), at a small cost with capsules (7.4%, 72.3%). Structure contacts rise with fewer pedestrian contacts (10.1% against 9.3% for the fused student with people), since an episode ends at its first contact. With the shield, the fine-tuned and the released fused students do equally well (0.3% and 0.4% with people). With the characters walking, each pedestrian's pose following its own gait, the results hold: 9.1% for the released fused student and 6.4% for the fine-tuned one (6.5% with two characters held out of the draw), 9.5% and 7.7% for the lidar-only students. Evaluations with people are not bitwise reproducible, unlike those with capsules: over four runs, contacts in which the robot moved varied by at most 0.6 points and goal rates by at most 0.8.

Limitations

• All results are in simulation; the stack has been run against a simulated robot over ROS 2, not yet on the B2-W.
• The robot is simulated as a kinematic body following a model of its controller; the controller model is fitted to the training ranges, not yet to logs of this robot (sysid.py fits it once there are logs).
• Pedestrians are kinematic and scripted, and their contacts end an episode; most remaining contacts with the shield are pedestrians walking into a robot that has stopped.
• In training the lidar and the depth cameras see pedestrians as capsules. Posed Isaac Sim characters stand in for people in Table 9: the 10 characters of the demo at 8 phases of one walk cycle, while contacts are still those of the capsule.
• The shield sees what the lidar, the depth cameras and the map show; low objects beside the robot that none of them covers are remembered only by the map. Without a person detector, people are recognized only by moving.
• In the denser civil engineering lab the policy reaches 57% of its goals with the shield (71% in the lab hall), and training with that lab has not improved it.
• The camera student's alignment is measured on image-to-image transfers of simulated replays, not on images from the robot; the field of view of the B2-W's front color camera (90° horizontal) is assumed.
• The lab tiles are modeled on photographs; a scan of the hall can be turned into a digital twin (scripts/b2w/occupancy_to_twin.py) and evaluated before driving there.