Bounding boxes¶
This tutorial places an RGB camera and a bounding-box camera at one pose above Amsterdam Ave, looking north across W 120th St. It draws the visible and amodal 2D boxes and the projected 3D boxes of the road users, and exports the labels of the frame as JSON.
The bounding-box camera¶
bp = lib.find("sensor.camera.bounding_boxes")
bp.set_attribute("fov", 65)
bp.set_attribute("amodal", 48) # amodal box and occlusion for the 48 largest objects
sensor.camera.bounding_boxes returns the labels of the instance segmentation camera without its image: one
ObjectLabel per object with at least min_pixels visible pixels (30 by default). By default each label has the
visible box only. With amodal = N, the N largest objects in view are also rendered alone, which gives their amodal
box, the extent of the whole object including the parts hidden behind others, and their occlusion, 1 minus the ratio
of visible to amodal area. The N largest objects are counted over all classes, and buildings and trees are often the
largest, so this script asks for 48. Each amodal object costs about 5 ms per capture.
2D boxes and occlusion¶
users = boxes.filter(classes=ROAD_USERS, min_area=150)
...
rectangle(img2d, label.bbox, colour, width=3)
if label.amodal_bbox and (label.occlusion or 0) > 0.05:
rectangle(img2d, label.amodal_bbox, colour, width=2, dash=8) # the occluded extent, dashed
Boxes are [x, y, w, h] in pixels from the top-left corner. bbox bounds the visible pixels, area counts them, and
truncated marks a box that touches the image border. filter selects labels by class name, visible area and
whether they belong to an API actor (actors_only=True).
3D boxes¶
def box_corners(label):
c, e, yaw = label.location, label.extent, math.radians(label.yaw)
fx, fy = math.cos(yaw), math.sin(yaw)
return [Location(c.x + sx * e.x * fx - sy * e.y * fy, c.y + sx * e.x * fy + sy * e.y * fx, c.z + up * 2 * e.z)
for sx in (-1, 1) for sy in (-1, 1) for up in (0, 1)]
...
pts = [project_point(p, boxes.transform, boxes.width, boxes.height, boxes.fov) for p in box_corners(label)]
Labels carry the object's pose in the world frame where it is known. location is the point on the ground below the
object's centre, extent holds its half sizes (x along its heading, y across it, z up) and yaw its heading in
degrees, counter-clockwise from east. Vehicles have all three; pedestrians have location and extent but no
heading, trees and street furniture a location only, and buildings none of them. valdrada.util.project_point
projects a world point into the camera that captured the labels, whose pose is the event's transform, and returns
the pixel coordinates and the depth, or None behind the camera.
Per-frame JSON¶
save_to_disk writes to_dict(): the frame number, the timestamp, the image size, the field of view, the sensor's
world pose, the renderer's description of the camera (with the intrinsics K) and one entry per object. One entry of
the file this run wrote:
{
"id": 64079,
"class_id": 19,
"class_name": "car",
"actor_id": 0,
"bbox": [
452,
283,
27,
27
],
"amodal_bbox": [
377,
272,
102,
40
],
"occlusion": 0.8034,
"truncated": false,
"area": 563,
"yaw": -28.121,
"location": {
"x": 989.03,
"y": 2919.66,
"z": 3.39
},
"extent": {
"x": 2.115,
"y": 1.05,
"z": 0.825
}
}
The script prints the road users in view, largest first:
144 labelled objects, 20 road users
class visible box amodal box occlusion distance
car [390, 508, 204, 212] [390, 508, 204, 212] 0.00 14.9 m
car [931, 276, 68, 41] [931, 276, 68, 41] 0.02 45.6 m
pedestrian [253, 409, 32, 77] [253, 409, 33, 80] 0.04 22.2 m
car [944, 222, 44, 34] [944, 222, 44, 34] 0.00 70.9 m
pedestrian [281, 404, 37, 70] [281, 404, 38, 85] 0.43 21.9 m
car [847, 216, 35, 30] [847, 216, 35, 30] 0.00 75.9 m
pedestrian [1242, 306, 31, 52] [1242, 306, 31, 52] 0.00 39.6 m
car [452, 283, 27, 27] [377, 272, 102, 40] 0.80 45.4 m
car [720, 217, 29, 25] [720, 217, 29, 25] 0.00 77.4 m
truck [884, 188, 24, 24] [884, 188, 24, 24] 0.02 115.7 m
pedestrian [485, 287, 14, 43] [485, 287, 14, 43] 0.00 38.8 m
pedestrian [450, 371, 25, 77] [450, 371, 25, 77] 0.66 23.9 m
car [799, 196, 24, 21] [799, 196, 24, 22] 0.07 104.7 m
car [804, 213, 23, 19] [804, 213, 23, 19] 0.00 87.6 m
truck [741, 189, 20, 26] [739, 189, 22, 26] 0.30 107.8 m
car [874, 208, 24, 18] [874, 208, 24, 18] 0.12 94.9 m
car [777, 190, 34, 11] [777, 190, 35, 12] 0.10 134.1 m
pedestrian [1080, 237, 11, 30] [1080, 237, 11, 30] 0.00 63.7 m
pedestrian [1030, 230, 12, 28] [1030, 230, 12, 28] 0.00 68.0 m
pedestrian [638, 232, 10, 28] [638, 232, 10, 28] 0.00 62.1 m
The complete script¶
"""Bounding boxes: an RGB camera and a bounding-box camera at one pose on Amsterdam Ave, looking north across W 120th
St. Draws the visible and amodal 2D boxes and the projected 3D boxes of the road users, and exports the frame's labels
as JSON. Needs numpy.
python bounding_boxes.py [--host 127.0.0.1] [--port 2000] [--out _out/bounding_boxes]
"""
import argparse
import math
import os
import sys
import numpy as np
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..")) # run from a source checkout
import valdrada # noqa: E402
from valdrada import Location, Rotation, Transform # noqa: E402
from valdrada.png import write_png # noqa: E402
from valdrada.util import project_point # noqa: E402
AMSTERDAM_120 = (40.80955, -73.95905) # W 120th St & Amsterdam Ave, Morningside Heights
UPTOWN = 61.0 # Manhattan's avenues run at this yaw (degrees counter-clockwise from east)
ROAD_USERS = ("car", "bus", "truck", "bicycle", "pedestrian")
COLOUR = {"car": (255, 196, 0), "bus": (80, 200, 255), "truck": (80, 200, 255), "bicycle": (150, 255, 90),
"pedestrian": (255, 70, 120)}
def line(img, p, q, colour, width=2):
"""Draw a segment from p to q (pixel coordinates) into an (H, W, 3) array."""
n = int(max(abs(q[0] - p[0]), abs(q[1] - p[1]))) + 2
xs, ys = np.linspace(p[0], q[0], n), np.linspace(p[1], q[1], n)
for dx in range(-(width // 2), width - width // 2):
for dy in range(-(width // 2), width - width // 2):
x, y = np.round(xs + dx).astype(int), np.round(ys + dy).astype(int)
ok = (x >= 0) & (x < img.shape[1]) & (y >= 0) & (y < img.shape[0])
img[y[ok], x[ok]] = colour
def rectangle(img, box, colour, width=2, dash=0):
"""[x, y, w, h]; dash > 0 draws dashes of that length."""
x, y, w, h = box
corners = [(x, y), (x + w, y), (x + w, y + h), (x, y + h), (x, y)]
for p, q in zip(corners, corners[1:]):
if not dash:
line(img, p, q, colour, width)
continue
length = math.hypot(q[0] - p[0], q[1] - p[1])
for s in np.arange(0.0, length, 2 * dash):
t0, t1 = s / length, min(1.0, (s + dash) / length)
line(img, (p[0] + (q[0] - p[0]) * t0, p[1] + (q[1] - p[1]) * t0), (p[0] + (q[0] - p[0]) * t1, p[1] + (q[1] - p[1]) * t1), colour, width)
def box_corners(label):
"""The 8 world-frame corners of an object's 3D box: location is the ground point under its centre, extent the half
sizes (x along its heading), yaw its heading."""
c, e, yaw = label.location, label.extent, math.radians(label.yaw)
fx, fy = math.cos(yaw), math.sin(yaw)
return [Location(c.x + sx * e.x * fx - sy * e.y * fy, c.y + sx * e.x * fy + sy * e.y * fx, c.z + up * 2 * e.z)
for sx in (-1, 1) for sy in (-1, 1) for up in (0, 1)]
EDGES = [(i, j) for i in range(8) for j in range(i + 1, 8) if bin(i ^ j).count("1") == 1] # the 12 box edges
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, default=2000)
ap.add_argument("--out", default="_out/bounding_boxes")
a = ap.parse_args()
os.makedirs(a.out, exist_ok=True)
client = valdrada.Client(a.host, a.port)
client.set_timeout(120.0)
world = client.get_world()
original = world.get_settings()
world.apply_settings(valdrada.WorldSettings(synchronous_mode=True, fixed_delta_seconds=0.05))
m = world.get_map()
here = m.geolocation_to_location(*AMSTERDAM_120)
world.get_spectator().set_transform(Transform(here + Location(0, 0, 30)))
world.wait_until_loaded()
junction = m.get_junctions(center=here, radius=60)[0]
# 7 m above the west side of the avenue, 45 m south of the junction, looking up the avenue
up = Location(math.cos(math.radians(UPTOWN)), math.sin(math.radians(UPTOWN)), 0)
east = Location(up.y, -up.x, 0)
pose = Transform(junction.location - up * 45 - east * 9 + Location(0, 0, 7), Rotation(pitch=-12, yaw=UPTOWN + 8))
lib = world.get_blueprint_library()
latest, cameras = {}, []
for kind in ("rgb", "bounding_boxes"):
bp = lib.find("sensor.camera." + kind)
bp.set_attribute("fov", 65)
if kind == "bounding_boxes":
bp.set_attribute("amodal", 48) # amodal box and occlusion for the 48 largest objects
cam = world.spawn_actor(bp, pose)
cam.listen(lambda data, kind=kind: latest.__setitem__(kind, data))
cameras.append(cam)
try:
for _ in range(40):
world.tick()
rgb, boxes = latest["rgb"], latest["bounding_boxes"]
rgb.save_to_disk(os.path.join(a.out, "rgb.png"))
boxes.save_to_disk(os.path.join(a.out, "labels.json")) # every labelled object of the frame
users = boxes.filter(classes=ROAD_USERS, min_area=150)
img2d, img3d = rgb.to_numpy()[..., :3].copy(), rgb.to_numpy()[..., :3].copy()
print(f"{len(boxes.labels)} labelled objects, {len(users)} road users")
print(f" {'class':10s} {'visible box':22s} {'amodal box':22s} occlusion distance")
for label in sorted(users, key=lambda l: -l.area):
colour = COLOUR[label.class_name]
rectangle(img2d, label.bbox, colour, width=3)
if label.amodal_bbox and (label.occlusion or 0) > 0.05:
rectangle(img2d, label.amodal_bbox, colour, width=2, dash=8) # the occluded extent, dashed
dist = label.location.distance(boxes.transform.location) if label.location else float("nan")
occl = f"{label.occlusion:9.2f}" if label.occlusion is not None else " -"
print(f" {label.class_name:10s} {str(label.bbox):22s} {str(label.amodal_bbox):22s} {occl} {dist:6.1f} m")
if label.yaw is None or label.extent is None: # 3D boxes for the objects with a heading
continue
pts = [project_point(p, boxes.transform, boxes.width, boxes.height, boxes.fov) for p in box_corners(label)]
if all(pts):
for i, j in EDGES:
line(img3d, pts[i][:2], pts[j][:2], colour, width=2)
write_png(os.path.join(a.out, "boxes_2d.png"), rgb.width, rgb.height, img2d.tobytes(), 3)
write_png(os.path.join(a.out, "boxes_3d.png"), rgb.width, rgb.height, img3d.tobytes(), 3)
finally:
for cam in cameras:
cam.destroy()
world.apply_settings(original)
if __name__ == "__main__":
main()