Semantic and instance segmentation¶
This tutorial places an RGB camera, a semantic segmentation camera and an instance segmentation camera at one pose, 16 m above the west side of Lenox Ave south of W 125th St, looking up the avenue. It saves the class mask in its palette and as raw class ids, the instance codes with the labels of every object in view, and a view with one colour per object.
Label cameras¶
pose = Transform(junction.location + Location(-45, -40, 16), Rotation(pitch=-14, yaw=42))
for kind in ("rgb", "semantic_segmentation", "instance_segmentation"):
bp = lib.find("sensor.camera." + kind)
bp.set_attribute("fov", 75)
cam = world.spawn_actor(bp, pose)
cam.listen(lambda data, kind=kind: latest.__setitem__(kind, data))
Cameras with the same parent, transform, field of view and image size share one capture per step: the server renders
the view once and renders the label images from the same scene state, so the masks register with the RGB image.
The three events carry the same frame number, which the script checks. Label cameras render at the server's resolution
unless image_size_x and image_size_y say otherwise.
Semantic segmentation¶
sem.save_to_disk(os.path.join(a.out, "semantic.png")) # class palette
sem.save_to_disk(os.path.join(a.out, "semantic_ids.png"), colorize=False) # one byte per pixel: the class id
ids = sem.to_numpy() # (H, W) uint8 class ids
A SemanticSegmentationImage holds one class id per pixel. to_numpy() returns them as an H × W array of
uint8, save_to_disk writes them either in the class palette (colorize=True, the default) or as an 8-bit
grey image of the ids, the format to train on. world.get_semantic_classes() returns the class table; the colours
follow Cityscapes where the classes overlap.
The classes, their palette colours, and whether their objects receive instance ids:
| id | class | colour | instances |
|---|---|---|---|
| 0 | unlabeled |
0, 0, 0 | |
| 1 | sky |
70, 130, 180 | |
| 2 | building |
70, 70, 70 | yes |
| 3 | road |
128, 64, 128 | |
| 4 | sidewalk |
244, 35, 232 | |
| 5 | curb |
170, 0, 180 | |
| 6 | road_marking |
255, 255, 255 | |
| 7 | crosswalk |
0, 250, 250 | |
| 8 | lane_marking_yellow |
250, 255, 100 | |
| 9 | bike_lane |
0, 230, 60 | |
| 10 | bus_lane |
180, 80, 20 | |
| 11 | gutter |
60, 20, 190 | |
| 12 | detectable_warning |
200, 140, 0 | |
| 13 | plaza |
170, 150, 70 | |
| 14 | footpath |
230, 200, 180 | |
| 15 | terrain |
152, 251, 152 | |
| 16 | grass |
40, 180, 10 | |
| 17 | vegetation |
107, 142, 35 | yes |
| 18 | water |
30, 80, 160 | |
| 19 | car |
0, 0, 142 | yes |
| 20 | bus |
0, 60, 100 | yes |
| 21 | truck |
0, 0, 70 | yes |
| 22 | bicycle |
119, 11, 32 | yes |
| 23 | pedestrian |
220, 20, 60 | yes |
| 24 | traffic_signal |
250, 170, 30 | yes |
| 25 | pedestrian_signal |
210, 90, 190 | yes |
| 26 | street_sign |
220, 220, 0 | yes |
| 27 | street_light |
153, 153, 153 | yes |
| 28 | hydrant |
230, 110, 80 | yes |
| 29 | street_furniture |
140, 100, 250 | yes |
| 30 | bus_shelter |
90, 220, 220 | yes |
| 31 | subway_entrance |
110, 0, 250 | yes |
| 32 | scaffold |
210, 180, 90 | yes |
| 33 | building_appurtenance |
120, 70, 0 | yes |
| 34 | roof_structure |
120, 160, 210 | yes |
| 35 | bridge |
20, 170, 120 |
Instance segmentation¶
inst.save_to_disk(os.path.join(a.out, "instance.png")) # 24-bit instance codes + instance.json
iid = inst.instance_ids() # (H, W) uint32
pixels = int(inst.mask(label).sum()) # the object's mask from the id image
An InstanceSegmentationImage encodes an instance id in the red, green and blue bytes of every pixel, and carries a
label for every object in view (inst.labels, a list of ObjectLabel). instance_ids() decodes the image into
an H × W array of ids, 0 where a pixel belongs to no object (the sky and the ground classes); mask(label) returns
one object's boolean mask. save_to_disk writes the codes as a lossless PNG and the labels as JSON next to it
(instance.json). Each label has the object's id, class, visible box and pixel count (area), the id of the API
actor it belongs to (0 for the city's own objects), and the object's 3D pose where it is known; the
bounding boxes tutorial uses those. An object keeps its id from frame to frame, so the ids also
serve as track ids.
The script prints the classes in view by their share of the image, and the largest objects with their visible box, their area from the labels and the pixel count of their mask:
classes in view:
2 building rgb (70, 70, 70) 41.6 %
17 vegetation rgb (107, 142, 35) 16.5 %
3 road rgb (128, 64, 128) 11.5 %
1 sky rgb (70, 130, 180) 8.2 %
15 terrain rgb (152, 251, 152) 7.2 %
4 sidewalk rgb (244, 35, 232) 6.3 %
16 grass rgb (40, 180, 10) 3.0 %
11 gutter rgb (60, 20, 190) 1.0 %
34 roof_structure rgb (120, 160, 210) 0.9 %
7 crosswalk rgb (0, 250, 250) 0.6 %
23 pedestrian rgb (220, 20, 60) 0.5 %
31 subway_entrance rgb (110, 0, 250) 0.4 %
21 truck rgb (0, 0, 70) 0.4 %
19 car rgb (0, 0, 142) 0.4 %
27 street_light rgb (153, 153, 153) 0.3 %
6 road_marking rgb (255, 255, 255) 0.2 %
5 curb rgb (170, 0, 180) 0.2 %
29 street_furniture rgb (140, 100, 250) 0.2 %
24 traffic_signal rgb (250, 170, 30) 0.2 %
10 bus_lane rgb (180, 80, 20) 0.2 %
243 labelled objects; the largest:
id 156436 building bbox [0, 102, 299, 618] area 160876 mask 160876
id 156283 building bbox [1012, 150, 268, 331] area 54634 mask 54634
id 65723 building bbox [0, 0, 322, 348] area 48698 mask 48698
id 244 vegetation bbox [1060, 448, 220, 272] area 34235 mask 34235
id 156364 building bbox [696, 171, 309, 139] area 31611 mask 31611
id 247 vegetation bbox [963, 384, 212, 300] area 27766 mask 27766
id 65724 building bbox [846, 78, 290, 219] area 23478 mask 23478
id 65732 building bbox [1071, 0, 209, 150] area 19342 mask 19342
The complete script¶
"""Semantic and instance segmentation: RGB, semantic and instance cameras at one pose above W 125th St and Lenox Ave.
Saves the class-palette mask, the raw class ids, the instance codes with their labels, and a colour-per-instance view;
prints the classes in view and the largest objects. Needs numpy.
python segmentation.py [--host 127.0.0.1] [--port 2000] [--out _out/segmentation]
"""
import argparse
import json
import os
import sys
import numpy as np
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..")) # run from a source checkout
import valdrada # noqa: E402
from valdrada import Location, Rotation, Transform # noqa: E402
from valdrada.png import write_png # noqa: E402
LENOX_125 = (40.80776, -73.94549) # W 125th St & Lenox Ave, Harlem
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, default=2000)
ap.add_argument("--out", default="_out/segmentation")
a = ap.parse_args()
os.makedirs(a.out, exist_ok=True)
client = valdrada.Client(a.host, a.port)
client.set_timeout(120.0)
world = client.get_world()
original = world.get_settings()
world.apply_settings(valdrada.WorldSettings(synchronous_mode=True, fixed_delta_seconds=0.05))
m = world.get_map()
here = m.geolocation_to_location(*LENOX_125)
world.get_spectator().set_transform(Transform(here + Location(0, 0, 30)))
world.wait_until_loaded()
junction = m.get_junctions(center=here, radius=40)[0]
# three cameras with the same pose and field of view share one capture per tick
pose = Transform(junction.location + Location(-45, -40, 16), Rotation(pitch=-14, yaw=42))
lib = world.get_blueprint_library()
latest, cameras = {}, []
for kind in ("rgb", "semantic_segmentation", "instance_segmentation"):
bp = lib.find("sensor.camera." + kind)
bp.set_attribute("fov", 75)
cam = world.spawn_actor(bp, pose)
cam.listen(lambda data, kind=kind: latest.__setitem__(kind, data))
cameras.append(cam)
try:
for _ in range(40):
world.tick()
rgb, sem, inst = latest["rgb"], latest["semantic_segmentation"], latest["instance_segmentation"]
assert rgb.frame == sem.frame == inst.frame # the three images are of the same step
rgb.save_to_disk(os.path.join(a.out, "rgb.png"))
sem.save_to_disk(os.path.join(a.out, "semantic.png")) # class palette
sem.save_to_disk(os.path.join(a.out, "semantic_ids.png"), colorize=False) # one byte per pixel: the class id
inst.save_to_disk(os.path.join(a.out, "instance.png")) # 24-bit instance codes + instance.json
classes = world.get_semantic_classes() # [{id, name, rgb, isthing}]
with open(os.path.join(a.out, "classes.json"), "w", encoding="utf-8") as f:
json.dump(classes, f, indent=1)
ids = sem.to_numpy() # (H, W) uint8 class ids
share = np.bincount(ids.ravel(), minlength=256) / ids.size
print("classes in view:")
for c in sorted(classes, key=lambda c: -share[c["id"]]):
if share[c["id"]] >= 0.001:
print(f" {c['id']:2d} {c['name']:22s} rgb {tuple(c['rgb'])} {100 * share[c['id']]:5.1f} %")
# per-instance ids (0: stuff and background); one random colour per object
iid = inst.instance_ids() # (H, W) uint32
uniq, inverse = np.unique(iid, return_inverse=True)
colours = np.random.default_rng(7).integers(48, 256, size=(len(uniq), 3), dtype=np.uint8)
colours[uniq == 0] = 0
vis = colours[inverse.reshape(iid.shape)]
write_png(os.path.join(a.out, "instance_colors.png"), inst.width, inst.height, np.ascontiguousarray(vis).tobytes(), 3)
print(f"{len(inst.labels)} labelled objects; the largest:")
for label in sorted(inst.labels, key=lambda l: -l.area)[:8]:
pixels = int(inst.mask(label).sum()) # the object's mask from the id image
print(f" id {label.id:6d} {label.class_name:14s} bbox {label.bbox} area {label.area} mask {pixels}")
finally:
for cam in cameras:
cam.destroy()
world.apply_settings(original)
if __name__ == "__main__":
main()