Cameras¶
This tutorial spawns a taxi on Lenox Ave, hands it to the traffic simulation, attaches a chase camera and a dashcam, and saves one frame per simulated second from each camera while the taxi drives north through W 125th St.
Spawning a vehicle¶
while True:
light = world.get_traffic_light_state(UPTOWN)
if light.state == "green" and light.time_left > 12.0:
break
world.tick()
...
start = m.get_waypoint(here - up * 60.0 + right * 5.0, heading=UPTOWN)
taxi = world.spawn_actor(lib.find("vehicle.taxi2"), start.transform)
taxi.set_autopilot(True, route=["straight", "straight"]) # the traffic simulation drives it
Every signal in the city runs the same cycle. get_traffic_light_state(yaw) returns the state of the signals for
traffic travelling at that heading and the time left in it, and the script steps the world until a green phase for
uptown traffic has just begun, so that the taxi reaches W 125th St while the light is green. It also thins the city's
background traffic with world.set_ambient_traffic(vehicles=200), so that no queue blocks the lane, and restores the
previous amount when it exits.
get_waypoint snaps a location to the nearest lane of the loaded street network and returns a Waypoint with the
lane's pose, width and speed limit; heading selects the direction of travel. Lenox Ave has a median, so the query
point lies 5 m east of the centre line, over the northbound carriageway, 60 m south of the junction. A waypoint's
transform is a valid spawn pose; spawn_actor sets vehicles and walkers on the ground under the location it is given.
A spawned vehicle is under manual control until set_autopilot(True). On autopilot the traffic simulation drives it:
it follows its lane, keeps its distance to the vehicle ahead, stops for red lights and yields to pedestrians. route
lists the turns to take at the next junctions, here straight on twice. The blueprint library holds the vehicle kinds
of the NYC fleet (lib.filter("vehicle.*")).
Attaching cameras¶
rigs = {
"chase": (Transform(Location(-7.0, 0.0, 3.0), Rotation(pitch=-12)), 90),
"dashcam": (Transform(Location(0.9, 0.0, 1.55), Rotation(pitch=-3)), 100),
}
...
cam = world.spawn_actor(bp, offset, attach_to=taxi)
frames[name] = queue.Queue()
cam.listen(frames[name].put)
With attach_to, a camera's transform is an offset in its parent's frame (x forward, y left, z up) and the camera
follows the parent. The chase camera sits 7 m behind the taxi and 3 m above the road, pitched down by 12°; the dashcam
sits behind the windscreen with a wider field of view. fov is the horizontal field of view in degrees. RGB cameras
render at the server's resolution (--res).
Saving frames¶
frame = world.tick()
images = {name: q.get(timeout=30) for name, q in frames.items()} # one image per camera per tick
...
image.save_to_disk(os.path.join(a.out, name, f"{frame:06d}.png"))
Each listening camera delivers one image per step before tick() returns, so one image per camera is waiting in its
queue after every tick. save_to_disk writes a PNG and creates the folders it needs; to_numpy() returns the image as
an H × W × 4 array instead.
The first RGB camera that listens is the primary view: its image is the frame the server renders for the step, and the server window shows it. Every further camera pose costs one more rendered frame per step. Cameras at the same pose share one capture, which the following tutorials use.
The script prints:
uptown light: TrafficLightState(green, 14.9 s left, north_south)
taxi 101 on road 848 lane 1 speed limit 40.2 km/h
t = 0.00 s frame 802 taxi 0.1 m/s at Location(x=2144.017, y=2684.707, z=3.387)
t = 1.00 s frame 822 taxi 2.0 m/s at Location(x=2144.546, y=2685.654, z=3.387)
t = 2.00 s frame 842 taxi 3.8 m/s at Location(x=2145.984, y=2688.228, z=3.387)
t = 3.00 s frame 862 taxi 5.6 m/s at Location(x=2148.303, y=2692.380, z=3.387)
t = 4.00 s frame 882 taxi 7.3 m/s at Location(x=2151.471, y=2698.053, z=3.387)
t = 5.00 s frame 902 taxi 8.3 m/s at Location(x=2155.315, y=2704.936, z=3.387)
t = 6.00 s frame 922 taxi 8.6 m/s at Location(x=2159.474, y=2712.381, z=3.387)
t = 7.00 s frame 942 taxi 2.8 m/s at Location(x=2162.146, y=2717.166, z=3.387)
t = 8.00 s frame 962 taxi 1.0 m/s at Location(x=2162.972, y=2718.645, z=3.387)
t = 9.00 s frame 982 taxi 0.3 m/s at Location(x=2163.240, y=2719.126, z=3.387)
t = 10.00 s frame 1002 taxi 0.0 m/s at Location(x=2163.292, y=2719.218, z=3.387)
t = 11.00 s frame 1022 taxi 0.0 m/s at Location(x=2163.292, y=2719.218, z=3.387)
The complete script¶
"""Cameras: spawn a taxi on autopilot on Lenox Ave, attach a chase camera and a dashcam to it, and save their frames
while it drives north through W 125th St.
python cameras.py [--host 127.0.0.1] [--port 2000] [--seconds 12] [--every 20] [--out _out/cameras]
"""
import argparse
import math
import os
import queue
import sys
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..")) # run from a source checkout
import valdrada # noqa: E402
from valdrada import Location, Rotation, Transform # noqa: E402
LENOX_125 = (40.80776, -73.94549) # W 125th St & Lenox Ave, Harlem
UPTOWN = 61.0 # Manhattan's avenues run at this yaw (degrees counter-clockwise from east)
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, default=2000)
ap.add_argument("--seconds", type=float, default=12.0, help="simulated time to record")
ap.add_argument("--every", type=int, default=20, help="save every N-th step (20 steps = 1 s)")
ap.add_argument("--out", default="_out/cameras")
a = ap.parse_args()
client = valdrada.Client(a.host, a.port)
client.set_timeout(120.0)
world = client.get_world()
original, ambient = world.get_settings(), world.get_ambient_traffic()
world.apply_settings(valdrada.WorldSettings(synchronous_mode=True, fixed_delta_seconds=0.05))
world.set_ambient_traffic(vehicles=200) # thinner background traffic, for a clear run up the avenue
m = world.get_map()
here = m.geolocation_to_location(*LENOX_125)
world.get_spectator().set_transform(Transform(here + Location(0, 0, 30)))
world.wait_until_loaded()
# every signal runs the same cycle: wait for a fresh green uptown, so that the taxi drives through the junction
while True:
light = world.get_traffic_light_state(UPTOWN)
if light.state == "green" and light.time_left > 12.0:
break
world.tick()
print("uptown light:", light)
# a northbound lane 60 m south of the junction (east of the median): its waypoint pose is a valid spawn pose
up = Location(math.cos(math.radians(UPTOWN)), math.sin(math.radians(UPTOWN)), 0)
right = Location(up.y, -up.x, 0)
start = m.get_waypoint(here - up * 60.0 + right * 5.0, heading=UPTOWN)
lib = world.get_blueprint_library()
taxi = world.spawn_actor(lib.find("vehicle.taxi2"), start.transform)
taxi.set_autopilot(True, route=["straight", "straight"]) # the traffic simulation drives it
print("taxi", taxi.id, "on road", start.road_id, "lane", start.lane_id, "speed limit", start.speed_limit, "km/h")
# attachment offsets are in the taxi's frame: x forward, y left, z up
rigs = {
"chase": (Transform(Location(-7.0, 0.0, 3.0), Rotation(pitch=-12)), 90),
"dashcam": (Transform(Location(0.9, 0.0, 1.55), Rotation(pitch=-3)), 100),
}
frames, cameras = {}, []
for name, (offset, fov) in rigs.items():
bp = lib.find("sensor.camera.rgb")
bp.set_attribute("fov", fov)
cam = world.spawn_actor(bp, offset, attach_to=taxi)
frames[name] = queue.Queue()
cam.listen(frames[name].put)
cameras.append(cam)
try:
for step in range(int(a.seconds / 0.05)):
frame = world.tick()
images = {name: q.get(timeout=30) for name, q in frames.items()} # one image per camera per tick
if step % a.every:
continue
for name, image in images.items():
image.save_to_disk(os.path.join(a.out, name, f"{frame:06d}.png"))
print(f"t = {step * 0.05:5.2f} s frame {frame} taxi {taxi.get_speed():4.1f} m/s at {taxi.get_location()}")
finally:
for cam in cameras:
cam.destroy()
taxi.destroy()
world.set_ambient_traffic(vehicles=ambient["vehicles"])
world.apply_settings(original)
if __name__ == "__main__":
main()