Skip to content

Ground truth

Every rendered frame comes with the state of each agent, its box in each view, and the danger label of the safety model. This page lists those records and shows how to export them for detector training.

Per-frame records

The runner builds one FrameRecord per tick. It holds the frame index, the time t in seconds, the decision state, the alerting pair if there is one, the danger assessment, and one AgentState per agent.

AgentState field Unit Meaning
id Agent id from the scenario
kind person or bike
sx, sy m Scripted position in the scenario frame
heading Scripted heading in the scenario frame
speed m/s Scripted ground speed
location m Centre of the simulated actor's 3D box, CARLA world frame
rotation degrees (pitch, yaw, roll) of the box
extent m Half sizes of the box along the actor's forward, right and up axes
role scripted or ambient
model Rendering hint from the scenario
blueprint CARLA blueprint id that was spawned
track_error_m m Ground-plane distance between scripted and simulated position

Boxes per view

project_box(agent, camera) projects the eight corners of an agent's oriented 3D box into a view and returns None when the agent is outside it. frame_ground_truth(frame, view) does this for every agent of a frame.

Field Meaning
id, kind, role Copied from the agent
bbox [x1, y1, x2, y2] in pixels, the 2D box around the projected 3D box
corners_image The eight projected corners, an 8 × 2 array
distance_m Distance from the camera to the agent
truncation Fraction of the box that falls outside the image
visible_fraction Fraction of the agent that is not occluded, when the view has a depth or instance buffer

Pinhole boxes are clipped against the near plane edge by edge. An agent that straddles the camera plane keeps a correct box, where dropping the corners behind the camera would shrink it. Fisheye boxes go through the lens model, so they follow the distortion of the image.

from bikeped.sim.groundtruth import frame_ground_truth

for obj in frame_ground_truth(frame, "pole"):
    print(obj["id"], obj["kind"], obj["bbox"], round(obj["distance_m"], 1))

The same records are written to gt/<view>.jsonl, one JSON object per frame:

{"index": 0, "t": 0.0, "width": 1920, "height": 1080, "objects": [{"id": 1, "kind": "person", "role": "scripted", "bbox": [...], "corners_image": [...], "distance_m": ..., "truncation": ..., "visible_fraction": ...}]}

visible_fraction is null when the view has no depth or instance buffer. The runner attaches those buffers to the pole view only, when the server offers depth and instance cameras, so the other views carry boxes without a visibility estimate.

Danger label

FrameRecord.danger is the result of decision_testbench.compute_ground_truth_danger(scenario, t), the kinematic safety model that labels every number in the paper. It is evaluated on the scripted paths.

Field Unit Meaning
is_danger The cyclist is on a collision course and cannot stop in time, or time to collision is low
actionable is_danger, and a warning can still change the outcome
severity 0 to 1 Grows with the square of the cyclist's speed
ttc s Time to collision, infinite when there is none
d_min_future m Minimum future distance within the lookahead
closing_speed m/s Positive when approaching
bike_speed m/s Cyclist ground speed
stopping_dist m Distance the cyclist needs to stop
warning_budget s Warning time left before closest approach
distance m Current distance of the pair

Safety model explains how these are derived.

Export to COCO or YOLO

python -m bikeped.sim export renders/ue4_fast-approach_clear_noon --format coco
python -m bikeped.sim export renders/ue4_fast-approach_clear_noon --format yolo --view fisheye
Flag Default Meaning
RUN_DIR A finished run directory
--format coco or yolo
--view pole View whose boxes and frames are exported
--out inside RUN_DIR Annotation file (COCO) or dataset folder (YOLO)

The exporters turn one view of a run into a detection dataset with the classes person and bicycle. They read manifest.json and gt/<view>.jsonl, so they need no server.

COCO YOLO
Output One annotation file and an images folder next to it A folder with images/, labels/ and data.yaml
Box [x, y, width, height] in pixels, origin at the top-left corner class cx cy w h, centre and size divided by the image width and height
Classes person is id 1, bicycle is id 2 person is 0, bicycle is 1

Images are copied from the saved frames when the run has them. Without saved frames they are decoded from the clean video of the view, whose frame k belongs to line k of the ground-truth file. YOLO file names carry the run id, so several runs can be exported into one folder, and a frame without boxes gets an empty label file and counts as a negative.

The Python functions take filters that the defaults leave off:

from bikeped.sim.export import export_coco, export_yolo

run = "renders/ue4_fast-approach_clear_noon"
export_coco(run, "dataset/annotations.json", view="pole", min_visible=0.3, every=5)
export_yolo(run, "dataset_yolo", view="fisheye", min_area=64.0)
Argument Default Meaning
min_visible 0.0 Drop boxes whose visible_fraction is known and below this value. Boxes without a visibility estimate are kept
min_area 0.0 Drop boxes smaller than this area in square pixels
every 1 Export every n-th frame