Skip to content

CARLA renderer

bikeped.sim plays a scenario from the scenario library in a CARLA world and records what a camera at the intersection would see, together with the decision state and ground-truth boxes for every frame. One command line drives both engine generations: Unreal Engine 4 (CARLA 0.9.14 to 0.9.16) and Unreal Engine 5 (CARLA 0.10.0).

How a render runs

The runner connects to a CARLA server and puts the world in synchronous mode with a fixed step of 1 / fps seconds, so simulated time advances only when the runner ticks. It maps the scenario frame onto a crosswalk from crosswalks.json, spawns one actor per scripted agent and one CARLA sensor per camera view, then steps through the scenario. On every tick it moves the agents along their scripted paths, reads back the simulated poses and images, evaluates the decision pipeline and the ground-truth danger label, and passes a FrameRecord to the Recorder. When the scenario ends, or the run fails, the session destroys the actors it spawned and restores the world settings.

Data flow of one CARLA render, from the scenario to the files in the run directory Scenario bikeped.scenarios paths, models, hints Runner session, actors, camera rigs one tick advances 1 / fps seconds FrameRecord agent poses, camera models, images one per tick spawn, move, tick poses, images CARLA server UE4 (0.9.14 to 0.9.16) or UE5 (0.10.0) synchronous mode Decision and danger label DecisionPipeline.evaluate compute_ground_truth_danger Recorder writes the run directory renders/<run_id>/ manifest.json video/*.mp4 gt/*.jsonl stills/
One render: the runner drives the server tick by tick and the recorder writes the run directory.

The decision pipeline and the danger model read the scripted positions (sx, sy in the scenario frame), so the state shown in the overlay does not depend on how closely a simulated actor followed its path. The gap between the scripted and the simulated position is stored for every agent as track_error_m.

UE4 and UE5

UE4 UE5
CARLA releases 0.9.14 to 0.9.16 0.10.0
Launcher script CarlaUE4.sh CarlaUnreal.sh
Python client wheels Python 3.10 to 3.12 cp310 only (Python 3.10)
Rendering Unreal Engine 4 Unreal Engine 5 with Lumen and Nanite
Maps Town01 to Town15, Town10HD_Opt by default Town10HD_Opt and Mine_01
Actor catalogue Full set of two-wheelers and walkers, wheelchair walkers on 0.9.16 Four-wheeled vehicles and walkers; no rideable two-wheeler, cyclists need --cyclist-proxy walker
Validation Developed and run end to end against CARLA 0.9.16 Tested against an in-memory stand-in for the 0.10.0 API; not yet run against a real 0.10.0 server (current limits)
Install root variable CARLA_UE4_ROOT CARLA_UE5_ROOT
--engine value ue4 ue5

The two lines differ in what they can spawn, so the renderer asks the server instead of assuming. python -m bikeped.sim probe reports the engine, the versions, the available maps and the capabilities the actor code depends on. Setup for each line is on its own page: UE4 and UE5.

A minimal session

export CARLA_UE4_ROOT=/path/to/CARLA_0.9.16   # directory that holds CarlaUE4.sh

python -m bikeped.sim server --engine ue4     # terminal 1, keeps running
python -m bikeped.sim probe                   # terminal 2, check the connection
python -m bikeped.sim render --scenario fast-approach --rig pole,fisheye

The render writes renders/ue4_fast-approach_clear_noon/: a manifest, one video per view, per-frame ground truth and a few key stills. The layout is described in Run output.

Commands

Command What it does
python -m bikeped.sim probe Connects to a server and prints engine, version, maps and capabilities
python -m bikeped.sim server Starts a CARLA server from CARLA_UE4_ROOT or CARLA_UE5_ROOT
python -m bikeped.sim presets Lists the weather, rig and quality presets
python -m bikeped.sim render Renders one scenario, a suite, or every scenario
python -m bikeped.sim export Converts the ground truth of a finished run to COCO or YOLO

Every flag is listed in the CLI reference.

In this section

  • UE4 setup

    CARLA 0.9.14 to 0.9.16 with the full two-wheeler and walker catalogue.

  • UE5 setup

    CARLA 0.10.0 with Lumen and Nanite, and what to check before rendering.

  • Cameras

    The pole, fisheye, chase, orbit, street and top-down views, and how the fisheye is built.

  • Environment

    Map, crosswalk, weather, ambient traffic, seed and frame rate.

  • Actors

    How scripted agents become CARLA walkers and vehicles, and how they are driven.

  • Ground truth

    Boxes, visibility and danger labels per frame, and export to COCO or YOLO.

  • Run output

    The files of one run and the structure of manifest.json.

  • Troubleshooting

    Connection, version and blueprint problems and what to do about them.