Command line¶
Every script and module entry point in the repository, with what it does, a usage example and its help text. Run the commands from the repository root: the scripts open config.yaml and camera_calibration.json by relative path.
| Command | Purpose |
|---|---|
python decision_testbench.py |
Evaluate the decision pipeline on the conformance scenarios. |
python run_experiments.py |
Run every experiment in the paper in sequence. |
python run_height_pitch_sweep.py |
Monte Carlo sweep over camera height and pitch. |
python crosswalk_analysis.py |
Performance against road width for one and two cameras. |
python generate_figures.py |
Write the publication figures. |
python sim_visualizer.py |
Render schematic scenario videos. |
python -m bikeped.scenarios |
List, inspect, validate, evaluate, generate and export scenarios. |
python -m bikeped.sim |
Probe a CARLA server, start one, and render scenarios. |
python calibration/calibrate_fisheye.py |
Fit the lens model to checkerboard frames. |
python calibration/capture_calibration.py |
Capture checkerboard frames from a live camera. |
python calibration/compare_fisheye_models.py |
Fit four projection models and compare their error. |
python tools/eval_models.py |
Evaluate detector checkpoints and cache per-class AP and AR. |
python tools/latency_report.py |
Per-step latency statistics from a recorded run. |
python tools/carla_find_crosswalks.py |
Extract crosswalk geometry from a CARLA map. |
python tools/build_docs.py |
Build this site. |
python bridge.py |
Run the live starter-kit bridge (from bridge_starter/). |
Testbench and experiments¶
decision_testbench.py¶
Runs the scripted scenarios through the decision pipeline and prints one row of frame counts and rates per scenario. The mode flags are checked in this order, and the first one present decides what runs: --error-map, --compare, --monte-carlo, --latency-sweep, --optimize. With no mode flag, the default configuration runs once over all scenarios.
python decision_testbench.py --compare # every named configuration, with localisation error
python decision_testbench.py --no-bbox-noise --compare # the same without localisation error
python decision_testbench.py --monte-carlo --mc-trials 50 --failure-model gilbert --burst-length 3
python decision_testbench.py --optimize --w-fn 5 --w-fp 1 --w-lat 2
--no-bbox-noise applies to the five modes. The plain run without a mode flag never adds localisation error, which is why its totals equal those of --no-bbox-noise --compare for the default configuration. --failure-model and --burst-length are read by --monte-carlo only, and the three --w-* weights by --optimize only.
usage: decision_testbench.py [-h] [--optimize] [--error-map] [--compare]
[--monte-carlo] [--mc-trials MC_TRIALS]
[--failure-model {iid,gilbert}]
[--burst-length BURST_LENGTH] [--bbox-noise]
[--no-bbox-noise] [--w-fn W_FN] [--w-fp W_FP]
[--w-lat W_LAT] [--latency-sweep]
[--eval-cache EVAL_CACHE]
[--eval-model EVAL_MODEL]
Decision Testbench
options:
-h, --help show this help message and exit
--optimize Optimize decision parameters
--error-map Print BBox error analysis
--compare Compare pipeline configurations
--monte-carlo Run Monte Carlo with stochastic detection failures
--mc-trials MC_TRIALS
Number of Monte Carlo trials (default: 50)
--failure-model {iid,gilbert}
Detection failure model: iid (default) or gilbert
--burst-length BURST_LENGTH
Mean burst length for Gilbert model (default: 3.0
frames)
--bbox-noise Include fisheye BBox localization error (default: on)
--no-bbox-noise Disable fisheye BBox localization error
--w-fn W_FN Severity-weighted FN weight (default: 5.0)
--w-fp W_FP False positive weight (default: 1.0)
--w-lat W_LAT Latency weight (default: 2.0)
--latency-sweep Sweep camera latency 0–500 ms and generate figure
--eval-cache EVAL_CACHE
Path to eval_cache.json from eval_models.py
--eval-model EVAL_MODEL
Model name in eval cache to use (default: first)
run_experiments.py¶
Calls the testbench once per experiment, then generate_figures.py and sim_visualizer.py --no-display, and stops at the first step that fails. The script's docstring puts the full run at about 45 minutes and the run with --skip-optimizer at about 10.
usage: run_experiments.py [-h] [--skip-optimizer]
Reproduce all paper experiments
options:
-h, --help show this help message and exit
--skip-optimizer Skip the optimizer (saves ~30 min)
run_height_pitch_sweep.py¶
Takes no flags. It sweeps 13 camera heights (1.5 to 7.5 m) against 10 pitches (0 to -90 degrees) on a 40 ft road with 20 Monte Carlo trials per cell, and writes height_pitch_sweep.json to the current directory. If eval_cache.json is present, its measured recall replaces the flat defaults.
crosswalk_analysis.py¶
Takes no flags. It evaluates the pipeline as a function of road width for one-camera and two-camera deployments and prints the result.
generate_figures.py¶
Takes no flags. It writes six figures as EPS and PDF into paper/: fig_height_pitch, fig_ar_curve, fig_deployment, fig_pipeline, fig_combined and fig_warning_budget. The height-pitch heatmap needs height_pitch_sweep.json and the recall curve needs eval_cache.json; each is skipped with a message when its input is missing. The script does not create the output directory:
Three scripts do not parse arguments
run_height_pitch_sweep.py, crosswalk_analysis.py and generate_figures.py ignore their command line. Passing --help starts the run.
sim_visualizer.py¶
Renders the scenarios as a schematic 3D view through the fisheye model, with the pipeline state drawn on each frame. Each scenario becomes sim_<index>_<name>.mp4 in the current directory. --scenarios takes zero-based indices into the conformance list, and --pipeline takes the name of a testbench configuration.
$ python sim_visualizer.py --no-display --scenarios 7
Rendering 1 scenarios, single-cam
Rendering: Fast Approach (1-cam, 6.5s)
Saved: sim_07_fast_approach.mp4 (196 frames)
Done.
usage: sim_visualizer.py [-h] [--scenarios SCENARIOS [SCENARIOS ...]]
[--fps FPS] [--pipeline PIPELINE] [--two-cam]
[--height HEIGHT] [--pitch PITCH] [--errors]
[--no-save] [--no-display]
3D fisheye scenario visualizer
options:
-h, --help show this help message and exit
--scenarios SCENARIOS [SCENARIOS ...]
Scenario indices (0-indexed)
--fps FPS
--pipeline PIPELINE
--two-cam Render split-screen two-camera view
--height HEIGHT Camera height in meters (default: 5.5)
--pitch PITCH Camera pitch in degrees (negative=down, default: auto)
--errors Show bbox localization error vectors and save
observation data
--no-save
--no-display
Scenario library¶
python -m bikeped.scenarios¶
python -m bikeped.scenarios list [--suite NAME] [--family NAME] [--json]
python -m bikeped.scenarios show ID
python -m bikeped.scenarios validate [--suite NAME]
python -m bikeped.scenarios evaluate [--suite NAME] [--config default] [--json]
python -m bikeped.scenarios generate --family NAME [--n 20] [--seed 0] [--out suite.json]
python -m bikeped.scenarios export --out docs_src/assets/scenarios [--simulator simulator]
| Subcommand | Purpose |
|---|---|
list |
Print the scenarios of one suite or of all suites, optionally filtered by family. |
show |
Print one scenario. ID is a slug, an exact name, or suite:index such as conformance:7. |
validate |
Check format, speeds and unique ids; exits with status 1 on problems. |
evaluate |
Run each scenario through simulate_scenario with ideal perception and a named pipeline configuration, and print danger, alert and false-alert time, alert lead and dominant state. |
generate |
Draw --n scenarios from a parametric family with a fixed seed and write them to a JSON file. |
export |
Write catalog.json and one SVG thumbnail per scenario; with --simulator DIR, also DIR/scenarios.extended.js for the browser simulator. |
usage: python -m bikeped.scenarios [-h] COMMAND ...
List, check, evaluate, generate and export bike-pedestrian scenarios. Suites:
conformance, extended, stress.
positional arguments:
COMMAND
list list scenarios with suite, family, agents and duration
show print one scenario as JSON
validate check format, speeds and unique ids; exit 1 on problems
evaluate run the decision pipeline with ideal perception and summarise
generate write a seeded parametric suite as JSON
export write catalog.json and one SVG thumbnail per scenario
options:
-h, --help show this help message and exit
See scenarios and generators.
CARLA renderer¶
python -m bikeped.sim¶
python -m bikeped.sim [-v] probe [--host 127.0.0.1] [--port 2000] [--json]
python -m bikeped.sim [-v] server --engine ue4|ue5 [--root PATH] [--port 2000] [--quality Epic|Low] [--windowed]
python -m bikeped.sim [-v] presets
python -m bikeped.sim [-v] render (--scenario ID [--suite NAME] | --suite NAME --all) [options]
python -m bikeped.sim [-v] export RUN_DIR --format coco|yolo [--view pole] [--out PATH]
render options and their defaults:
| Option | Default | Meaning |
|---|---|---|
--scenario ID |
Slug, exact name, suite:index, or an index into --suite (conformance when no suite is given) |
|
--suite NAME, --all |
Render every scenario of a suite | |
--engine auto\|ue4\|ue5 |
auto |
Engine you expect; auto accepts whatever the server runs |
--host, --port |
127.0.0.1, 2000 |
Server address |
--launch, --root PATH |
off | Start a managed server for this run from --root (or CARLA_UE4_ROOT / CARLA_UE5_ROOT) and stop it afterwards |
--map NAME |
current map | Map to load first; a fresh server has Town10HD_Opt |
--crosswalk N |
4 |
Crosswalk index: crosswalks.json on Town10HD_Opt, live map polygons elsewhere |
--crosswalks-file PATH |
crosswalks.json |
Alternative crosswalk file |
--rig LIST |
pole |
Comma list of views: pole, fisheye, chase, orbit, street, bev |
--weather NAME |
scenario hint, else clear_noon |
One of the nine weather presets |
--drive velocity\|kinematic |
kinematic |
How actors follow the script |
--ambient none\|light\|busy |
none |
Background traffic and pedestrians |
--quality draft\|standard\|cinematic |
standard |
Image size and post-processing |
--layout single\|split\|quad |
single |
Arrangement of video/composite.mp4 |
--no-overlay |
off | Skip the HUD videos |
--save-frames none\|jpg\|png |
none |
Also write every frame as an image |
--out DIR |
renders |
Parent folder of the run directory |
--seed N, --fps N |
0, 30 |
Random seed; frame rate and simulation step |
--config NAME |
default |
Key of decision_testbench.PIPELINE_CONFIGS for the decision state |
--cyclist-proxy walker |
off | Render cyclists as walkers on servers without a rideable two-wheeler |
--max-frames N |
all | Stop after N frames |
| Subcommand | Purpose |
|---|---|
probe |
Connect to a server and print the engine, the versions, the maps and what the build can spawn. |
server |
Start a CARLA server. CARLA_UE4_ROOT and CARLA_UE5_ROOT give the install location when --root is omitted. |
presets |
List the weather, rig and quality presets. |
render |
Render one scenario, a suite or everything, and write videos, a manifest and ground truth. |
export |
Convert the ground truth of a finished run to COCO or YOLO format. |
usage: python -m bikeped.sim [-h] [-v]
{probe,server,presets,render,export} ...
Render bikeped scenarios in CARLA (Unreal Engine 4: 0.9.14 to 0.9.16; Unreal
Engine 5: 0.10.0).
positional arguments:
{probe,server,presets,render,export}
probe print engine, version, maps and capabilities of a
server
server start a CARLA server and keep it running until Ctrl-C
presets list weather, rig and quality presets
render render one scenario, a suite or all scenarios
export export ground truth of a run as COCO or YOLO
options:
-h, --help show this help message and exit
-v, --verbose log debug messages
See CARLA and render output.
Calibration¶
calibration/calibrate_fisheye.py¶
Fits the equidistant lens model to checkerboard observations by bundle adjustment and writes camera_calibration.json. One of --images or --video is required.
usage: calibrate_fisheye.py [-h]
(--images IMAGES [IMAGES ...] | --video VIDEO)
[--interactive] [--board-width BOARD_WIDTH]
[--board-height BOARD_HEIGHT]
[--square-size SQUARE_SIZE] [--pad-size PAD_SIZE]
[--crop-size CROP_SIZE] [--interval INTERVAL]
[--max-frames MAX_FRAMES]
[--initial-fov INITIAL_FOV] [--fit-distortion]
[--output OUTPUT] [--visualize]
[--save-viz SAVE_VIZ]
Calibrate equidistant fisheye camera
options:
-h, --help show this help message and exit
--images IMAGES [IMAGES ...]
Image paths or glob patterns showing checkerboard
--video VIDEO Video file containing checkerboard frames
--interactive Interactively select frames from video (requires
--video)
--board-width BOARD_WIDTH
Inner corners along board width (default: 9)
--board-height BOARD_HEIGHT
Inner corners along board height (default: 6)
--square-size SQUARE_SIZE
Checkerboard square size in metres (default: 0.025)
--pad-size PAD_SIZE Preprocessing pad size (default: 3840)
--crop-size CROP_SIZE
Preprocessing crop size (default: 3500)
--interval INTERVAL Frame sampling interval for video mode (default: 30)
--max-frames MAX_FRAMES
Max frames to use from video (default: 50)
--initial-fov INITIAL_FOV
Initial FOV guess in degrees (default: 200)
--fit-distortion Also fit k1, k2 radial correction terms
--output OUTPUT Output JSON file consumed by mqtt_bridge /
decision_testbench / sim_visualizer (default:
camera_calibration.json). A sibling .yaml with the
full bundle (preprocessing block, config_yaml_updates)
is written next to it.
--visualize Show reprojection visualizations per frame
--save-viz SAVE_VIZ Directory to save reprojection images
Fisheye Camera Calibration
==========================
Calibrates the equidistant fisheye model (r = f * theta) used by mqtt_bridge.py.
Standard OpenCV fisheye calibration uses Kannala-Brandt with polynomial distortion
coefficients, which does not fit this 200-degree spherical lens on a 16:9 sensor.
This script uses a custom bundle adjustment that directly fits the equidistant
projection model to checkerboard observations.
Parameters calibrated:
cx, cy : optical center in the preprocessed frame (pixels)
focal_length : equidistant projection focal length (pixels)
k1, k2 : optional radial correction terms (r = f*(theta + k1*theta^3 + k2*theta^5))
Usage:
# From a set of images showing a checkerboard:
python calibrate_fisheye.py --images calibration_frames/*.jpg
# From a video (extracts frames at an interval):
python calibrate_fisheye.py --video calibration.avi --interval 30
# With a known board size and square dimensions:
python calibrate_fisheye.py --images *.jpg --board-width 9 --board-height 6 --square-size 0.025
# Enable radial correction terms for higher accuracy:
python calibrate_fisheye.py --images *.jpg --fit-distortion
# Interactive mode: step through a video and press 's' to select frames:
python calibrate_fisheye.py --video calibration.avi --interactive
calibration/capture_calibration.py¶
Opens a camera, shows a preview and saves frames for the calibration.
usage: capture_calibration.py [-h] [--camera CAMERA] [--outdir OUTDIR]
[--display-width DISPLAY_WIDTH]
Capture calibration frames from fisheye camera
options:
-h, --help show this help message and exit
--camera CAMERA Camera index (default: 0)
--outdir OUTDIR Output directory (default: calibration_frames)
--display-width DISPLAY_WIDTH
Preview window width (default: 1200)
calibration/compare_fisheye_models.py¶
Calibrates the same frames with the equidistant, equisolid, stereographic and orthographic models and reports the RMS reprojection error of each.
usage: compare_fisheye_models.py [-h] --images IMAGES [IMAGES ...]
[--board-width BOARD_WIDTH]
[--board-height BOARD_HEIGHT]
[--square-size SQUARE_SIZE]
[--pad-size PAD_SIZE] [--crop-size CROP_SIZE]
[--initial-fov INITIAL_FOV] [--output OUTPUT]
Compare fisheye projection models
options:
-h, --help show this help message and exit
--images IMAGES [IMAGES ...]
Image paths or glob patterns
--board-width BOARD_WIDTH
--board-height BOARD_HEIGHT
--square-size SQUARE_SIZE
--pad-size PAD_SIZE
--crop-size CROP_SIZE
--initial-fov INITIAL_FOV
--output OUTPUT Output JSON (default: model_comparison.json)
Compare fisheye projection models
=================================
Runs calibration with each supported projection model on the same
checkerboard frames and reports RMS reprojection error for each.
Models tested:
equidistant r = f * theta
equisolid r = 2f * sin(theta/2)
stereographic r = 2f * tan(theta/2)
orthographic r = f * sin(theta)
Usage:
python compare_fisheye_models.py --images calibration_frames/frame_*.png
python compare_fisheye_models.py --images calibration_frames/frame_*.png --board-width 7 --board-height 5
Tools¶
tools/eval_models.py¶
Evaluates YOLO checkpoints on a dataset and caches per-class AP and AR in eval_cache.json, which the testbench reads through --eval-cache.
python tools/eval_models.py --models yolo11x.pt --data cocoaug.yaml
python decision_testbench.py --compare --eval-cache eval_cache.json
usage: eval_models.py [-h] [--models MODELS [MODELS ...]] [--data DATA]
[--cache CACHE] [--cache-only] [--no-sizes]
Evaluate YOLO models and cache per-class AP/AR
options:
-h, --help show this help message and exit
--models MODELS [MODELS ...]
Model paths to evaluate
--data DATA Dataset config YAML (default: cocoaug.yaml)
--cache CACHE Cache file path (default: eval_cache.json)
--cache-only Only print cached results
--no-sizes Skip per-size computation (faster)
tools/latency_report.py¶
Reads the per-frame latency array written by the bridge and prints statistics per pipeline step. The script reads its arguments by hand: the first argument that does not start with a dash is the path, and --csv is the only flag.
| Argument | Default | Meaning |
|---|---|---|
PATH |
latency_data.npy |
Latency array to load. |
--csv |
off | Also write the statistics to a .csv file next to the input. |
tools/carla_find_crosswalks.py¶
Connects to a running CARLA server and lists the crosswalks of the current map. crosswalks.json in the repository root was exported with it.
| Flag | Default | Meaning |
|---|---|---|
--host |
localhost |
Server address. |
--port |
2000 |
Server port. |
--map NAME |
current map | Load this map first. |
--goto N |
none | Move the spectator to crosswalk N. |
--interactive |
off | Mark world points interactively. |
--export PATH |
none | Write the crosswalk data to a JSON file. |
tools/build_docs.py¶
Builds this site from docs_src/ into docs/ and copies the browser simulator next to it.
| Flag | Meaning |
|---|---|
--serve |
Serve a live preview instead of writing the output directory. |
--open |
Build, then open the result in the default browser. |
--strict |
Treat warnings as errors. |
--out DIR |
Write the site to DIR instead of docs/. |
Bridge starter kit¶
bridge_starter/bridge.py¶
The live system of the starter kit. Settings live in the USER CONFIG block at the top of the file, and three flags override them for a single run. Run it from inside bridge_starter/.
| Flag | Meaning |
|---|---|
--video PATH |
Read this video file instead of the webcam. |
--no-mqtt |
Disable MQTT publishing. |
--no-window |
Run without a display window. |
See the starter kit.