Skip to content

How the system works

bikeped watches one crosswalk through a pole-mounted fisheye camera and sets a warning light from the ground positions of the pedestrians and cyclists it sees. This page follows one frame from the sensor to the light and names the page that covers each step.

One frame from the fisheye camera through the processing stages to the warning light and MQTT 197.9° fisheye 3.66 m pole Jetson Orin every frame, 30 FPS Crop pad 3840 crop 3500 Detect YOLO at 1280 px Locate pixel to ground x, y Track ids and history Decide three stages Light one colour per state MQTT track messages
The live path. Every stage runs once per frame on the edge device.

One frame, step by step

Step What happens Where it is configured
Capture The camera delivers a 3840 × 2160 frame holding a fisheye image circle that is taller than the sensor. video in config.yaml
Crop The frame is padded to 3840 × 3840 and centre-cropped to 3500 × 3500, which puts the 1750 px image circle in the middle of a square frame. preprocessing.pad_size, preprocessing.crop_size
Detect A YOLO detector runs on the square frame at 1280 px input size. In the bridge, a person box that overlaps a bike box is relabelled as the rider. detection in config.yaml, RIDER_* in bridge.py
Locate The bottom-centre pixel of each box is sent back through the lens model onto the ground plane, giving (x, y) in metres relative to the camera. The bridge precomputes this as a lookup table at start-up. camera in config.yaml, camera_calibration.json
Track A ground-plane tracker matches detections to tracks by metric distance and carries a track through missed detections with constant-velocity prediction. GroundPlaneTracker
Decide DecisionPipeline evaluates three stages and returns one of four states. decision in config.yaml
Act The state sets the light colour, optionally plays a sound, and the tracks are published over MQTT. feedback, mqtt in config.yaml

The geometry of the Locate step is on the perception page and the Decide step on the decision pipeline page.

The four states

State Condition Light Sound in the bridge
idle No pedestrian is tracked. Blue
safe A pedestrian is present and no cyclist has been seen within the memory window. Green Safe cue, if configured
warning A pedestrian and a recent cyclist are present, and no pair is closing. Orange
alert A cyclist is closing on a pedestrian inside the distance band. Red Alert cue, if configured

The colour mapping is apply_feedback in bridge_starter/bridge.py. The starter kit ships a SimulatedLight that prints each colour change, so the whole path runs without indicator hardware. sim_visualizer.py draws the states in the same four colours.

Live and offline paths share the decision code

decision_pipeline.py has no dependency beyond the standard library and is imported unchanged by the bridge and by decision_testbench.py. The testbench replaces the first three steps with scripted agent paths and then puts back the imperfections of a real detector: it projects each agent's 3D box through the same fisheye model to get the bounding-box localisation error, drops detections according to measured recall, and runs the same kind of ground-plane tracker before calling the pipeline. Ground truth for each frame comes from a kinematic safety model that is independent of the pipeline.

That arrangement is why a parameter change in config.yaml can be evaluated on the 24 conformance scenarios before it reaches an intersection. See the testbench for how to run it.

In this section